Skip to main content
NCBI home page
F1000Research logoLink to F1000Research
. 2021 Jul 15;10:84. Originally published 2021 Feb 8. [Version 2] doi: 10.12688/f1000research.50890.2

SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance

Shaun Treweek 1,a, Stephanie Gallant 2, Annie S Anderson 3
PMCID: PMC8080942  PMID: 33968362

Version Changes

Revised. Amendments from Version 1

The main changes are: relative risk data are now presented alongside absolute risks; the total number of women who attended the host trial primary outcome visit has been given in Table 1; we have added reference to SWAT 77 and SWAT 86 in our Discussion; we have added some context to the Discussion around the cost of the SWAT intervention; minor text clarifications throughout.

Abstract

Background: Retention is considered the second highest trial methods priority in the UK after recruitment.

  Methods: This Study Within A Trial (SWAT) evaluated whether sending a pre-notification card around one month before a face-to-face primary outcome measurement visit compared to not sending the card increased trial retention. The SWAT was a two-arm, parallel randomised (1:1 allocation ratio), stratified by centre, study. It was embedded within the ActWELL host trial, which evaluated whether women receiving lifestyle change counselling from volunteer coaches improved outcomes including weight and physical activity.

The SWAT primary outcome was the difference in the proportion of participants attending the host trial primary outcome measurement visit. The secondary outcome was the direct cost of sending cards. Host trial participants and research staff at the primary outcome visit were blind to the SWAT. Analysis was intention-to-treat. Certainty of evidence was assessed using GRADE.

Results: 558 host trial participants took part in the SWAT and were included in the analysis. Of the 274 women sent a card, 231 attended the primary outcome visit (84.3%) compared to 230/284 (81.0%) for those not receiving a card. Sending a pre-notification card may result in a slight increase in attendance at a face-to-face primary outcome measurement visit at 1-year: risk difference = 3.3% (95% confidence interval = -3.0% to 9.6%). This is GRADE low certainty evidence. A recording error meant it was unclear whether 17 participants allocated to the card were actually sent one but a sensitivity analysis did not change the overall result or conclusion. The direct cost of producing and sending the cards was £192 GBP (€213 EUR; $260 USD), or £21.33 (€23.55; $28.77 per additional retained participant.

  Discussion: Trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits but further evaluations are needed.

Keywords: SWAT, retention, pre-notification cards, randomised trial

Introduction

Retention is considered the second highest trial methods priority in the UK after recruitment. 1 A recent UK study found that the median loss-to-follow-up in a sample of 151 trials was 11%. 2 This reduces the amount of trial data available for analysis, which is especially problematic for the trial's most important outcome - the primary outcome - because this is the outcome that will be used to judge whether the trial intervention is effective. Ensuring that retention is high is therefore of great important to trialists. Reminders are generally an effective way of increasing response rates to questionnaires and there is evidence that pre-notification (contacting participants to say that they will soon be sent a questionnaire) is beneficial, although it is not high certainty evidence. 3

There is no clear evidence from the Cochrane systematic review of trial retention interventions that pre-notification is effective for trial retention for face-to-face visits. 4 However, a Study Within A Trial (SWAT) in a trial involving women aged between 70 and 84 years at high risk of osteoporotic fractures did find that sending a newsletter to participants approximately six weeks before a trial questionnaire increased retention by 1.6% (P = 0.05). 5

SWAT question

Does sending a pre-notification card around one month before a face-to-face primary outcome measurement visit compared to not sending the card increase trial retention?

Methods

SWAT protocol

This SWAT is registered on the SWAT repository as SWAT 76. See: http://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/FileStore/Filetoupload,864300,en.pdf

Host trial

This SWAT evaluation was embedded in the ActWELL trial ( ISRCTN11057518). 6 ActWELL evaluates whether women who receive two, face-to-face lifestyle change sessions from volunteer coaches followed by up to nine telephone calls over a year, compared to no counselling, improves a range of lifestyle outcomes. The two primary outcomes were weight change and change in physical activity at 12-months. Women were invited to take part in ActWELL when they attended their routine mammography appointment (all women aged 50 - 70 in Scotland receive an offer of mammography every three years) at one of four Scottish National Health Service Breast Screening centres. A total of 560 women were randomised into the ActWELL trial.

Participants

All host trial participants who had not withdrawn were eligible.

Intervention

The intervention is a pre-notification card sent around one month before the face-to-face primary outcome measurement visit. Women were thanked to make them feel valued, were told their data were valuable regardless of how things had gone in the trial and the number of other women in the trial was highlighted. The card was signed by the Chief Investigator of the host trial and the Chief Executive of Breast Cancer Now, the charity involved in delivering the host trial intervention. The card is shown in Figure 1.

Figure 1. The ActWELL 12-month pre-notification card.

Figure 1.

Figure 1.

Open in a new tab

Comparator

No pre-notification card.

Outcomes

Primary outcome: the difference in the proportion of participants attending the host trial primary outcome measurement visit (i.e., retention).

Secondary outcome: the direct cost of sending pre-notification cards.

Sample size

The sample size was determined by the host trial so no sample size calculation was done. See Trial Forge Guidance 1 for more information about SWAT sample size calculation. 7

Randomisation

Two-arm, parallel randomised with a 1:1 allocation ratio, stratified by centre. One of the authors (ST) prepared a central randomisation list for each centre for up to 150 participants using https://www.random.org/sequences/. This was then passed to the trial manager and trial administrator who sent out the pre-notification cards.

Blinding

Women in the host trial had no knowledge of the SWAT. Host trial primary outcome visits were organised and done by research nurses, who had no knowledge of the SWAT or host trial allocation. The SWAT primary outcome, retention, was objective.

Approvals

The study was approved by East of Scotland Research Ethics Service REC 1 as part of the host ActWELL trial (17/ES/0073). The low risk nature of the SWAT evaluation meant that individual informed consent from host trial participants to be involved was not required by the ethics committee, in line with most SWATs in the UK.

Statistical analysis

The difference in the proportion of attended visits between groups was calculated using fixed effects in Comprehensive Meta-Analysis Version 3 ( https://www.meta-analysis.com/).

GRADE was used to assess the certainty of the evidence. 8 In addition to the numerical result, the result is summarised as an informative statement as per GRADE Guidelines 26. 9

Results

Two host trial participants withdrew before the 12-month host trial primary outcome measurement meaning 558 were included in the SWAT, which ran between March 2018 and July 2019 ( Figure 2; Table 1). One host trial centre recruited 151 participants, which was beyond its recruitment target and one participant beyond the randomisation list for that centre. The extra participant was manually allocated to the comparator arm. A discrepancy between the randomisation log (which indicated who should get a card) and the host trial's tracker system (which confirmed that a card had been sent to a participant) meant that we could not confirm whether 17 participants (3.0% of the total) who should have been sent the pre-notification card were actually sent one. Three further participants who should have received a card are known to have not been sent a card because the participant was called in for a host trial measurement visit before the card could be sent.

Figure 2. Flow diagram summarising the flow of participants through the SWAT evaluation.

Figure 2.

Open in a new tab

The summary statement below and the primary analysis in Table 1 are intention-to-treat as per the randomisation schedule. A sensitivity analysis done using the tracker data is also shown.

Summary statement of result: Sending a pre-notification card may result in a slight increase in attendance at a face-to-face primary outcome measurement visit at 1-year. Risk difference = 3.3% (95% confidence interval = -3.0% to 9.6%). GRADE = low certainty evidence.

Table 1. Attendance at the 12-month primary outcome measurement visit for those sent a pre-notification card and those not sent a card.

Attendance at 12-months – Intention-to-treat analysis (SWAT primary analysis)
Total number of participants who attended the primary outcome visit = 461
Allocated to be sent a pre-notification card (n = 274)* Allocated to not be sent a notification card (n = 284)
Attended visit Did not attend visit Attended visit Did not attend visit Risk difference (95% confidence interval) Risk ratio (95% confidence interval)
231 (84.3%) 43 (15.7%) 230 (81.0%) 54 (19.0%) 3.3% (−3.0% to 9.6%) 1.04 (0.97 to 1.12)
*This includes three participants who were allocated to receive a card but who are known not to have been sent one.
Attendance at 12-months – As per tracker system (sensitivity analysis)
Total number of participants who attended the primary outcome visit = 461
Tracker registers participant was sent a pre-notification card (n = 254) Tracker registers participant was not sent a notification card (n = 304)
Attended visit Did not attend visit Attended visit Did not attend visit Risk difference (95% confidence interval) Risk ratio (95% confidence interval)
213 (83.9%) 41 (16.1%) 248 (81.6%) 56 (18.4%) 2.3% (−4.0% to 8.6%) 1.03 (0.95 to 1.11)
GRADE assessment
Study limitations No serious issues.
Inconsistency of results Downgrade 1 level because this is a single study (sparsity of data).
Indirectness of evidence No serious issues.
Imprecision Downgrade 1 level because confidence interval is wide and crosses risk difference of 0.
Reporting bias No serious issues.
Open in a new tab

Direct costs

The direct costs of printing the cards was £72 GBP. Design work was extremely modest, bundled with other host trial design work and not charged separately. Second class (i.e., delivery within two days) postage costs were run through the University of Dundee mailroom at an estimated cost of £120 GBP. The total direct cost was therefore £192 GBP (€212 EUR; $259 USD).

Cost per additional participant retained

If the 274 participants who received the card had attended at the same rate as those who did not, a total of 222 participants would have attended the next visit. In fact, 231 attended, meaning an extra nine participants were retained. The cost per additional participant retained was £192/9, or £21.33 (€23.55; $28.77).

Discussion

Sending a simple card about one month prior to a face-to-face primary outcome measurement visit may result in a slight improvement in attendance. This is GRADE low certainty evidence because there is just this single evaluation and it is imprecise.

We are not aware of other pre-notification interventions that target face-to-face trial visits. The 2021 update of the Cochrane retention review 4 found no additional completed pre-notification studies (ST is a co-author of this update). There are at least two pre-notification protocols (SWAT 77 and SWAT 86) registered on the SWAT repository ( https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/) in addition to our own SWAT 76, meaning there may be additional studies underway but not yet complete. Mitchell and colleagues 5 added their evaluation to a meta-analysis of pre-notification evidence done outside trials and healthcare. As might be expected, there was substantial heterogeneity but the overall direction of effect was also in favour of pre-notification.

Strengths and limitations

We had a record-keeping error, which means we cannot say with confidence that all participants who should have been sent a pre-notification card were sent one. However, the number of participants affected was relatively small and our overall results and conclusion remain the same regardless of whether we analyse according to the randomisation schedule or the tracker system, which suggests the results are robust in the face of this error. There are currently only two evaluations of pre-notification in trials, our own of cards aimed at face-to-face visits and that of Mitchell and colleagues of a newsletter to increase questionnaire response. 5 This does not provide a broad range of contexts for this evidence base. Both evaluations were done in the UK and in women only. Finally, although the text on the card did target factors thought to influence attendance we did not use formal behavioural change theory to identify these factors, or to suggest text that might target these factors.

Implications for trial practice

Given the paucity of evidence to support retention decisions, 4 trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits at 1-year. Trials are expensive and in that context pre-notification cards are a very cheap intervention that may provide a small increase in the proportion of primary outcome data a trial team obtains. We had no negative reaction to them from participants (i.e. there were no complaints) and our cost of around £21 per additional retained participant is likely to be substantially cheaper that recruiting an additional participant to replace these lost primary outcome data.

Implications for SWAT research

Looking at the existing evidence and referring to Trial Forge Guidance 2 as to whether further SWATs evaluating this intervention are required, 10 we conclude that more evaluations are needed because the GRADE certainty in the evidence is not high, there is only a single evaluation meaning cumulative meta-analysis cannot converge and few host trial contexts are included.

Further evaluations of pre-notification in trials could target either face-to-face or questionnaires but should aim to add new host trial contexts. Future host trials should involve men. Formal approaches to developing intervention content may increase effect sizes.

Data availability

Underlying data

Open Science Framework: SWAT 76 data for host trial ActWELL, https://doi.org/10.17605/OSF.IO/N64HU. 11

This project contains the following underlying data:

  • Primary analysis 8-1-2021 (public).csv

  • Sensitivity analysis 8-1-2021 (public).csv

Reporting guidelines

Open Science Framework: CONSORT checklist for ‘SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance’, https://doi.org/10.17605/OSF.IO/B78JT. 12

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Funding Statement

The SWAT evaluation and the ActWELL trial were funded by the Scottish Government (BC/Screening/17/01).

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

[version 2; peer review: 2 approved]

References

  • 1.Smith CT, Hickey H, Clarke M, et al. : The trials methodological research agenda: results from a priority setting exercise. Trials 2014;15:32. 10.1186/1745-6215-15-32 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Walters SJ, Bonacho dos Anjos Henriques-Cadby I, Bortolami O, et al. : Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme. BMJ Open 2017;7(3):e015276. 10.1136/bmjopen-2016-015276 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 3.Edwards PJ, Roberts I, Clarke MJ, et al. : Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev 2009; Issue3. Art. No.:MR000008. 10.1002/14651858.MR000008.pub4 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.Gillies K, Kearney A, Keenan C, et al. : Strategies to improve retention in randomised trials. Cochrane Database Syst Rev 2021; Issue 3. Art. No.: MR000032. Accessed 09 June 2021. 10.1002/14651858.MR000032.pub3 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 5.Mitchell N, Hewitt CE, Lenaghan E, et al. : Prior notification of trial participants by newsletter increased response rates: a randomized controlled trial. J Clin Epidemiol 2012;65:1348–52. 10.1016/j.jclinepi.2012.05.008 [DOI] [PubMed] [Google Scholar]
  • 6.Anderson AS, Craigie AM, Gallant S, et al. : Randomised controlled trial to assess the impact of a lifestyle intervention (ActWELL) in women invited to NHS breast screening. BMJ Open 2018;8:e024136. 10.1136/bmjopen-2018-024136 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7.Treweek S, Bevan S, Bower P, et al. : Trial Forge Guidance 1: What is a Study Within A Trial (SWAT)? Trials 2018;19:139. 10.1186/s13063-018-2535-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Guyatt GH, Oxman AD, Kunz R, et al. : Going from evidence to recommendations. BMJ 2008;336:1049–51. 10.1136/bmj.39493.646875.AE [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 9.Santesso N, Glenton C, Dahm P, et al. : GRADE Guidelines 26: informative statements to communicate the findings of systematic reviews of interventions. J Clin Epidemiol 2020;119:126–35. 10.1016/j.jclinepi.2019.10.014 [DOI] [PubMed] [Google Scholar]
  • 10.Treweek S, Bevan S, Bower P, et al. : Trial Forge Guidance 2: How to decide if a further Study Within A Trial (SWAT) is needed. Trials 2020;21:33. 10.1186/s13063-019-3980-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Treweek S: SWAT 76 data for host trial ActWELL. 2021, January 14. 10.17605/OSF.IO/N64HU [DOI]
  • 12.Treweek S: SWAT 76 data for host trial ActWELL. 2021, January 14. 10.17605/OSF.IO/B78JT [DOI]
F1000Res. 2021 Aug 11. doi: 10.5256/f1000research.58441.r89686

Reviewer response for version 2

Christopher Sutton 1

I am pleased to have had the opportunity to review and comment on the revised article and can confirm that I support the decisions and amendments made by the authors and have no further comments.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Clinical trial methodology, particularly around trial conduct and retention methods.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2021 Apr 27. doi: 10.5256/f1000research.53982.r79341

Reviewer response for version 1

Michelle E Kho 1

Treweek et al. conducted a study within a trial to evaluate whether a post card sent 1 month prior to an in-person primary outcome research visit improved retention within the multicentre ActWELL randomized trial of lifestyle coaching. 558 participants from the host trial were sent a pre-notification card. Of this cohort, 84% who were allocated a pre-notification card attended the primary outcome visit, compared to 81% who were not allocated a pre-notification card. This is a well-written, well-designed, and well-reported study. It will make important contributions to further understanding ways to improve retention to clinical trials. I have some considerations for the authors.

Moderate issues:

  1. Results – To help the reader understand the impact on the trial, I recommend the authors first report the total number of women who attended their primary outcome follow-up visit, then report the proportion by group.

  2. Results – I initially found Table 1 confusing. Since those who did not attend the visit is the complement of those who did attend a visit, I suggest reporting only those who attended the visit. The risk difference represents the difference between those who attended the visit. I also suggest adding the 95% CI around the estimates, and report 1 decimal point for consistency with the risk difference.

  3. Can the authors report whether patients recalled receiving the pre-notification card before their outcomes visit?

  4. Implications for trial practice – I suggest adding text to further contextualize the implications of the relative cost of this intervention, the overall investment costs in the enrolled patients to-date, and the importance of ascertaining the primary outcome. Even a 3% improvement in primary outcome ascertainment could affect trial results.

Minor issues:

  1. Abstract – I suggest reporting absolute values in the intervention and control groups, then the risk difference.

  2. Introduction – I suggest adding a statement regarding the importance of the primary outcome in a trial and the need for interventions to optimize ascertainment.

  3. Intervention – I suggest moving the text regarding behavioural change theory in the development of the pre-notification card from the methods to the limitations section of the manuscript.

  4. Results – Data error for 17 participants (3%) – suggest also reporting % of total group to help the reader easily evaluate the small number. Interestingly, the risk difference was also 3%. The authors conducted a sensitivity analysis, and the results were similar. I suggest this is also a strength of the study.

  5. Implications for trial practice. I suggest framing the study findings specific to 1-year follow-up measures.

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Critical care clinical trials; health research methodology; rehabilitation; physiotherapy

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2021 Jul 5.
Shaun P Treweek 1

Response to comments on F1KR00CDE F1R-VER53982-A

5/7/2021

Thanks for the reviewers’ comments on our manuscript. Our responses are below.  We have made the necessary changes to the manuscript using track changes so that they can be easily seen. We have also submitted a clean version with the changes accepted.

Reviewer #1

1. Introduction:  clearly state that the previous SWAT of the impact of a newsletter on retention was statistically significant (p<0.05 … just!)

Done. We have also added the exact difference of 1.6% rather than saying ‘around 1%’.

2. I would query the statement that “All trial participants were eligible.”  

We have changed our text to:

‘All host trial participants who had not withdrawn were eligible.’

3. The primary outcome would be better specified as a binary outcome measure (i.e. at the participant level e.g. “attendance at the host trial primary outcome measurement visit”).

We have thought about this but since the primary result we report is the difference in attendance between those who got the card and those who did not (i.e. 3.3%) and not the actual attendance for each group (84% and 81%), we think the way we describe the primary is correct.

4. Statistical analysis: the method of analysis should be stated, in addition to the software used

We have added that we used a fixed effects model.

5. Present relative relative measures of effect as well.

Good idea, apologies for not doing it originally. Done.

6. I would like to see more information on the economic analysis.  For example, how much did the intervention cost per participant retained?

We only collected direct costs so our economic analysis is very limited. However, we liked the suggestion to calculate a cost per (extra) participant retained, which means we have added the text below to the cost information. We have also added the cost per additional participant retained to the abstract.

‘ Cost per additional participant retained

If the 274 participants who received the card had attended at the same rate as those who did not, a total of 222 participants would have attended the next visit. In fact, 231 attended, meaning an extra nine participants were retained. The cost per additional participant retained was £192/9, or £21.33 (€23.55; $28.77).’

7. In the Discussion and Strengths and Limitations, the authors state that there are no additional pre-notification studies; it would be better to stated that there are no additional completed pre-notification studies

We have now amended the Discussion text to:

‘The 2021 update of the Cochrane retention review [4] found no additional completed pre-notification studies (ST is a co-author of this update). There are at least two pre-notification protocols (SWAT 77 and SWAT 86) registered on the SWAT repository (https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/) in addition to our own SWAT 76, meaning there may be additional studies underway but not yet complete.’

8. Reference list: references 11 and 12 re identical (Reference 12 should refer to the CONSORT checklist for this SWAT).

References 11 and 12 were added by the publisher, not us, but they do work as they should in the version I downloaded yesterday (8/6/2021). There may perhaps have been an error in the version the reviewer was sent that has now been corrected by the publisher.

Reviewer #2

1. Results – To help the reader understand the impact on the trial, I recommend the authors first report the total number of women who attended their primary outcome follow-up visit, then report the proportion by group.

We have added this number to the header row of Table 1 and we have done this for both the intention-to-treat and sensitivity analyses.

2. Results – I initially found Table 1 confusing. Since those who did not attend the visit is the complement of those who did attend a visit, I suggest reporting only those who attended the visit. The risk difference represents the difference between those who attended the visit. I also suggest adding the 95% CI around the estimates, and report 1 decimal point for consistency with the risk difference.

We have thought about the reviewer’s comment re. only showing the number who attended a visit but have decided that we would prefer to be explicit and give both the number who did attend and the number who did not. The main reason is that it avoids the reader having to do a calculation and all the raw numbers are there for readers to do other calculations should they wish.

There is no confidence interval for the proportions attending a visit: the numbers are not an estimate but the actual number attending. The risk difference (and now relative risk in response to Reviewer 1, Comment #5) does have a confidence interval around it. We have  changed our presentation so that we now consistently report to 1 decimal place as requested by the reviewer.

3. Can the authors report whether patients recalled receiving the pre-notification card before their outcomes visit?

Participants were not asked whether they recalled getting a card so we cannot unfortunately report on this. Both the participants and the research nurses who took measurements at the primary outcome visits did not know that we were running this study (i.e. they were blind) and to maintain the blind, we did not ask about the cards. It is possible that some participants mentioned this to the nurse but we have no record of this.

4. Implications for trial practice – I suggest adding text to further contextualize the implications of the relative cost of this intervention, the overall investment costs in the enrolled patients to-date, and the importance of ascertaining the primary outcome. Even a 3% improvement in primary outcome ascertainment could affect trial results.

Good point.  We have re-written the Implications for trial practice section:

‘Given the paucity of evidence to support retention decisions [Reference 4], trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits. Trials are expensive and in that context pre-notification cards are a very cheap intervention that may provide a small increase in the proportion of primary outcome data a trial team obtains. We had no negative reaction to them from participants (i.e. there were no complaints) and our cost of around £21 per additional retained participant is likely to be substantially cheaper that recruiting an additional participant to replace these lost primary outcome data.’      

   

5. Abstract – I suggest reporting absolute values in the intervention and control groups, then the risk difference.

Done.

6. Introduction – I suggest adding a statement regarding the importance of the primary outcome in a trial and the need for interventions to optimize ascertainment.

We have added the following text to the Introduction:

‘This reduces the amount of trial data available for analysis, which is especially problematic for the trial’s most important outcome– the primary outcome– because this is the outcome that will be used to judge whether the trial intervention is effective. Ensuring that retention is high is therefore of great importance to trialists.’

7. Intervention – I suggest moving the text regarding behavioural change theory in the development of the pre-notification card from the methods to the limitations section of the manuscript.

Done.

8. Results – Data error for 17 participants (3%) – suggest also reporting % of total group to help the reader easily evaluate the small number. Interestingly, the risk difference was also 3%. The authors conducted a sensitivity analysis, and the results were similar. I suggest this is also a strength of the study.

We have added the proportion (3%) to the Results and also mentioned that the similarity of the sensitivity analysis results to the intention-to-treat results shows the results are robust in the face of this recording error.   

9. Implications for trial practice. I suggest framing the study findings specific to 1-year follow-up measures.

We have made the change suggested and also added ‘at 1-year’ to our evidence summary statement in the Abstract and Results.

F1000Res. 2021 Mar 1. doi: 10.5256/f1000research.53982.r79342

Reviewer response for version 1

Christopher Sutton 1

Overall, this is an excellent manuscript and an extremely welcome and useful addition to the literature on trial retention.  The work provides a clear citation of the relevant literature.

The methodology is sound and the trial performed well; it has significant academic merit. There are, however, aspects of the methods and analysis which are a little under-specified and unclear, and suggestions for clarification are detailed below.  Likewise, the statistical and economic analysis is appropriate, although I do suggest additional analyses and results be presented to enhance interpretation and generalisability. 

The source data underlying the results are available.  The conclusions are drawn adequately and supported by the results, with an appropriate sensitivity analysis performed to investigate the potential impact of some degree of failure to definitively send the pre-notification card to 20 participants allocated to receive the card.

Specific comments:

  1. Introduction:  clearly state that the previous SWAT of the impact of a newsletter on retention was statistically significant (p<0.05 … just!).

  2. I would query the statement that “All trial participants were eligible.”  From my perspective, the eligibility criteria were (a) Trial participant of the ActWELL trial (b) remaining in ActWELL at 1 month prior to the 12-month final follow-up.

  3. The primary outcome would be better specified as a binary outcome measure (i.e. at the participant level e.g. “attendance at the host trial primary outcome measurement visit”).

  4. Statistical analysis: the method of analysis should be stated, in addition to the software used.

  5. In line with CONSORT 2010, for binary outcome measures both absolute and relative measures of effect are recommended.  Across different trial populations, interventions, duration of follow-up etc., it is likely that there are different ‘control’ rates of retention and the impact of any intervention is probably more likely to be on a relative rather than an absolute scale.  For example, the newsletter trial referenced in the Introduction (Mitchell et al., (2012)) had a control group retention rate of 94.6% (and an absolute increase of 1.6% in the intervention group), compared with this trial which had a control group retention rate of 81.0% (and an absolute increase of 3.3%).  It is therefore recommended that the results are also presented as a Risk Ratio (Relative Risk) or Odds Ratio.

  6. I would like to see more information on the economic analysis.  For example, how much did the intervention cost per participant retained?

  7. In the Discussion and Strengths and Limitations, the authors state that there are no additional pre-notification studies; it would be better to stated that there are no additional completed pre-notification studies (there are at least two other pre-notification trials registered on the SWAT Repository Store (SWAT77 and SWAT86).

  8. Reference list: references 11 and 12 re identical (Reference 12 should refer to the CONSORT checklist for this SWAT).

Is the work clearly and accurately presented and does it cite the current literature?

Yes

If applicable, is the statistical analysis and its interpretation appropriate?

Partly

Are all the source data underlying the results available to ensure full reproducibility?

Yes

Is the study design appropriate and is the work technically sound?

Yes

Are the conclusions drawn adequately supported by the results?

Yes

Are sufficient details of methods and analysis provided to allow replication by others?

Yes

Reviewer Expertise:

Clinical trial methodology, particularly around trial conduct and retention methods.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2021 Jul 5.
Shaun P Treweek 1

Response to comments on F1KR00CDE F1R-VER53982-A

5/7/2021

Thanks for the reviewers’ comments on our manuscript. Our responses are below.  We have made the necessary changes to the manuscript using track changes so that they can be easily seen. We have also submitted a clean version with the changes accepted.

Reviewer #1

1. Introduction:  clearly state that the previous SWAT of the impact of a newsletter on retention was statistically significant (p<0.05 … just!)

Done. We have also added the exact difference of 1.6% rather than saying ‘around 1%’.

2. I would query the statement that “All trial participants were eligible.”  

We have changed our text to:

‘All host trial participants who had not withdrawn were eligible.’

3. The primary outcome would be better specified as a binary outcome measure (i.e. at the participant level e.g. “attendance at the host trial primary outcome measurement visit”).

We have thought about this but since the primary result we report is the difference in attendance between those who got the card and those who did not (i.e. 3.3%) and not the actual attendance for each group (84% and 81%), we think the way we describe the primary is correct.

4. Statistical analysis: the method of analysis should be stated, in addition to the software used

We have added that we used a fixed effects model.

5. Present relative relative measures of effect as well.

Good idea, apologies for not doing it originally. Done.

6. I would like to see more information on the economic analysis.  For example, how much did the intervention cost per participant retained?

We only collected direct costs so our economic analysis is very limited. However, we liked the suggestion to calculate a cost per (extra) participant retained, which means we have added the text below to the cost information. We have also added the cost per additional participant retained to the abstract.

‘ Cost per additional participant retained

If the 274 participants who received the card had attended at the same rate as those who did not, a total of 222 participants would have attended the next visit. In fact, 231 attended, meaning an extra nine participants were retained. The cost per additional participant retained was £192/9, or £21.33 (€23.55; $28.77).’

7. In the Discussion and Strengths and Limitations, the authors state that there are no additional pre-notification studies; it would be better to stated that there are no additional completed pre-notification studies

We have now amended the Discussion text to:

‘The 2021 update of the Cochrane retention review [4] found no additional completed pre-notification studies (ST is a co-author of this update). There are at least two pre-notification protocols (SWAT 77 and SWAT 86) registered on the SWAT repository (https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/Repositories/SWATStore/) in addition to our own SWAT 76, meaning there may be additional studies underway but not yet complete.’

8. Reference list: references 11 and 12 re identical (Reference 12 should refer to the CONSORT checklist for this SWAT).

References 11 and 12 were added by the publisher, not us, but they do work as they should in the version I downloaded yesterday (8/6/2021). There may perhaps have been an error in the version the reviewer was sent that has now been corrected by the publisher.

Reviewer #2

1. Results – To help the reader understand the impact on the trial, I recommend the authors first report the total number of women who attended their primary outcome follow-up visit, then report the proportion by group.

We have added this number to the header row of Table 1 and we have done this for both the intention-to-treat and sensitivity analyses.

2. Results – I initially found Table 1 confusing. Since those who did not attend the visit is the complement of those who did attend a visit, I suggest reporting only those who attended the visit. The risk difference represents the difference between those who attended the visit. I also suggest adding the 95% CI around the estimates, and report 1 decimal point for consistency with the risk difference.

We have thought about the reviewer’s comment re. only showing the number who attended a visit but have decided that we would prefer to be explicit and give both the number who did attend and the number who did not. The main reason is that it avoids the reader having to do a calculation and all the raw numbers are there for readers to do other calculations should they wish.

There is no confidence interval for the proportions attending a visit: the numbers are not an estimate but the actual number attending. The risk difference (and now relative risk in response to Reviewer 1, Comment #5) does have a confidence interval around it. We have  changed our presentation so that we now consistently report to 1 decimal place as requested by the reviewer.

3. Can the authors report whether patients recalled receiving the pre-notification card before their outcomes visit?

Participants were not asked whether they recalled getting a card so we cannot unfortunately report on this. Both the participants and the research nurses who took measurements at the primary outcome visits did not know that we were running this study (i.e. they were blind) and to maintain the blind, we did not ask about the cards. It is possible that some participants mentioned this to the nurse but we have no record of this.

4. Implications for trial practice – I suggest adding text to further contextualize the implications of the relative cost of this intervention, the overall investment costs in the enrolled patients to-date, and the importance of ascertaining the primary outcome. Even a 3% improvement in primary outcome ascertainment could affect trial results.

Good point.  We have re-written the Implications for trial practice section:

‘Given the paucity of evidence to support retention decisions [Reference 4], trialists could consider using pre-notification as they may gain a slight increase in retention to face-to-face trial measurement visits. Trials are expensive and in that context pre-notification cards are a very cheap intervention that may provide a small increase in the proportion of primary outcome data a trial team obtains. We had no negative reaction to them from participants (i.e. there were no complaints) and our cost of around £21 per additional retained participant is likely to be substantially cheaper that recruiting an additional participant to replace these lost primary outcome data.’      

   

5. Abstract – I suggest reporting absolute values in the intervention and control groups, then the risk difference.

Done.

6. Introduction – I suggest adding a statement regarding the importance of the primary outcome in a trial and the need for interventions to optimize ascertainment.

We have added the following text to the Introduction:

‘This reduces the amount of trial data available for analysis, which is especially problematic for the trial’s most important outcome– the primary outcome– because this is the outcome that will be used to judge whether the trial intervention is effective. Ensuring that retention is high is therefore of great importance to trialists.’

7. Intervention – I suggest moving the text regarding behavioural change theory in the development of the pre-notification card from the methods to the limitations section of the manuscript.

Done.

8. Results – Data error for 17 participants (3%) – suggest also reporting % of total group to help the reader easily evaluate the small number. Interestingly, the risk difference was also 3%. The authors conducted a sensitivity analysis, and the results were similar. I suggest this is also a strength of the study.

We have added the proportion (3%) to the Results and also mentioned that the similarity of the sensitivity analysis results to the intention-to-treat results shows the results are robust in the face of this recording error.   

9. Implications for trial practice. I suggest framing the study findings specific to 1-year follow-up measures.

We have made the change suggested and also added ‘at 1-year’ to our evidence summary statement in the Abstract and Results.

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Availability Statement

    Underlying data

    Open Science Framework: SWAT 76 data for host trial ActWELL, https://doi.org/10.17605/OSF.IO/N64HU. 11

    This project contains the following underlying data:

    • Primary analysis 8-1-2021 (public).csv

    • Sensitivity analysis 8-1-2021 (public).csv

    Reporting guidelines

    Open Science Framework: CONSORT checklist for ‘SWAT 76 evaluation: randomised evaluation of sending pre-notification cards to trial participants before a face-to-face primary outcome measurement to increase attendance’, https://doi.org/10.17605/OSF.IO/B78JT. 12

    Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

    Articles from F1000Research are provided here courtesy of F1000 Research Ltd

    RESOURCES