Abstract
Background: Poor response to questionnaires collecting outcome data in randomised controlled trials (RCTs) can affect the validity of trial results. The aim of this study within a trial (SWAT) was to evaluate the effectiveness of including a pen with a follow-up postal questionnaire on response rate.
Methods: A two-armed RCT was embedded within SSHeW (Stopping Slips among Healthcare Workers), a trial of slip-resistant footwear to reduce slips in NHS staff. Participants were randomised 1:1 to receive a pen or no pen with their follow-up questionnaire. The primary outcome was the proportion of participants who returned the questionnaire. Secondary outcomes were: time to response, completeness of response, and whether a postal reminder notice was required. Data were analysed using logistic regression, linear regression and Cox proportional hazards regression.
Results: Overall, 1466 SSHEW trial participants were randomised into the SWAT. In total, 13 withdrew from the host trial before they were due to be sent their follow-up questionnaire, 728 participants received a pen with their questionnaire, and 725 did not receive a pen. A questionnaire was returned from 67.7% of the pen group and 64.7% of the group who did not receive a pen. There was no significant difference in return rates between the two groups (OR 1.15, 95% CI 0.92 to 1.43, p=0.22), nor level of completeness of the questionnaires (AMD -0.01, 95% CI 0.06 to 0.05, p=0.77). There was weak evidence of a reduction in the proportion of participants requiring a reminder and in time to response in the pen group.
Conclusion: Inclusion of a pen with the follow-up postal questionnaire sent to participants in the SSHeW trial did not statistically significantly increase the response rate. These results add to the body of evidence around improving response rates in trials.
Trial registration: ISRCTN 33051393 (for SSHEW). Registered on 14/03/2017.
Keywords: randomised controlled trial, embedded trial, postal questionnaire, response rate, pen
Introduction
Randomised controlled trials (RCTs) are key to evaluating the effectiveness of interventions and often use postal questionnaires to collect outcome data. However, low response rates can limit the validity of the trial findings by reducing the power of the study and introducing bias 1.
Numerous strategies to increase response rates have been studied 2, 3 including sending a pen with the questionnaire. The pen acts both as a facilitator to aid completion of the questionnaire, and an incentive to return it 4, 5. The effectiveness of this intervention is equivocal with some studies reporting an increase in response rate 5– 7 whilst others failed to show a positive impact 4, 8. These studies displayed considerable heterogeneity and only two were embedded in RCTs 6, 7. A Study within a Trial (SWAT) is a self-contained study embedded within a host trial that can be used to evaluate strategies designed to improve trial efficiency 9. This SWAT evaluated the effectiveness of enclosing a pen with a follow-up postal questionnaire on response rates in the SSHeW trial 10.
Methods
Design
This two-armed RCT was embedded in the SSHeW trial, a trial evaluating the effectiveness of slip-resistant footwear to reduce slips in NHS staff 10. The SSHeW trial was registered (ISRCTN 33051393) and the trial protocol has been published 10.
Participants
The SWAT was conducted in seven NHS Trusts in England and included all eligible participants in the SSHeW trial who were due to be sent their 14-week postal questionnaire between 04.07.2018 and 12.02.2019.
Intervention
The intervention group were sent a York Trials Unit, University of York branded pen with their questionnaire. The control group did not receive a pen.
Outcomes
The SWAT outcomes are outlined in Table 1.
Table 1. SWAT outcomes.
| Outcome | Type | Definition |
|---|---|---|
| Proportion of participants
who return questionnaire (Primary Outcome) |
Binary (returned/not
returned) |
Proportion of 14-week questionnaires returned to York Trials Unit.
(Returns were censored at 11.06.2019) |
| Time to response | Time to event (days) | Number of days between the date the 14-week questionnaire was sent
and the date the returned questionnaire was received by York Trials Unit. |
| Completeness of response | Continuous (0–5) | Number of completed responses to 5 key questions on the 14-week
questionnaire. |
| Reminder notice sent | Binary (sent/not sent) | Proportion of participants sent a reminder questionnaire (sent three
weeks after the initial questionnaire if no response had been received, no additional pens were sent with reminders). |
| Cost | Continuous | Consideration of cost effectiveness of pen inclusion |
Sample size
As is usual with an embedded trial, a formal sample size calculation was not undertaken as the sample size was determined by the number of participants due to receive their 14-week questionnaire.
We anticipated that randomising 2,000 participants into the SWAT would provide 80% power to detect an absolute difference of 6% (two-sided α=0.05) in response rates between the two groups, assuming a control rate of 60%.
Randomisation
Participants were allocated to either the intervention (pen) or control (no pen) group using simple randomisation in a 1:1 ratio. The allocation sequence was generated by the SSHeW trial statistician, who was not involved in sending out the questionnaires.
Blinding
Participants were not aware of their involvement in this SWAT but due to the nature of the intervention participants and study team members could not be blinded to group allocation.
Approvals
This SWAT was approved by the Department of Health Sciences Research Ethics Committee at the University of York and the Health Research Authority (HSRGC/2016/187/A).
Statistical analysis
Data were analysed using Stata version 15 11 on an intention-to-treat basis, using two-sided tests at the 5% significance level. The models used for each outcome are given in Table 2, the values associated with the pen allocation from each model is presented with its 95% confidence interval and p-value. All models were adjusted for main trial group allocation (slip-resistant footwear or wait-list control) and pen sub-study allocation (pen or no pen).
Table 2. Analysis models.
| Outcome | Analysis model | Value presented |
|---|---|---|
| Proportion of participants who
return questionnaire |
Logistic regression | Odds ratio (OR) |
| Time to response | Cox proportional hazards
regression |
Hazard ratio |
| Completeness of response | Linear regression | Adjusted mean difference |
| Reminder notice sent | Logistic regression | OR |
Costing
The total cost of a standard SSHeW questionnaire pack was £2.42 (envelope and postage: £0.86; questionnaire and cover letter: £0.65; pre-paid envelope and postage: £0.91). The additional cost of including a pen was £0.32. The cost analysis incorporates the changes in number of questionnaires returned and reminders required.
Results
A total of 1466 participants were included in the SWAT (pen, n=733; no pen, n=733). In total, 13 participants withdrew from the main SSHeW trial after they had been randomised into the SWAT but before being sent their follow-up questionnaire, leaving 1453 participants ( Figure 1). Baseline characteristics are summarised descriptively in Table 3.
Table 3. Baseline characteristics of participants included in the analysis.
| Pen (n = 728) | No pen (n = 725) | Overall (n = 1453) | |
|---|---|---|---|
|
Main trial allocation, n (%)
Usual Care Intervention |
355 (48.8) 373 (51.2) |
376 (51.9) 349 (48.1) |
731 (50.3) 722 (49.7) |
|
Age (years),
mean (SD) |
43.0 (11.1) |
42.9 (11.5) |
43.0 (11.3) |
|
Gender, n (%)
Male Female Prefer not to say |
111 (15.3) 616 (84.6) 1 (0.1) |
90 (12.4) 635 (87.6) 0 (0.0) |
201 (13.8) 1251 (86.1) 1 (0.1) |
|
Job role, n (%)
Admin and IT Facilities Direct patient care Other |
44 (6.0)
50 (6.9) 610 (83.8) 24 (3.3) |
51 (7.0) 38 (5.2) 614 (84.7) 22 (3.0) |
95 (6.5) 88 (6.1) 1224 (84.2) 46 (3.2) |
| Average working hours, mean (SD) |
35.0 (5.2) |
35.1 (4.9) |
35.0 (5.0) |
|
Injury resulting from a slip or fall
(in previous 12 months), n (%) |
43 (5.9) |
30 (4.1) |
73 (5.0) |
Figure 1. CONSORT flow diagram of participants in the embedded trial.
Results are presented in Table 4. Overall, 962 (66.2%) questionnaires were returned (pen, 67.7%; no pen, 64.7%) and an average of 4.9/5 items were completed. There was no evidence of a difference in return rate between the groups (OR 1.15, 95% CI 0.92 to 1.43, p=0.22), nor number of items completed (AMD -0.01, 95% CI 0.06 to 0.05, p=0.77).
Table 4. Summary of results.
OD, odds ratio; HR, hazards ratio; AMD, adjusted mean difference
| Results | |||||
| Returns, n/total (%) | OR | 95% CI | p-value | ||
| Pen | No pen | Overall | |||
| 493/728 (67.7) | 469/725 (64.7) | 962/1453 (66.2) | 1.15 | 0.92, 1.43 | 0.22 |
| Time to response (days), median (IQR) | HR | 95% CI | p-value | ||
| Pen | No pen | Overall | |||
| 15 (9-33) | 18 (9-37) | 16 (9-35) | 1.12 | 0.98, 1.27 | 0.09 |
| Completeness of response, mean (SD) | AMD | 95% CI | p-value | ||
| Pen | No pen | Overall | |||
| 4.9 (0.4) | 4.9 (0.4) | 4.9 (0.4) | -0.01 | -0.06, 0.05 | 0.77 |
| Reminder sent, n/total (%) | OR | 95% CI | p-value | ||
| Pen | No pen | Overall | |||
| 339/728 (46.6) | 369/725 (50.9) | 708/1453 (48.7) | 0.83 | 0.68, 1.02 | 0.08 |
There was weak evidence of a difference, in favour of the pen group, in both time to return (median time to return 15 vs 18 days; HR 1.12, 95% CI 0.98 to 1.27, p=0.09) ( Figure 2), and in the proportion of participants requiring a reminder (OR 0.83, 95% CI 0.68 to 1.02, p=0.08).
Figure 2. Kaplan-Meier curve for time to questionnaire return.
Costing
A 3% difference in questionnaire response rate and an absolute difference in the percentage of participants who required a reminder of 1.1% were found. Considering these to be true effects, in order to receive one additional questionnaire, 33 participants would have to be sent a pen, at a cost of approximately 33x32p=£10.56. Approximately 91 participants would need to be sent a pen to prevent one reminder mailing and therefore to save £2.42. Hence, roughly one reminder is required per three retained participants, and the cost per retained participant is approximately £10.
Discussion
Whilst the results of all outcomes in this SWAT favoured the pen group, we found that the addition of a pen did not statistically significantly increase the response rate to, or completeness of, a follow-up questionnaire sent at 14 weeks post-randomisation among participants of the SSHeW trial. There was some evidence of a reduction in time to response and the number of reminders required.
It may be that, in this group of participants, the pen failed to act as a facilitator or was not a sufficient incentive to return the questionnaire, given the fact that participants in the trial already received a free pair of shoes (although offer of shoes was not conditional on returning the questionnaire).
However, the trial ultimately only had about 40% power to detect a difference of 3% in response rates (from 64.7 to 67.7%) and is therefore at risk of a type II error. Another potential weakness is that, due to the select population of healthcare workers, the results may not be generalisable to other populations or contexts.
The strength of this study is that it was a randomised trial.
Conclusion
This SWAT suggests that enclosing a pen in a questionnaire mail out may be an effective method to increase response rates but was likely underpowered to detect a statistically significant difference of the 3% observed. Since pens are inexpensive, even a small difference is likely to be cost-effective. The results contribute to the body of evidence regarding this intervention and may be included in future meta-analyses to improve power.
Data availability
Underlying data
Open Science Framework: SSHeW Trial Pen SWAT, https://doi.org/10.17605/OSF.IO/YQ76U 12.
Reporting guidelines
Open Science Framework: CONSORT checklist for ‘Enclosing a pen to improve response rate to postal questionnaire: an embedded randomised controlled trial’, https://doi.org/10.17605/OSF.IO/YQ76U 12.
Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).
Acknowledgements
The authors would like to thank the embedded trial participants who returned their trial documentation and the data administration staff at the YTU who worked to support the study.
The paper was written by the authors on behalf of the SSHeW Trial team, which is made up of the following individuals: Emily Bain (Health and Safety Executive); Sarah Cockayne (University of York); Rachel Cunningham-Burley (University of York); Caroline Fairhurst (University of York); Gillian Frost (Health and Safety Executive); Catherine Hewitt (University of York); Heather Iles-Smith (Leeds Teaching Hospital NHS Trust); Mark Liddle (Health and Safety Executive), Misbah Mogradia (Health and Safety Executive); David Torgerson (University of York), Michael Zand (Health and Safety Executive) and others. With the exception of Emily Bain and Misbah Morgradia, who supported Health Economic Evaluation for the host SSHeW Trial, all members of the SSHeW study team were involved in the designing this SWAT.
We would also like to acknowledge the support of individuals from the PROMETHEUS programme.
Funding Statement
This work was conducted as part of the SSHeW Trial which was funded by the NIHR Public Health Research Programme (Project reference number 15/05/28) and the Health and Safety Executive (HSE). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR, HSE or the Department of Health and Social Care. The University of York is the study sponsor and has legal responsibility for the initiation and management of the study: sponsor representative Mr Michael Barber, Research and Enterprise Directorate, Innovation Office, York Science Park, Heslington, York, YO10 5DG, UK. The embedded trial was supported by the MRC PROMETHEUS programme (grant number MR/R013748/1).
The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
[version 1; peer review: 2 approved]
References
- 1. Edwards P, Roberts I, Clarke M, et al. : Increasing response rates to postal questionnaires: systematic review. BMJ. 2002;324(7347):1183. 10.1136/bmj.324.7347.1183 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Edwards PJ, Roberts I, Clarke MJ, et al. : Methods to increase response to postal and electronic questionnaires. Cochrane Database Syst Rev. 2009; (3):MR000008. 10.1002/14651858.MR000008.pub4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Brueton VC, Tierney JF, Stenning S, et al. : Strategies to improve retention in randomised trials: a Cochrane systematic review and meta-analysis. BMJ Open. 2014;4(2):e003821. 10.1136/bmjopen-2013-003821 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Clark T, Khan K, Gupta J: Provision of pen along with questionnaire does not increase the response rate to a postal survey: a randomised controlled trial. J Epidemiol Community Health. 2001;55(8):595–6. 10.1136/jech.55.8.595 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. White E, Carney PA, Kolar AS: Increasing response to mailed questionnaires by including a pencil/pen. Am J Epidemiol. 2005;162(3):261–6. 10.1093/aje/kwi194 [DOI] [PubMed] [Google Scholar]
- 6. Sharp L, Cochran C, Cotton SC, et al. : Enclosing a pen with a postal questionnaire can significantly increase the response rate. J Clin Epidemiol. 2006;59(7):747–54. 10.1016/j.jclinepi.2005.10.014 [DOI] [PubMed] [Google Scholar]
- 7. Bell K, Clark L, Fairhurst C, et al. : Enclosing a pen reduced time to response to questionnaire mailings. J Clin Epidemiol. 2016;74:144–50. 10.1016/j.jclinepi.2015.12.004 [DOI] [PubMed] [Google Scholar]
- 8. Aveyard P, Manaseki S, Griffin C: The cost effectiveness of including pencils and erasers with self-completion epidemiological questionnaires. Public Health. 2001;115(1):80–1. 10.1038/sj.ph.1900714 [DOI] [PubMed] [Google Scholar]
- 9. Treweek S, Bevan S, Bower P, et al. : Trial forge guidance 1: what is a study within a trial (SWAT)? Trials. 2018;19(1):139. 10.1186/s13063-018-2535-5 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10. Cockayne S, Fairhurst C, Frost G, et al. : SSHeW study protocol: does slip resistant footwear reduce slips among healthcare workers? A randomised controlled trial. BMJ Open. 2018;8(11):e026023. 10.1136/bmjopen-2018-026023 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. StataCorp: Stata Statistical Software: Release 15.In: College Station, TX:StataCorp LLC;2017. Reference Source [Google Scholar]
- 12. Cunningham-Burley R, Roche J, Fairhurst C, et al. : SSHeW Trial Pen SWAT. 2020. 10.17605/OSF.IO/YQ76U [DOI] [Google Scholar]