Best practice guidance for the use of strategies to improve retention in randomized trials developed from two consensus workshops

Abstract

Objectives

To develop best practice guidance for the use of retention strategies in randomized clinical trials (RCTs).

Study Design and Setting

Consensus development workshops conducted at two UK Clinical Trials Units. Sixty-six statisticians, clinicians, RCT coordinators, research scientists, research assistants, and data managers associated with RCTs participated. The consensus development workshops were based on the consensus development conference method used to develop best practice for treatment of medical conditions. Workshops commenced with a presentation of the evidence for incentives, communication, questionnaire format, behavioral, case management, and methodological retention strategies identified by a Cochrane review and associated qualitative study. Three simultaneous group discussions followed focused on (1) how convinced the workshop participants were by the evidence for retention strategies, (2) barriers to the use of effective retention strategies, (3) types of RCT follow-up that retention strategies could be used for, and (4) strategies for future research. Summaries of each group discussion were fed back to the workshop. Coded content for both workshops was compared for agreement and disagreement. Agreed consensus on best practice guidance for retention was identified.

Results

Workshop participants agreed best practice guidance for the use of small financial incentives to improve response to postal questionnaires in RCTs. Use of second-class post was thought to be adequate for postal communication with RCT participants. The most relevant validated questionnaire was considered best practice for collecting RCT data. Barriers identified for the use of effective retention strategies were: the small improvements seen in questionnaire response for the addition of monetary incentives, and perceptions among trialists that some communication strategies are outdated. Furthermore, there was resistance to change existing retention practices thought to be effective. Face-to-face and electronic follow-up technologies were identified as retention strategies for further research.

Conclusions

We developed best practice guidance for the use of retention strategies in RCTs and identified potential barriers to the use of effective strategies. The extent of agreement on best practice is limited by the variability in the currently available evidence. This guidance will need updating as new retention strategies are developed and evaluated.

1. Introduction

Loss to follow-up in randomized clinical trials (RCTs) can lead to biased results. Differential loss to follow-up from different treatment groups does not generally occur in RCTs [1]. Until recently, the evidence for strategies to improve participant follow-up in research was limited to broad systematic reviews of methods to improve response to questionnaires in research [2,3] or methods to improve retention in prospective population-based cohort studies [4]. Narrative reviews describe retention strategies to maximize in-person follow-up in research [5,6]. However, none of these reviews focus on evaluations of strategies to improve retention in RCTs.

In a Cochrane systematic review of strategies to improve retention specifically in RCTs, six types of strategies were evaluated, namely incentives; new questionnaire formats; and communication, behavioral, methodology, and participant case management strategies [7]. The strategies that improved retention were: offering or adding monetary incentives and, based on the results of single RCTs, recorded delivery of questionnaires, and a package of strategies designed for sending postal questionnaires known as the Total Design Method (TDM) [8,9]. A related qualitative study found incentives, communication, and new questionnaire format strategies are routinely used by trialists to try to improve retention in UK primary-care RCTs, based on research experience rather than any knowledge of their effect [10].

Although these two studies examine the use and effect of strategies to improve retention in RCTs, to our knowledge, guidance on the use of retention strategies in practice does not exist. To address this, we aimed to develop consensus-based guidance for the use of retention strategies in RCTs based on the evidence available. Three commonly used methods for developing consensus for best practice are: (1) the Delphi method, (2) the Nominal Group Technique (NGT), and (3) Consensus Development Conferences [11,12]. These methods differ in how (1) data are collected, for example, through questionnaires or face-to-face contact, (2) opinion is aggregated, and (3) decisions are fed back to participants for reconsideration [12]. The Delphi method uses rounds of postal questionnaires to record experts’ views on a topic [11,12]. NGT uses structured group discussions with experts associated with a topic, and Consensus Development Conferences bring individuals related to a topic together to hear the best evidence available to help make decisions about best practice [12]. This method was used by the National Institutes of Health to develop best practice for the monitoring and treatment of medical conditions [12–14].

We used the Consensus Development Model to bring together trial personnel to (1) explore the evidence available for the use and effect of strategies to improve retention in RCTs, (2) develop best practice guidance for the use of retention strategies in RCTs, (3) identify barriers to the use of retention strategies, and (4) identify retention strategies for future research.

2. Methods

2.1. Selection of consensus workshop participants

Research personnel associated with two UK Clinical Trials Units (CTUs) with expertise in the design and management of RCTs conducted across diverse disease areas, clinical, and geographical settings were recruited to participate in the consensus development workshops. All research personnel listed on the seminar list for each CTU were invited via e-mail to contribute to a workshop to develop best practice guidance for the use of retention strategies in RCTs. The invitation included an abstract summarizing the results of the Cochrane review and the qualitative study. The invitation was sent 1 week before each workshop with a reminder sent on the morning of the workshop.

2.2. Format of consensus workshops

Our consensus workshops were held in November and December 2013 during a regular time-tabled seminar slot at each CTU. Workshops commenced with an introduction and overview of the purpose and format of the workshop, followed by a 20-minute presentation of evidence for the effect and use of strategies to improve retention in RCTs from the Cochrane systematic review and the qualitative study [7,10] (Table 1). Three concurrent facilitated group discussions followed to discuss the evidence for (a) incentives (group 1), (b) communication strategies (group 2), and (c) questionnaire format strategies (group 3). The evidence for three seldom used retention strategies, i.e., methodology, behavioral, and case management, was discussed after the questionnaire format discussions at workshop 1 and after the communication strategy discussions at workshop 2.

Table 1.

Summary of evidence from the Cochrane systematic review of strategies to improve retention in RCTs and qualitative study on the use of retention strategies in RCTs

	Systematic review results
	Method of data collection	Number of RCTs in meta-analysis	Total number of participants in meta-analysis	RR 95% CI	P value	Absolute benefit based on 50% baseline response	Qualitative study results
Effective retention strategies
Monetary incentives
Addition of monetary incentive vs. none [15–17]	Postal questionnaire	3	3,166	RR 1.18; 1.09–1.28	P < 0.0001	76 questionnaires per 1,000 sent	Incentives are used in cash or voucher format given up front or on questionnaire completion. General agreement that small monetary incentives are viewed favorably by ethics committees. Uncertainty about effect of monetary incentives given up front or offered for questionnaire return.
Offer of a monetary incentive vs. none [18]a	Web-based questionnaire	2	3,613	RR 1.25; 1.14–1.38, heterogeneity P value = 0.14	P < 0.00001	100 questionnaires per 1,000 sent	Offers of monetary incentives used.
Higher value monetary incentive vs. lower value monetary incentive (Bailey unpublished)	Postal questionnaire	2	902	RR 1.12; 1.04–1.22	P = 0.005	55 questionnaires per 1,000 sent	£5–£20 monetary incentives used. Concern about coercion with higher valued incentives.
Strategies with some evidence of effect based on single RCTs
Communication
Total Design Method (TDM) vs. customary postal communication [8]	Postal questionnaire	1	226	RR 1.43; 1.22–1.67	P < 0.0001	—	Some elements of TDM used to improve postal questionnaire response.
Recorded delivery vs. telephone reminder [9]	Postal questionnaire	1	192	RR 2.08; 1.11–3.87	P = 0.02	—	Recorded delivery used to send further copy of questionnaire/study materials. Mixed opinions on usefulness.
Methodology strategies Open vs. blind RCT design [19]	Postal questionnaire	1	538	RR 1.37; 1.16–1.63	P = 0.0003	—	Open trial design not used to improve retention. Masking RCT participants to the intervention used to avoid bias associated with open RCTs.
Strategies with unclear evidence of effect
New questionnaire strategies
Short questionnaires vs. long (Edwards unpublished, Svoboda, unpublished) [20,21]a	Postal questionnaire	5	7,277	RR 1.04; 1.00–1.08	P = 0.07	20 questionnaires per 1,000 sent	Shorter follow-up questionnaires used with a second reminder. Long questionnaires thought to be off putting for participants.
More relevant questionnaires (i.e., those relating to alcohol use) vs. less relevant [21]a	Web based	2	3,893	RR 1.07; 1.01–1.14	P = 0.03	—	No comments on the use of more or less relevant questionnaires.
Noneffective strategies
Nonmonetary incentives
Addition of nonmonetary incentive vs. none [22–24]	Postal questionnaire	6	6,322	RR 1.00; 0.98–1.02, some heterogeneity (P value = 0.02)	P = 0.91	—	Gifts used as reminders about RCTs. Uncertainty about effectiveness.
Offer of a nonmonetary incentive vs. no offer [25,26]	Postal questionnaire	2	1,138	RR 0.99; 0.95–1.03	P = 0.60	—	Offers of gifts not mentioned as a strategy to improve retention.
Addition of monetary incentive vs. offer of prize draw entry [18]	Postal questionnaire	2	297	RR 1.04; 0.91–1.19	P = 0.56	—	Offers of entry into a prize draw seldom used but thought to potentially be useful.
Offer of monetary donation to charity vs. none [18]	Web-based questionnaire	1	815	RR 1.02; 0.78–1.32	P = 0.90	—	Offers of donations to charity not mentioned as a way to improve retention.
Communication strategies
Enhanced letter vs. standard letter [23,27]	Postal questionnaire	2	2,479	RR 1.01; 0.97–1.05	P = 0.70	—	Enhanced letter routinely used to improve questionnaire return.
Priority post vs. regular post [23,24,28]	Postal questionnaire	7	1,888	RR 1.02; 0.95–1.09	P = 0.55	—	First-class post routinely used to send post to participants.
Additional reminder vs. usual follow-up practices [29–32]a	Postal questionnaire	6	3,401	RR 1.03; 0.99–1.06	P = 0.13	—	SMS text reminders thought useful for contacting young RCT participants. Thought similar system used for text reminders for NHS clinic appointments may improve follow-up in RCTs. Telephone reminders routinely used. Concerns about harassment with too many reminders. E-mail reminders thought useful for improving response.
Early vs. late questionnaire administration [23]	Postal questionnaire	1	664	RR 1.10; 0.96–1.26	P = 0.19	—	Questionnaires sometimes posted later in week to arrive at weekend.
Additional monthly reminder to RCT site vs. usual reminder (Land unpublished)	Return to research site	1	272	RR 0.96; 0.83–1.11	P = 0.57	—	Additional reminders to sites not mentioned as a way to improve retention.
Addition of telephone survey vs. monetary incentive plus questionnaire [33]	Postal questionnaire	1	700	RR 1.08; 0.94–1.24	P = 0.27	—	Telephone survey seldom used to improve retention. Telephone calls used by nurses to contact participants.
New questionnaire strategies
Disease/condition questions before generic vs. generic questions before disease/condition questions [34]a	Postal questionnaire	2 quasi-randomized	9,435	RR 1.00; 0.97–1.02	P = 0.75	—	Suggestions to improve questionnaire format include: < 10 pages, clear succinct questions, avoid repetition, include participant feedback section, use illustrations, color coordinate questionnaires for each time point.
Long and clear questionnaires vs. shorter condensed questionnaires [35]	Postal questionnaire	1	900	RR 1.01; 0.95–1.07	P = 0.86	—	Shorter questionnaires used where possible.
Behavioral/motivational strategies
Behavioral/motivational strategies vs. standard information [36,37]	Return to research site	2	273	RR 1.08; 0.93–1.24	P = 0.31	—	Not used, very negative about the usefulness of using behavioral strategies for retention.
Case management
Case management vs. usual follow-up [38]	Return to research site	1	703	RR 1.00; 0.97–1.04	P = 0.99	—	Case management, seldom used, thought to be potentially useful for retention but expensive.

Abbreviations: RCT, randomized clinical trial; CI, confidence interval; RR, risk ratio.

^aPublication reports more than one retention RCT.

Questions for each discussion group were agreed a priori by the authors (V.B., F.S., S.P.S., and G.R.). Discussion groups were asked (1) whether they were convinced by the evidence; (2) to identify clinical areas and types of follow-up the strategy could be used for; and (3) to identify barriers to the use of the retention strategy. For strategies with no evidence of an effect on retention (i.e., nonmonetary incentives, priority/first-class post, enhanced letters, modified questionnaires, case management, and behavioral strategies), the workshop participants were asked to (1) consider whether those strategies were in current use and (2) identify barriers that prevent changing the use of such retention strategies (Appendix 1 at www.jclinepi.com).

Questions for each discussion group and tabulated summaries of the Cochrane review and qualitative study results for each discussion were distributed to the discussion group facilitators (F.S., J.T., S.P.S., and F.S.) before each workshop (Appendix 1 at www.jclinepi.com, Table 1). The workshop participants were assigned to a discussion group by numbers 1–3. The occupation/role, research area, contact details, and discussion group allocation were recorded for each workshop participant. Discussion groups were asked to consider the evidence presented and, where possible, to agree best practice for the use of retention strategies in RCTs. Facilitators encouraged participants to draw on their knowledge and expertise of retention in RCTs and to focus their discussions on the retention strategy allocated. Summaries of each group discussion and the best practice guidance agreed by the group were presented to each entire workshop for agreement.

2.3. Ethics approval

The consensus workshops focused on discussions of published evidence and were held in the full knowledge of senior management at each CTU. Research personnel at each CTU were informed, before the consensus workshop, that the aim was to develop best practice guidance for retention in RCTs. Consent to participate in the workshops was considered given when research personnel attended. The Cochrane review, qualitative study, and consensus workshops contributed to a PhD thesis. Ethics approval for the qualitative study was sought from University College London Ethics Committee UCL 2342/002.

2.4. Data management and analysis

The workshop discussions were recorded by either handwritten contemporaneous notes (workshop 1), or digitally by voice recorder (Olympus WS-300M, or Sony model ICD-UX522) (workshop 2). The discussion notes were subsequently typed (by V.B.), and the digital recordings transcribed (by V.B.) and anonymized by removing RCT identifiers and acronyms. Each discussion group transcript was e-mailed to the discussion group facilitator to check for accuracy, and any additions and corrections were clarified by e-mail. Broad codes were used to code textual data for: how convinced participants were by the results; the types of RCTs using the retention strategy; the types of follow-up retention strategies were used for, for example, questionnaire follow-up; barriers to use of effective retention strategies; further research; and guidance for best practice. Discussion group notes and transcripts were read and reread. The discussion group questions were used as a framework for content analysis. Coded text was identified, summarized, and interpreted grounded in the discussion group transcripts/notes. The results were compared across both workshops.

3. Results

Sixty-six self-selected RCT personnel associated with both CTUs participated in the workshops. They represented the spectrum of research personnel working on RCTs including chief and principal investigators, statisticians, RCT managers, data managers, research assistants, research associates, and PhD students. Three group discussions were held during each workshop. The characteristics and number of participants attending each discussion group are illustrated in Table 2. Discussion groups were heterogeneous in terms of the participants’ occupation/research role and research area.

Table 2.

Consensus workshop characteristics and participants

Discussion group	No. of participants	Research roles of participants	Research areas represented
Workshop 1
Incentives	10a	Statisticians (n = 5)	Sexual health, alcohol reduction, e-health, learning disabilities, cardiovascular disease
		Trial managers (n = 1)
		Research assistants (n = 1)
		Data managers (n = 1)
		Clinicians (n = 2)
Communication	7	Research scientist/fellow (n = 2)	Aging, e-health, mental health, smoking cessation, cardiovascular disease, primary care
		Clinicians (n = 2)
		PhD students (n = 2)
		Qualitative researchers (n = 1)
New questionnaire formats and other strategies	9	Statisticians (n = 2)	Sexual health, smoking cessation, cardiovascular disease, primary care
		Research assistants (n = 4)
		Research fellows/associates (n = 1)
		Clinicians (n = 2)
Workshop 2
Incentives	19a	Statisticians (n = 3)	Cancer, infectious diseases, statistical trial methodology
		Trial managers (n = 5)
		Trial assistants (n = 2)
		Data managers (n = 5)
		Research scientists/fellows (n = 2)
		Clinicians (n = 2)
Communication, and other strategies	12a	Statisticians (n = 6)	Cancers, infections
		Data managers (n = 5)
		Clinicians (n = 1)
New questionnaire formats	9	Statisticians (n = 4)	Cancers, infections
		Data manager (n = 1)
		Communication specialist (n = 1)
		Research fellow/associates (n = 1)
		Clinicians (n = 2)

^aMore workshop participants expressed an interest in these discussion groups.

3.1. Incentive strategies

Table 1 summarizes the results of the Cochrane review and the qualitative study. The consensus workshop participants agreed with the results of the Cochrane review and the qualitative study that financial incentives could be used to improve questionnaire response in RCTs [7,10]. They were not convinced that incentives would improve retention in all RCTs. They felt that the addition of a monetary incentive depended on the age, socioeconomic group, educational level, and medical condition associated with RCTs participants. The small benefit gained from adding a monetary incentive to improve questionnaire response and the additional administration needed to use such incentives were thought to be potential barriers to the use of monetary incentives (Table 1).

In agreement with the results of the qualitative study [10], the workshop participants felt that monetary incentives could be perceived as coercive and that the value of monetary incentives should not be so high that RCT participants become suspicious about the use of research resources. The workshop participants also felt that the value of monetary incentives used to improve retention in RCTs should not be so low that RCT participants feel undervalued. A value of £5–£20 was agreed for financial incentives.

The workshop participants agreed with the results of the Cochrane review [7] and the qualitative study [10] that nonmonetary incentives, for example, mugs and pens with RCT information, for example, logos, may not improve retention. They felt that branded study gifts (i.e., letters, pens, and mugs) could impact negatively on retention in RCTs, particularly if the gift implied that the participant was associated with a medical condition that they felt uncomfortable about. Although there was no evidence of effect for nonmonetary incentives in the Cochrane review [7], the workshop and qualitative study participants acknowledged that nonmonetary incentives were used to thank RCT participants for their participation and they were keen to continue to do this.

3.2. Communication strategies

The workshop participants were unconvinced by the results for communication strategies in the Cochrane review [7]. The review showed that enhanced letters, first-class post, sending questionnaires early, and additional reminders (i.e., telephone, e-mail, text messages, calendars with reminders, telephone surveys, and monthly reminders to sites of upcoming assessments) had no impact on questionnaire response in RCTs. Yet, the qualitative study found that additional telephone, letter, and e-mail reminders are routinely used with the aim of improving follow-up in primary-care RCTs [10]. The workshop participants felt that the evidence of effect and no effect for communication strategies was limited as the results were based on few retention RCTs. They also felt that the use of a communication strategy to improve retention in RCTs was dependent on other factors, for example, the medical condition, age, socioeconomic status of the RCT participants, and the method of data collection, for example, postal questionnaire, or face-to-face contact. Additional reminders were thought to be particularly important for improving low response to questionnaires in RCTs of behavioral interventions, for example, smoking cessation, or in RCTs with healthy volunteers. The workshop participants were reluctant to change this practice because of the improvements they believed they had seen to participant follow-up in RCTs.

Based on the results of single RCTs in the Cochrane review, recorded delivery [9], and a package of postal communication strategies for questionnaire follow-up known as the TDM [8] improved questionnaire response. The TDM encompasses a hand-signed letter, white envelope with a hospital logo and commemorative stamp, and a self-addressed and stamped envelope (Table 1). The workshop participants thought that recorded delivery may inconvenience RCT participants if they were out when their post was delivered. The TDM was thought to be outdated, but some elements were thought to potentially improve questionnaire follow-up, for example, sending personalized letters with questionnaires. Electronic communication with RCT participants was thought to be used more than paper methods in current practice, and the workshop participants felt that adapting the TDM for use with electronic questionnaires could help improve questionnaire response in RCTs. The workshop participants thought that a personalized approach to retention for RCT participants including an additional visit after recruitment to determine their preferred mode/s of contact, for example, by e-mail, or SMS text message, could improve retention. More evaluations of communication strategies were thought to be needed.

The only result for communication strategies that the workshop participants were convinced by was the evidence of no effect for first-class post. There was agreement that first-class post was costly and second-class post could be now be used for sending routine post to RCT participants.

3.3. New questionnaire formats

The workshop participants were convinced by the evidence from the Cochrane review for new questionnaire formats [7]. The results of the review suggest that there was no clear evidence that long and clear questionnaires are more effective than short condensed questionnaires or that placing disease/condition questions before generic questions improves response. The findings also suggest that more relevant questionnaires (in the context of alcohol use) may improve response. The qualitative study showed that shorter questionnaires are used to try to improve response in primary-care RCTs [10] (Table 1). Human nature, the RCT participant’s medical condition, and other factors, for example, RCT participant’s time and priorities were thought to influence questionnaire response. Based on their experience, the workshop participants perceived that questionnaires measuring outcomes for treatments of terminal conditions, for example, cancers, have a higher response than questionnaires collecting behavioral outcomes, for example, smoking cessation. They felt that RCT participants may abandon completing an electronic questionnaire without an option to save and return to later, or where the questionnaire was perceived to be too long. There was general agreement that offering alternative ways to complete outcome data, for example, by post, text, or e-mail could improve response. There was skepticism about using less relevant questionnaires to collect outcome data [21] (Table 1). There was agreement that the most relevant and validated questionnaire should be used to measure RCT outcomes and that plain English should be used in questionnaires.

3.4. Other strategies

The workshop participants also discussed other retention strategies identified by the Cochrane review, which were seldom evaluated or used to improve retention in RCTs [7]. These strategies were: methodology (an open vs. closed RCT design [19]), case management (where case managers were assigned to RCT participants [38]), and behavioral strategies (provision of a motivational strategy delivering information about goal setting and time management to RCT participants [36,37]).

3.4.1. Methodology (open vs. closed RCT designs)

The workshop participants were not convinced by the evidence based on one RCT [19] from the Cochrane review that an open RCT design improves retention (Table 1). They agreed with the findings of the qualitative study that using an open RCT design to improve retention could bias RCT results as participants would be aware of their treatment allocation. They felt that the decision to mask the allocation was informed if not dictated by (1) the type of intervention, i.e., drug treatment, behavioral intervention and (2) the need to avoid biases associated with disclosing the intervention, for example, performance bias.

3.4.2. Case management

Some workshop participants said they would consider using case management [38] (Table 1) to improve retention for RCTs with elderly or disabled participants if they had more information about the time and resources needed. There was no evidence from the Cochrane review that this strategy improved retention in RCTs; however, the qualitative study found that elements of case management had been used with the aim of improving retention in RCTs conducted through primary care.

3.4.3. Behavioral strategies

There was also no clear evidence that behavioral retention strategies improve retention in RCTs. Although the results from the qualitative study were very negative about the use of this strategy, the workshop participants reported having no experience using such strategies to improve retention in RCT and one participant felt behavioral/motivational strategies [36,37] (Table 1) could increase retention in RCTs of interventions for the prevention and treatment of, for example, infectious diseases.

3.5. Retention strategies identified for further research

The workshop participants thought that more evaluations of (1) communication strategies to encourage RCT participants to return to sites for follow-up and (2) electronic follow-up technologies are needed. Some participants felt that some of the retention strategies evaluated to date were too similar to usual RCT follow-up practice to make a difference to retention, for example, sending a letter with an additional sentence estimating the length of time it should take to complete a questionnaire [27] (Table 1). The workshop participants generally agreed that retention strategies for future evaluation should be substantially different from usual follow-up procedures.

3.6. Best practice guidance for the use of retention strategies in RCTs

Best practice guidance agreed for the use of retention strategies in RCTs from the group discussions is summarized in Table 3.

Table 3.

Best practice guidance for the use of retention strategies in RCTs

Retention strategy	Guidance for the use of retention strategies in RCTs	Barrier to implementation
Incentives	Financial incentives valued £5–£20 can be considered to improve questionnaire response	Small benefit gained from adding incentives Additional administration involved in sending monetary incentives to RCT participants
	Nonmonetary incentives can be considered as a token of appreciation for RCTs participants with careful consideration of appropriate branding	None identified
Communication strategies	Second-class post can be used for routine postal communication with RCT participants	None identified
New questionnaire formats	Alternative ways to complete outcome data, for example, by post, text, or e-mail can be used to improve response	None identified
	A relevant and validated questionnaire should be used to measure RCT outcomes	None identified
	Plain English should be used in questionnaires	None identified

Abbreviation: RCT, randomized clinical trial.

4. Discussion

The consensus development workshop format provided an opportunity for RCT personnel to meet and discuss the evidence for strategies to improve retention in RCTs. Both workshops were well attended. Agreement was reached for the use of incentives, second-class post, and some general principles around questionnaire design to help improve retention in RCTs. Potential barriers to using effective retention strategies were identified, that is, the limited evidence available for each retention strategy identified by the Cochrane review, the heterogeneity of settings, and the small gains in response from the addition of monetary incentives. Barriers to changing the use of strategies with no effect were the workshop participants’ resistance to change the use of existing practices perceived to be effective. Strategies potentially worthy of future evaluation were also identified.

4.1. Strengths and limitations of the consensus workshops

The consensus workshops provided the opportunity for a multidisciplinary group with RCT expertise to consider the quantitative and qualitative evidence available, agree best practice for the use of retention strategies in RCTs, and discuss potential barriers to the use of effective strategies in RCTs. The guidance provides a baseline on which to add other best practice guidance as evidence on the effects of new retention strategies emerge.

The consensus workshop participants were self-selected and experienced in the leadership, design, management, and analyses of RCTs conducted across diverse disease areas and settings and were interested in improving retention in RCTs. They may have had prior knowledge of the results of the Cochrane review and qualitative study through the information provided in the workshop invitation and by attending conferences/meetings where preliminary results were presented. These characteristics and factors contributed to lively, well-informed group discussions about best practice for the use of retention strategies in RCTs and potential barriers to use.

The workshops were held at each CTU during a regular seminar slot and were very well attended. Convening workshops on CTU sites made it more convenient for participants to attend. Although the workshops were shorter than consensus development workshops held by the National Institutes of Health [12,14], we found that there was adequate time to discuss the focused questions about the specific retention strategy assigned to each discussion group.

The best practice guidance agreed by the workshop participants for the use of retention strategies in RCTs has been informed by evidence from a Cochrane review [7], qualitative study [10], and expert opinion. A limitation of the guidance is that the views of RCT participants themselves are not represented. Future guidance on the use of retention strategies in RCTs would benefit from the involvement of RCT participants to help trialists better understand the priorities, barriers, and facilitators to retention from a participant’s perspective. Moreover, it may also help identify new and preferred strategies needing future evaluation.

The extent of agreement on best practice for the use of retention strategies in RCTs is limited by the variability in the evidence from the systematic review and information from the qualitative study. Furthermore, no formal quantitative agreement through voting was used to agree feedback from the discussion groups. Nevertheless, there was qualitative agreement among the workshop participants at each workshop and opposing views were recorded. Although the consensus workshops were limited to two CTUs, this best practice guidance is broadly applicable to other UK and other CTUs internationally.

4.2. Meaning and implications

Although the Cochrane review, qualitative study, and development of best practice guidance were conducted in the UK, we feel that the guidance developed has broad application to RCTs conducted in other settings; however, this would depend on the trial context and follow-up procedures. Trialists can now consider adding small monetary incentives valued £5–£20 to improve questionnaire response in RCTs, knowing that the recommendation is based on the best available evidence and endorsed by those involved in their conduct. How monetary incentives are delivered, that is, given up front, or offered, will depend on the context of each RCT. Certainly, offers of incentives could be more cost-effective for RCTs with lower response rates, as nonresponders would receive no incentive. Evidence from Edwards Cochrane review showed monetary incentives given up front in epidemiological studies improved questionnaire response [2]. We are not aware of any evidence to suggest that monetary incentives given up front are better than offering a monetary incentive to improve questionnaire response in RCTs, and therefore, a test of monetary incentives given up front vs. an offer of a monetary incentive is needed.

Although the Cochrane review showed no effect for nonmonetary incentives (i.e., gifts), it is clear from the qualitative study and the workshops that gifts are used in RCTs [7,10], albeit with skepticism about the impact these have on retention. We are not aware of any research that identifies the most appropriate rewards for RCT participant’s time. Therefore, involving RCT participants in the development of future best practice guidance on the use of retention strategies in RCTs may help to identify more acceptable nonmonetary incentives for this group. More research studies are needed to identify and evaluate appropriate ways to demonstrate appreciation to RCT participants for their contributions to RCTs.

In considering the lack of evidence that priority/first-class post improves questionnaire response in RCTs [7], workshop participants agreed that using second-class post should be used to cut the costs of postal communication with RCT participants, with the savings redirected to other RCT costs, for example, staff training. This guidance can be used to persuade trial personnel to use second-class post for future postal communication with RCT participants.

Even without clear evidence that modifying the format of a questionnaire improved response in RCTs [7], questionnaire length, readability, content, and acceptability of the topic to RCT participants were still considered important factors for improving questionnaire response in the qualitative study [10] and in the consensus workshops. Although the consensus was that questionnaires should be clear, relevant, and validated to help to minimize bias and maximize precision in effect estimates, testing the validity and reliability of new questionnaires is time consuming and costly [39]. Therefore, trialists may wish to consider carefully the potential impact of questionnaire development on budgets and time lines for future RCT research proposals.

Lau’s (2015) recent systematic review of systematic reviews of strategies for improving implementation of complex interventions in primary-care practice found that educational outreach visits, educational meetings, audit, and feedback were the most effective ways to improve implementation of interventions [40]. An evaluation of the use of this best practice guidance for the use of retention strategies in RCTs would inform how well the guidance has been implemented in RCTs at the CTUs where we conducted our consensus workshops and the impact of the guidance on retention in those RCTs.

Our consensus development workshops identified some barriers to implementing the evidence for strategies to improve retention in RCTs. These barriers may change over time as new retention strategies are developed and will need to be considered when the next set of guidance is developed.

To our knowledge, this is the first set of guidance for the use of retention strategies in RCTs. We are aware of the results of other embedded RCTs published since the review that have evaluated the effectiveness of SMS text messages [41], e-mail reminders [42], a paper reminder to improve postal questionnaire response in RCTs [43], pens accompanying a questionnaire [44], and offers of incentives [45]. This best practice guidance will need updating when the Cochrane review of strategies to improve retention in RCTs is updated to incorporate this new evidence.

5. Conclusion

The consensus workshop discussions helped develop best practice guidance for the use of retention strategies in RCTs and identify potential barriers to the use of effective strategies. The extent of agreement is limited by the variability in the currently available evidence. More evaluations of newer retention strategies, particularly technological strategies, are needed. This guidance will require updating as evidence on the effects of new strategies becomes available.

Supplementary Material

Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.jclinepi.2017.05.010.

What is new?

Key findings

• Best practice guidance was agreed for the use of small financial incentives, second-class post, and relevant validated questionnaires in randomized clinical trials (RCTs).

• Barriers for the use of effective retention strategies in RCTs were identified.

What this adds to what was known?

• This is the first set of best practice guidance for the use of retention strategies in RCTs.

What is the implication and what should change now?

• The extent of agreement on best practice is limited by the variability in the currently available evidence.

• There is some resistance to change existing retention practices thought to be effective.

• More evaluations of face-to-face and electronic follow-up technologies to improve retention in RCTs are needed.

• This guidance will need updating as new retention strategies are developed and evaluated.