Systematic review identifies number of strategies important for retaining study participants

Abstract

Objective

Loss to follow-up threatens internal and external validity yet little research has examined ways to limit participant attrition. We conducted a systematic review of studies with a primary focus on strategies to retain participants in health care research.

Study Design

We completed searches of PubMed, CINAHL, CENTRAL, Cochrane Methodology Register, and EMBASE (August 2005). We also examined reference lists of eligible articles and relevant reviews. A data-driven thematic analysis of the retention strategies identified common themes.

Results

We retrieved 3,068 citations, 21 studies were eligible for inclusion. We abstracted 368 strategies and from these identified 12 themes. The studies reported a median of 17 strategies across a median of six themes. The most commonly reported strategies were systematic methods of participant contact and scheduling. Studies with retention rates lower than the mean rate (86%) reported fewer strategies. There was no difference in the number of themes used.

Conclusion

Available evidence suggests that investigators should consider using a number of retention strategies across several themes to maximize the retention of participants. Further research, including explicit evaluation of the effectiveness of different strategies, is needed.

Loss to follow-up of research participants threatens the internal and external validity of a study[1;2]. The study results may be biased by differential dropout between comparison groups or by differences between those participants who drop out and those that continue to participate. Loss to follow-up also may threaten the generalizability of a study, as well as its statistical power. Despite these threats, little attention has been paid to the optimal methods of maintaining participants in a study.

Much of the existing literature on strategies to retain participants in research studies is limited to descriptions of ‘lessons learned’. For example, based on their experiences in studies of people over 65 years old, Cassidy et al (2001) suggested that personalized attention, empathy and support from study staff resulted in higher participant completion[3]. Shumaker et al (2000) similarly drew on their own experiences to outline approaches to promote retention including screening out those likely to not remain in the study and early identification and tracking of study participants who are poor or non-adherers[4].

Coday et al (2005) collected lessons learned from 14 NIH-funded behavioral change trials [5]. They elicited perceived barriers to participant retention and 61 retention strategies from the project staff and investigators from these trials. The retention strategies were categorized into eight themes which study personnel then ranked based on perceived effectiveness. The strategy category of flexibility followed by incentives, benefits and persistence were rated as most effective by the study personnel.

Davis et al (2002) completed a review of trials between 1990 and 1999 identifying 21 studies that included a description of retention strategies and retention rates[6]. The authors provided a table listing the trials rank-ordered based on the retention rate (specifics not provided) and suggested that those studies with higher retention were those using a combination of strategies. The paper combined discussion about retention and recruitment strategies as well as about studies with mail or telephone follow-up versus those with in-person visits.

In the existing literature, we could not identify any explicit evaluation of the effectiveness of retention strategies, such as a comparison of follow-up rates using different strategies. To help comprehensively synthesize strategies for participant retention in research studies and to evaluate areas for future methodological research in this field, we conducted a systematic review of studies which described strategies for maximizing retention for in-person follow-up.

METHODS

Searching and Study Selection

We sought English-language publications reporting research that described retention strategies for in-person follow-up and included actual retention rates. We reviewed only those published reports with a primary focus on retention strategies. We included studies that provided data on retention rates, described data from a primary study, and provided information regarding strategies used to retain participants. We searched PubMed (August 11, 2005), Cumulative Index of Nursing and Allied Health Literature (CINAHL) (August 8, 2005), the Cochrane Controlled Trials Register (CENTRAL) and the Cochrane Methodology Register (Issue 3, 2005), and EMBASE (August 11, 2005). The search strategies combined text words and controlled vocabulary words for concepts of ‘attrition’, ‘retention’, ‘patient dropouts’ and ‘loss to follow-up’. The specific search strategies are provided in Appendix I. We also examined the reference lists of eligible articles and relevant reviews [3;7–31].

All retrieved citations were screened independently by two authors to determine eligibility. Results of the search and screening processes were maintained in a citations database (ProCite, ISI, Berkley, CA).

Data Abstraction and Synthesis

Two reviewers abstracted information about the study, including design, location and target population and health condition. We also abstracted all retention strategies and retention rates at all follow-up time points. Abstracted data were entered into a relational database (Microsoft Access, Redmond, WA).

We completed a data-driven thematic analysis of the retention strategies[32]. Using an iterative, multi-step process, we reviewed all abstracted retention strategies to identify themes and to classify each strategy within these themes. Initially, two authors independently reviewed the strategies and identified themes. Second, a third author reviewed these independent results, reconciled differences and proposed a list of common themes. Third, this list of themes and categorization of strategies was discussed at a team meeting and we developed consensus on a final list of themes. Fourth, two authors independently re-reviewed the strategies and assigned each strategy to one of the themes from the final list. Finally, a third author adjudicated all assigned themes. Any remaining disagreements were discussed and resolved at a team meeting.

RESULTS

Review process

Our search identified 3,068 potentially relevant citations of which 115 were considered further at the full-text level. Of these 115, we excluded an additional 94 primarily because the article did not describe a primary study (e.g., article was a review or commentary) or because the article did not describe specific retention strategies. There were 21 articles that met our eligibility criteria. Figure 1 summarizes the results of our search and selection processes.

Figure 1.

Summary of Search and Screening Results

Study Characteristics

The 21 studies in this review included 18,782 participants (mean=894) (Table 1) [33–53]. Only one study was conducted entirely outside of the United States. Thirteen of the studies were randomized controlled trials and the remaining eight were prospective cohort studies. Of the trials, seven included behavioral interventions, four were of a drug intervention, two combined behavioral and drug interventions, and one involved a surgical intervention. Six of the studies examined substance abuse, with four of these including a behavioral or drug intervention. Other populations or health conditions included mothers and individuals with cancer, heart disease, and AIDS.

Table 1.

Summary of study characteristics

Study	Study_type	Sample Size	Condition/Population	Demographics	Intervention	Number of Follow-up Visits*	Total length of Follow-up (months)	Retention Rate (%)
Katz, 2001	randomized controlled trial	286	new mothers	Mothers >=18 years old with no psychiatric diagnoses who had <5 prenatal visits during pregnancy or had prenatal care initiated in third trimester; predominantly unmarried; mostly African-American	behavioral program	1	12	59
Catlett, 1993	prospective cohort	145	mothers	Mothers who gave birth to infants who weighed less than 1,500 grams	no intervention	2	24	62
Pappas, 1998	randomized controlled trial	211	substance abuse/young urban people	Sixth-grade students at two inter-city schools	behavioral program	2	12	70
Parra-Medina, 2004	randomized controlled trial	189	diabetes	Overweight adults >=45 years old living in rural, medically underserved communities; 80% African American; 47% had less than high school education	behavioral program	3	12	82
Hessol, 2001	prospective cohort	2628	AIDS/women	HIV-positive women	no intervention	1	60	82
Motzer, 1997	randomized controlled trial	217	cancer	Families of female breast-cancer patients	behavioral program	1	10	83
Desland, 1991	prospective cohort	92	substance abuse	Heroine users; included self-referred heroine users and those identified by court	drug intervention	5	12	84
Mazzuca, 2004	randomized controlled trial	431	arthritis/women	Obese women 45 – 64 years old who had unilateral knee osteoarthritis	drug intervention	1	30	85
Dilworth-Anderson, 2004	prospective cohort	202	caregivers/African-American	African-Americans >65 years old living	no intervention	1	60	87
BootsMiller, 1998	randomized controlled trial	485	substance abuse and psychiatric illness	Patients newly admitted to hospital for mental illness; 74% male, 77% African-American, mean age of 33 years old; 30% homeless, 47% had prior convictions	drug intervention	5	24	88
Miller, 1999	randomized controlled trial	187	depression/elderly	Depressed elderly patients	drug and behavioral intervention	1		88
Dennis, 2000	prospective cohort	25	no specific disease/ethnic/racial minorities	Elderly African-Americans living in urban areas	drug and behavioral intervention	1	3	88
Dudley, 1995	prospective cohort	4954	AIDS/gay and bisexual men	Homosexual and bisexual men18 – 70 years old in four urban areas	no intervention	1	114	89
Meyers, 2003	prospective cohort	195	substance abuse/young people	Youth (mean age = 16 years old) in alcohol and drug treatment programs; 66% Caucasian, 66% male, 65% urban, 32% referred by court	no intervention	2	6	92
Bell, 1985	randomized controlled trial	3837	heart disease	People who had at least one myocardial infarction	drug intervention	3	36	92
Froelicher, 2003	randomized controlled trial	2481	heart disease	Patients who experienced depression and/or low social support following myocardial infarction; 66% male, 66% white, 53% high school education or less	behavioral program	1	54	93
Sprague, 2003	randomized controlled trial	440	fractures	Patients at 24 centers worldwide who had fractures of the tibial shaft	surgery	1	12	94
Robles, 1994	prospective cohort	829	substance abuse and pregnancy	Women >= 18 years old at fourth gestational month; included a group who had more than 3 drinks per week during the first trimester and a group who smoked two or more joints per month during first trimester; 53% non-Caucasian; average 12 years education; 73% earned less than $500/month	no intervention	3	36	95
Goldberg, 2005	randomized controlled trial	162	weight loss	Overweight and obese adults 25–80 years old living in major metropolitan area	behavioral program	3	18	96
Bailey, 2004	randomized controlled trial	176	cancer	Healthy women >=14 years old with cervical lesions	drug intervention	1	48	99
Desmond, 1995	randomized controlled trial	610	substance abuse	Illegal opioid users treated in a methadone clinic	behavioral program	1	12	99

^*number of follow-up visits with retention rate reported

No study explicitly compared the effectiveness of different retention strategies. The average follow-up time of the studies was approximately 30 months with a range from 3 months to 9.5 years. Nine of the studies included a total follow-up time of one year or less. Twelve studies reported retention rates for only one follow-up point at the end of the study; though the range among the studies was 1 to 5 visits (mean=2). The mean retention rate at the last, or only, follow-up time was 86% (range of 59% to 99%).

Synthesis of Retention Strategies

We abstracted 368 retention strategies from which we identified 12 themes. Table 2 lists the themes based on the order that the strategies would most likely be implemented. For instance, the theme ‘Community involvement’ is listed first as getting the community involved in the study, including in design issues such as retention strategies, could be initiated prior to the start of the study. ‘Study Identity’ follows ‘Community Involvement’ and includes strategies such as the creation and use of a study logo on t-shirts, calendars and all correspondence to create for the study participants a sense of identification with or belonging to a study. Although, many of the strategies could be classified across multiple themes we classified each strategy as belonging in one theme only. We maintained the data-driven themes rather than collapse categories. For instance, ‘Reminders’ were kept as a separate specific theme and not combined under ‘Contact and Scheduling Methods’, a category including systematic methods for maintaining patient contact.

Table 2.

Retention strategy themes

Theme	Description	Examples
Community Involvement	Involve community in study design, recruitment and retention	Present pilot project idea to church leadership and congregation Create community advisor panel and consult with panel for recommendations regarding protocol and participation
Study Identity	Create study identity for participants	Create a project identity by using similar colors and fonts on all study materials Give participants a t-shirt printed with study logo
Study Personnel	Characteristics, training and management of study personnel	Assign one primary clinician to each participant Encourage study personnel to show empathy towards subject’s personal situation in scheduling appointments/cancellations
Study Description	Explain study requirements and details, including potential benefits and risks, to participants	Inform subjects that they will be followed over time and specify the timetable and the methods that will be used to locate them Offer a copy of a newspaper article or study brochure to each participant
Contact and Scheduling Methods	Use systematic method for patient contact, appointment scheduling and cohort retention monitoring	Mail a newsletter to participants that includes a message from PI, photos of project staff, and preliminary findings Obtain multiple contacts for each participant, including 2contacts not residing with the participant
Reminders	Provide reminders about appointments and study participation	Mail reminder postcards to participants one week before appointment Visit in-patients before discharge to remind them of out- patient follow-up plan
Visit Characteristics	Minimize participant burden through characteristics and procedures of follow-up study clinic	Offer flexible clinic appointments (early morning, evenings, and weekends) Provide background music for restful atmosphere in clinic
Benefits of Study	Provide benefits to participants, and families, that are directly related to the nature of study	Provide free annual physical examination Form educational and support groups for families and patients
Financial Incentives	Provide financial incentives or payment	Provide payment to families in control group ($20/visit/4visits) Provide pharmacy gift certificate to participant at first follow-up visit ($25)
Reimbursement	Provide reimbursement for research-related expenses or tangible support to facilitate participation	Provide taxi fare or have staff member pick up study participants Provide child care during visit
Non-Financial Incentives	Provide non-financial incentives or tokens of appreciation	Provide an inexpensive token of appreciation (e.g., coffee mug, pen, refrigerator magnet) to participant at each visit Host holiday parties for study participants
Special Tracking Methods	Methods of tracking or dealing with hard-to-find or difficult participants	Conduct clinic and street outreach for lost to follow-up participants Identify and address obstacles hindering participation for problem patients

The largest number of strategies (n=123) was classified within the theme of ‘Contact and Scheduling Methods’. Eighty-six percent of studies reported using strategies from this theme which included systematic methods for patient contact, scheduling of appointments and monitoring of cohort retention. Specific strategies included obtaining updated contact information every two months [50] and making multiple attempts to contact subjects for complete data by phone and mail [43]. The theme ‘Visit Characteristics’ included the next highest number of strategies (n=57) and was also considered in 86% of the studies. This theme comprised strategies related to minimizing participant burden through the characteristics and procedures of the follow-up visit, such as offering to conduct the interview on the front porch or outside the home [40] and providing refreshments in the follow-up clinic [34].

Financial incentives were provided by eight studies. Eleven strategies involved payments for follow-up visits or interviews, including the provision of gift certificates. A specific dollar amount was provided for ten of these strategies with a range of $10USD to $50USD (median=$20USD). Two studies provided payments of $10USD or $20USD to family or friends for assistance in finding difficult to find study participants [35;39] and one reported a supplemental payment of $5 for resistant participants [52]. Finally, one study paid $5 for each urine sample provided [39]. The retention rate for those studies that reported using financial incentives was higher than for studies not reporting using financial incentives (mean 88% versus 85%, p=0.9).

There was a range of 3 to 42 strategies (median=17) and a range of 3 to 10 themes (median=6) per study (Table 3). No study reported strategies in all themes. Eight studies reported less than the mean retention rate of 86%. These studies had a mean retention rate of 76% with an average follow-up length of 22 months. The other 13 studies, with a mean retention rate of 92%, had a mean follow-up length of 35 months. The studies with retention rates below the mean reported using fewer strategies than those studies with a higher than mean retention rate (12 versus 21, p=0.05). These two groups of studies did not differ in the number of different themes (mean=6 for both). The small number of studies and the heterogeneity of their characteristics limits our ability to further examine relationships between retention strategies or themes and retention rates.

Table 3.

Number of strategies by theme

Study	Community Involvement	Study Identity	Study Personnel	Study Description	Contact and Scheduling	Reminders	Visit Characteristics	Benefits of Study	Financial Incentives	Reimbursement	Non-Financial Incentives	Special Tracking Methods	Total Number of Strategies Per Study	Total Number of Themes Per Study	Retention Rate (%)
Katz, 2001					2	1	1			1	2	1	8	6	59
Catlett, 1993			1					1		1			3	3	62
Pappas, 1998					3	2	1				1		7	4	70
Hessol, 2001	1		1		9	2	4	3	2	2	4	1	29	10	82
Parra-Medina, 2004		2	3		3	2	1		5	1	1		18	8	82
Motzer, 1997		1	3		6		2		1		4		17	6	83
Desland, 1991				1	5		1						7	3	84
Mazzuca, 2004				1	5			1	1		1		9	5	85
Dilworth-Anderson, 2004		1	7	9	6	2	9					1	35	7	87
BootsMiller, 1998			3		13	2	3		3		1		25	6	88
Dennis, 2000	4						1	1			3		9	4	88
Miller, 1999			3	1				3		1			8	4	88
Dudley, 1995	1				13	2	3	4			1	1	25	7	89
Bell, 1985			3		6	2	7	7		4			29	6	92
Meyers, 2003					13	3	2		1	2	1	1	23	7	92
Froelicher, 2003			7	3	13	1	5	2		4	2	5	42	9	93
Sprague, 2003				3	8		5					2	18	4	94
Robles, 1994			2	1	5		2	1	2	2	1		16	8	95
Goldberg, 2005		1	2	1	3		3						10	5	96
Bailey, 2004			2	1	2		3	1	1	2	1		13	8	99
Desmond, 1995			1	1	8		4		3				17	5	99
Number (%) of Studies with Theme	3 (14%)	4 (19%)	13 (62%)	10 (48%)	18 (86%)	10 (48%)	18 (86%)	10 (48%)	9 (43%)	10 (48%)	13 (62%)	7 (33%)
Total Number of Strategies Per Theme	6	5	38	22	123	19	57	24	19	20	23	12	368

DISCUSSION

Failure to retain study participants in a research study is an important methodological concern. Given the expense of conducing health care research, the limited financial resources to support such studies and the risks patients potentially incur from participation, efforts to reduce bias from loss to follow-up are an important research priority. Large overall loss to follow-up or differential loss to follow-up threatens the internal and external validity of studies and limits the ability to draw inferences. However, there is sparse evidence concerning strategies aimed at maximizing retention of study participants. We identified only 21 studies across all health domains and no study that explicitly evaluated retention strategies. We identified 368 strategies that were classified within 12 themes. The studies used a median of 17 strategies across a median of 6 themes. The most commonly reported strategies dealt with themes related to systematic methods of patient contact and scheduling procedures, and minimizing patient burden through modification of characteristics of the study visit or clinic.

Studies with lower retention rates (less than the mean rate of 86%) reported the use of fewer strategies than those studies with higher retention rates despite having a lower mean overall follow-up duration. The relatively small number of studies and their heterogeneity of the literature limited our ability to further quantitatively synthesize the results. Consequently, our best advice to investigators who are designing cohort retention protocols for studies with in-person follow-up is to use multiple strategies across multiple themes. A list of strategies abstracted from the 21 studies is available <insert hotlink>.

We are not the first to develop a list of strategies and themes. Davis et al (2002), in review of 21 community-based trials, identified 9 themes and noted that those studies with the highest retention rate appeared to use a combination of strategies[11]. Hunt and White (1998) chose to review four longitudinal studies to develop a list of general strategies to maximize retention [16]. Their four categories included some but not all of the same themes we identified: enrolment, consent and baseline activities; bonding; frequency of contact; staff characteristics; and incentives. Coday et al (2005) used principles of Social Cognitive Theory to identify 8 retention categories from the 61 strategies elicited from study staff[5]. Some of the retention themes are very similar to those identified in our review. However, we identified additional different sorts of themes, such as “Community involvement”, and also had narrower themes in some cases For instance, our theme of “Visit Characteristics” is very similar to the category “Be flexible” as well as the category “Give instrumental or tangible support” identified in the Coday study as both categories included offering more convenient visits, such as home visits. As another example, the data guided us to separate the strategies encompassing reimbursement, financial incentives, non-financial incentives which were combined by Coday et al. We identified similar strategies but we built on the findings of these earlier studies by systematically reviewing the literature across all health domains, incorporating all relevant study designs, to identify a larger and more comprehensive list of strategies and themes.

More research is needed in this field including explicit evaluation of the effectiveness of different cohort retention strategies. Given the need for all studies to retain participants, it may be more feasible and appropriate to focus on testing those retention strategies with higher costs, in terms of expense and research staff time. This research could include a comparison of the costs and benefits of different strategies. Finally, more papers on actual experiences with retention of participants in research studies should be published and study investigators should be encouraged to more explicitly report their retention strategies and retention rates. Adoption of standards for reporting retention strategies and rates would be helpful to improve the disclosure and consistency of these data.

Methods of recruiting participants also may be relevant for retaining participants. Consequently, research on recruitment strategies may be consulted. For instance, a recent Cochrane review of 15 controlled trials of strategies to improve the recruitment of study participants, categorized recruitment strategies into five types: (1) provision of information prior to invitation to join the study (pre-warning), (2) provision of extra information about study benefits and risks, (3) changes to the study design to account for patient preference, including not having a placebo study arm, (4) changes to the consent process and (5) incentives[54]. The authors noted that the results of a few studies suggest that financial incentives and provision of extra information are of some benefit. Similar to our review, the authors noted that the heterogeneity of the studies limited their ability to quantitatively synthesize the effect of specific strategies.

We acknowledge that our study has limitations. First, our efforts to classify retention strategies may have mis-classified some strategies. Second, given the significant heterogeneity of the studies we were not able to provide detailed quantitative analyses. This includes an inability to identify associations between the types of strategies, retention rate and the type of study population. Finally, we were able to abstract only the retention strategies as described in the reports of the studies and were unable to determine, for instance, the frequency or intensity of application of the strategies. Because of these limitations, we are unable to provide specific guidance on what works for whom or under what circumstances. Nevertheless, we have identified important relationships between the number of strategies used and retention rates.

Our study has some notable strengths. We built on earlier anecdotal examinations of retention strategies by systematically seeking and synthesizing studies addressing retention strategies. In addition, by classifying retention efforts into strategies and themes, and in identifying an association between number of strategies and retention rates, we can provide researchers with a guideline to plan cohort retention efforts in health care research studies.

Loss of study participants may threaten the power of a study and lead to bias. This common and important threat to validity has received very limited attention in the research literature. Use of a greater number of retention strategies from a wide variety of themes may improve study participant retention. However, further research is needed, including the explicit evaluation of the effectiveness of different strategies.

Appendix 1 - Search Strategies

PubMed

((attrition[tiab] OR retention[tiab] OR patient dropouts[mh] OR “loss to follow-up”[tiab]) AND (minimize[tiab] OR strateg*[tiab] OR procedure*[tiab] OR technique*[tiab] OR method[tiab]) AND (longitudinal[tiab] OR follow-up[tiab] OR cohort studies[mh] or trial*[tiab]) AND eng[la] NOT review[pt] NOT (animals[mh] NOT humans[mh]))

CINAHL

[(AB ( attrition OR retention OR “loss to follow-up” ) Or TI ( attrition OR retention OR “loss to follow-up” ) OR MH patient dropouts) AND (TI ( strateg* OR minimize OR procedure* OR technique* OR method ) Or AB ( strateg* OR minimize OR procedure* OR technique* OR method )) AND (TI ( longitudinal OR follow-up OR trial* ) Or AB ( longitudinal OR follow-up OR trial* ) OR MH prospective studies)]

The Cochrane Library (CENTRAL, Cochrane Methodology Database)

• #1 attrition OR retention OR “loss to follow-up” in Record Title or attrition OR retention OR “loss to follow-up” in Abstract or patient dropouts in Keywords in CENTRAL and CMR

• #2 minimize OR strateg* OR procedure* OR technique* OR method in Record Title or minimize OR strateg* OR procedure* OR technique* OR method in Abstract in CENTRAL and CMR

• #3 longitudinal OR follow-up OR trial* in Record Title or longitudinal OR follow-up OR trial* in Abstract or cohort studies in Keywords in CENTRAL and CMR

• #4 #1 AND #2 AND #3

EMBASE

• #1 attrition:ti,ab OR retention:ti,ab AND [english]/lim AND [humans]/lim

• #2 (minimize:ti,ab OR strateg*:ti,ab OR procedure*:ti,ab OR technique*:ti,ab OR method:ti,ab) AND [english]/lim AND [humans]/lim

• #3 (longitudinal:ti,ab OR ‘follow-up’:ti,ab OR ‘cohort analysis’/exp OR trial*:ti,ab) AND [english]/lim AND [humans]/lim

• #4 #1 AND #2 AND #3