Recruitment Strategies for Nurse Enrollment in an Online Study

Abstract

Background:

While there is a great deal of literature regarding effective recruitment and challenges of recruiting specific patient populations, there is less known about best practices for recruitment of nurses as study subjects.

Objectives:

The purpose of this paper is to report our experience with recruitment and retention for a randomized trial of an online educational program to prepare oncology nurses to discuss oncology clinical trials with patients.

Methods:

The study population included currently employed oncology nurses with direct patient interaction. There were three phases of this study: (1) qualitative interviews; (2) a pilot test; and (3) the randomized trial. Phase 3 was rolled out in five waves of recruitment. The distinct phases of the study—and the gradual roll out of recruitment during Phase 3—allowed us to test and refine our recruitment and retention methods for the randomized trial. Upon analysis of our response rate and attrition after the first wave of recruitment in Phase 3, we made several changes to improve recruitment and retention, including adding incentives, shortening the survey, and increasing the number of reminders to complete the program.

Results:

The response rate was higher when we used both email and U.S. postal mail solicitations. After the first wave of recruitment in the final phase, changes in our strategies did not increase our overall response rate significantly; however, the rate of attrition following baseline declined.

Discussion:

Recruitment planning is an important component of successful clinical research. The use of the Internet for both recruitment of subjects and testing of interventions remains a cost-effective and potentially high yield methodology. Our research demonstrated several successful approaches to yield increased participation and retention of subjects, including seeking formal relationships with professional organizations as sponsors or supporters, providing meaningful incentives to participants, keeping surveys or questionnaires as short as possible, and planning multiple follow-up contacts from the outset.

Recruiting research participants using Internet-based methods has been shown to be effective at reaching larger and more diverse audiences (Becker et al., 2014; Lane, Armin, & Gordon, 2015; Morgan, Jorm, & Mackinnon, 2013: Seltzer, Stolley, Mensah, & Sharp, 2014). However, recent reports identify several concerns, including higher attrition rates, sample bias, and low response rates. While there is a great deal of literature regarding recruitment of specific patient populations (Becker et al., 2014; Ferwerda et al., 2013; Lane et al., 2015), less is known about best practices for using Internet recruitment of nurses. There has been some study of barriers to nurse participation in research (Jacobson, Warner, Fleming, & Schmidt, 2008; Roxburgh, 2006), but the reports have not been specific to Internet research. We focus here on our experience with recruitment for Improving Communication with Patients about Clinical Trials (IMPACT), a randomized trial of an online educational program focused on preparing nurses to engage patients in discussions of oncology clinical trials.

Oncology nurses play a critical role in supporting patients through the process of treatment decision-making. They share a therapeutic relationship with their patients and are estimated to have twice as much contact with patients as their physician counterparts (Fasola, Aprile, & Aita, 2012; Kiteley & Vaitekunas, 2006). However, to our knowledge, there has not been a scientifically rigorous effort to increase and improve oncology nurse involvement in discussions of clinical trials with patients as part of routine care. The purpose of IMPACT was to assess and address barriers nurses may have in discussing clinical trials with patients.

A national sample of Oncology Nursing Society (ONS) members was used for each of three phases of the study:

1. qualitative interviews to identify barriers and knowledge gaps in order to aid in the design of the educational program;

2. a pilot test of the program and the website; and

3. the randomized trial phase of the study, comparing tailored video education with text material with a target n of 850 nurse participants.

The first two phases of this study, and the gradual roll out of recruitment during the third phase, allowed us to test and refine our recruitment and retention methods for the randomized trial. This paper outlines the different approaches used, how differing techniques varied in their success, and lessons learned.

Methods

Study Population

Eligibility criteria were the same for all participants in all phases of the study. Samples were drawn randomly from ONS membership. To facilitate this, we engaged an ONS administrative officer as a consultant on the project and obtained permission to access their membership through the organization’s usual procedures. To be eligible, subjects had to be currently employed and have direct interaction with patients. Table 1 summarizes the demographics of the participants who completed Phase 3 of the study

Table 1.

Phase 3 Demographics

		Control	Video	Combined
Gender	Male	30 (3.05%)	22 (2.24%)	52 (2.65%)
	Female	953 (96.95%)	959 (97.76%)	1912 (97.35%)
Age	Median	47	48	48
Race	White	855 (87.07%)	852 (87.03%)	1707 (87.05%)
	Other	127 (12.93%)	127 (12.97%)	254 (12.95%)
Hispanic or Latino?	Yes	31 (3.16%)	36 (3.67%)	67 (3.42%)
	No	950 (96.84%)	944 (96.33%)	1894 (96.58%)
Nursing Education	Less than Bachelors	200 (20.35%)	192 (19.59%)	392 (19.97%)
	Bachelors	527 (53.61%)	533 (54.39%)	1060 (54.00%)
	Masters or higher	256 (26.04%)	255 (26.02%)	511 (26.03%)
Years’ Experience (median)	Nursing	18	20	19
	Oncology Nursing	12	13	13

Study Design and Recruitment Methods

The institutional review board of an academic medical center in the United States approved this study. The randomized trial was based on the Theory of Planned Behavior (Azjen, 1991), measuring changes in knowledge, attitudes, perceived behavioral control, and subjective norms, with the intention to discuss trials as the primary outcome. The video intervention arm of the trial consisted of 13–17 two-minute videos, tailored to the individual according to his/her responses on the baseline survey. The control arm consisted of reading online text adapted from National Cancer Institute material that addressed the same topics as the videos. Table 2 summarizes the recruitment approach and incentives offered.

Table 2.

Recruitment methods and response rates for each phase of study

Phase		Target n	# of Invites	Recruitment Method	Incentive	# of Responses	Response Rate
1 Qualitative phone interviews		32	302	Written letter (US postal mail), followed 10 days later by individual email solicitation sent by the Oncology Nursing Society (ONS)	$50 Amazon gift card.	50	16.6%
2 Pilot of web site, intervention, and baseline survey		150–200	2000	Individual email solicitation only, sent by ONS	$40 Amazon gift card and entry in a drawing for an iPad	125	6.3%
3 Randomized trial	Wave 1	1030	2998	Written letter (U.S. postal mail), sent in 5 batches, followed by individual email, and subsequent repeat emails; full page ad in ONS clinical news journal; Facebook and Twitter posts; presentation at national conference; and ads in minority newsletters.	CE credit (2 contact hours) and entry in a drawing for an iPad, and (for batches 2–5) entry into a $250 Amazon gift card drawing.	382	12.8%
	Wave 2		2998			333	11.1%
	Wave 3		6117			839	13.7%
	Wave 4		4303			557	12.9%
	Wave 5		4306			432	10.0%
	Total		20722			2650	12.8%

The first of three phases consisted of qualitative interviews via phone with a small sample to identify barriers that oncology nurses perceived regarding discussions with cancer patients about clinical trials. Results were used to refine the instruments (Flocke et al., 2017). Phase 2 was a pilot test of the website and surveys. Phase 3 was the implementation of the randomized trial. All participants in Phases 2 and 3 completed baseline surveys measuring knowledge, attitudes, and beliefs about clinical trials.

For the qualitative interviews, we sent solicitation letters via U.S. postal mail followed by an email solicitations. A $50 Amazon gift card was offered as an incentive to participate. For the pilot, we sent an email solicitation only and offered a $40 Amazon gift card and entry into a drawing for an iPad as incentive. For the randomized trial we again sent both mailed solicitation letters and email solicitations. We also supplemented the mail and email solicitations with outreach and advertising, including a presentation at the 2017 ONS Annual Congress and a full-page advertisement in the ONS news magazine, ONS Voice. Initial incentives for this phase included two continuing education (CE) credits and entry into a drawing for an iPad.

Those who agreed to participate in the randomized controlled trial and were deemed eligible were randomly assigned to receive education about clinical trials via the tailored educational videos (experimental arm) or via text about clinical trials (control arm). Randomization was done by computer, stratified by race and practice setting (inpatient vs. outpatient and academic vs. community). After completion of the intervention, participants completed postintervention surveys—identical to the baseline survey—and an evaluation of the program, and then, three months later, a shorter follow-up survey.

Recruitment and Retention Results

For the first qualitative phase of the study, we sent 302 letters and emails, and received 50 responses (16.6%). Of those who responded, 35 were eligible to participate, and ultimately 33 interviews were completed (Flocke et al., 2017). For the second (pilot) phase of the study, with email-only solicitation, we received only 125 (6.3%) responses to the 2,000 email solicitations; 124 of those consented to participate. This was true despite increasing the incentives to a $40 Amazon gift card and entry into a drawing for an iPad.

Recruitment outcomes from the first two phases of the project demonstrated the need for an increased intensity of approach. Given the significantly larger response to the U.S. postal mail invitations followed by email for Phase 1, as compared to email only for Phase 2, we employed both U.S. postal mail and email for Phase 3. Invitation letters, with handwritten names and addresses, were sent in five waves. The first wave of letters and emails (n = 2998) resulted in 382 registrations (12.7% response rate). Of those, 353 consented, 265 were deemed eligible, and 241 (yield of 8.0% of invitations) completed the baseline survey. Of baseline completers, only 160 completed the intervention, for an attrition rate of 33.6%. Another 11.9% were lost to follow-up between the intervention and the postintervention survey.

After baseline completion of the first wave of recruitment was complete, an evaluation of recruitment and retention was implemented to determine what improvements should be made to the program and our methods to increase the response rate and decrease attrition. Time to complete surveys and the intervention were evaluated, as well how many sessions’ participants took to complete all components, characteristics of those who finished compared to those who did not. Finally, user comments from the evaluation survey were reviewed.

Examining all data regarding responses and dropouts, there were no obvious patterns that had implications for overall methods. While most participants who completed the evaluation questions expressed satisfaction with the intervention itself, many indicated a dissatisfaction with the survey length and redundancy. However, on review of the surveys, only one question was identified that lacked adequate correlation with others and was then dropped. Additional feedback from participants indicated that many would have liked to know about the time commitment to complete the program and the possibility of an audio/visual portion.

Based on our findings, we made the following changes:

• Removed one question from the surveys.

• Increased the number of incentives by adding 10 $250 Amazon gift card drawings to the existing incentives of five iPad drawings and two CEs for completion of follow-up.

• Updated the invitation letter and welcome page of the website to clarify time commitments and possibility of an audio/visual portion, and to outline incentives.

• Increased reminder emails from two to three.

• Increased time to finish intervention after baseline survey completion from 30 days to 90 days.

• Added another reminder to complete each part of the program.

• Added a reminder about the $250 drawing after baseline completion.

After these changes, we mailed a total of 17,724 letters in the subsequent four waves, and those resulted in 2,161 registrations, for a response rate of 12.2%. The response rate for the second and fifth waves were lower than the first, at 11.1% and 10.0%, respectively, while the third and fourth waves had a higher response rate at 13.7% and 12.9%, respectively, resulting in an overall response rate for all five waves of 12.8%. The number of baseline surveys yielded from the second through fifth waves was 1,648 (9.3%), resulting in a final yield of 1964 (9.5%).

While the response and baseline completion rates did not differ significantly from the first wave of letters, attrition following the baseline survey decreased from 33.6% to just 21.1% (T = −3.001234, p < 0.01), and attrition following the intervention fell from 11.9% to 5.6% (T = −2.082262, p = 0.02) for those in the second wave. For subsequent mailings, these numbers remained stable. The final overall attrition rate after the baseline survey was 24.2%; after the intervention was 6.4%.

Minority Recruitment

Nearing the end of the recruitment period, we made a final push to increase minority recruitment by targeting minority oncology nurse organizations. An advertisement was published in seven National Association of Hispanic Nurses (NAHN) weekly e-newsletters, and one National Black Nurses Association (NBNA) e-blast. This had minimal effect, yielding only an additional four minority participants. The overall minority recruitment rate was 13%, while the percentage of minorities in the ONS membership at that time was 14.8% African American and 10.7% other minorities.

Additional Participant Feedback

Among respondents (n = 1,758), an overwhelming majority (95%) reported that they heard about the study via the letter and email solicitations, rather than advertising in journals or at the annual conference, or from a friend. Among those who replied to the noncomplete survey (n = 42), most indicated that they did not complete the program because they did not have time (48%) or forgot (26%).

Discussion

We learned several lessons during recruiting and retaining a large national sample for an online educational program. Despite this being the age of digital communication, it is clear that recruiting a large sample requires the use of a variety of strategies. Our response rate was much better in the first phase when we mailed letters and emailed invitations than in the second phase when we relied solely on email, and then improved again in the last phase when we added individual hand-addressed letters. While recruitment via Internet is an attractive option because of low cost and ability to reach large numbers of potential participants, numerous limitations have been identified. These include the now common use of email filters that can classify survey invitations as spam, email overload resulting in deleting of what are seen as nonrelevant messages, and the use of multiple email addresses by an individual (Sheehan, 2001; Whitehead, 2007).

Research reports have varied widely in terms of sample characteristics as well as recruitment strategies and results. For example, Leece and colleagues (2004) compared response rates among surgeons between an Internet versus a paper version. Nonresponders received three follow-up requests. The paper group had a 58% response rate while the Internet group had a 45% response rate. In contrast, McRobert, Hill, Smale, Hay, and van der Windt (2018) reported significantly lower yield from an international survey of surgeons, internists, and physical therapists in Europe and Great Britain. Using multiple professional websites and social media, their survey was accessed by 2,326 persons but complete data obtained from only 387 (16.6%).

Although not specific to Internet recruitment nor nurses as subjects, a recent Cochrane review found the use of reminders to be effective in aiding recruitment (Treweek et al., 2013). Our experience of needing to supplement the original Internet-only recruitment and add additional invitations and incentives is similar to that described by Im and colleagues (2006). In one of the few reports of Internet research with nurse participants, Im et al. (2006) initially used postings on professional nursing organization websites. They progressively expanded approaches, eventually using individual emails to members of professional groups as well as snowball sampling and searching schools of nursing directories in order to reach a target sample of 100. In addition to others’ recommendations, they identify the importance of having a formal supporting relationship with the professional organization.

We also learned that participants need close repeated follow-up to keep them engaged. Additional incentives for completing specific portions of the program, as well as consistent reminders to complete the program and the incentive, seemed to increase engagement as measured by completion rate before and after the implemented changes. Lane et al. (2015) noted this challenge of retaining subjects for longitudinal studies, and Sheehan (2001) found that the number of follow-up contacts also was the most influential in boosting response rates.

Participants expressed dissatisfaction with the length of the survey, and this burden was likely the largest contributor to attrition; 38% who responded to the questionnaire about why they did not complete the program indicated that they did not have time. However, although the survey was not significantly shortened after the evaluation, attrition fell for subsequent mailings, likely due in part to revising the introductory material to clarify the time commitment.

Finally, identifying more creative and aggressive strategies for minority participation were needed. Further consideration needs to be given as to why minorities were more likely to drop out, and perhaps consultation with experts in the field of cultural sensitivity would be advantageous during the design phase of an online educational program. Sample bias is a recognized risk of Internet recruitment, both of patients and professionals (Im & Chee, 2004; Leach, Butterworth, Poyser, Battersham, & Farrer, 2017).

Conclusion

Use of the Internet for both recruitment of subjects and testing of interventions remains a cost-effective and potentially high yield methodology. Though, careful attention to approaches that have been found to result in increased participation and retention of subjects should be planned from the outset. These include seeking formal relationships with professional organizations such as sponsors or supporters, providing meaningful incentives to participants, keeping surveys as short as possible, and planning multiple follow-up contacts. Although supplementing email invitations with use of mailed letters is expensive and time-consuming, it can add substantially to recruitment success.