Use of Online Recruitment Strategies in a Randomized Trial of Cancer Survivors

Abstract

Background/Aims

Despite widespread Internet adoption, online advertising remains an underutilized tool to recruit participants into clinical trials. Whether online advertising is a cost-effective method to enroll participants compared to other traditional forms of recruitment is not known.

Methods

Recruitment for the SPIRIT trial, a community-based study of cancer survivors, was conducted from June 2015 through December 2016 via in-person community fairs, advertisements in periodicals, and direct postal mailings. In addition, “Right Column” banner ads were purchased from Facebook to direct participants to the SPIRIT website. Response rates, costs of traditional and online advertisements, and demographic data were determined and compared across different online and traditional recruitment strategies. Micro-trials optimizing features of online advertisements were also explored.

Results

Of the 406 respondents to our overall outreach efforts, 6% (24 of 406) were referred from online advertising. Facebook advertisements were shown over 3 million times (impressions) to 124,476 people, which resulted in 4,401 clicks on our advertisement. Of these, 24 people ultimately contacted study staff, 6 underwent pre-screening, and 4 enrolled in the study. The cost of online advertising per enrollee was $794 when targeting a general population versus $1426 when accounting for strategies that specifically targeted African Americans or men. By contrast, community fairs, direct mail, or periodicals cost $917, $799, or $436 per enrollee, respectively. Utilization of micro-trials to assess online ads identified subtleties (e.g. use of an advertisement title) that substantially impacted viewer interest in our trial.

Conclusions

Online advertisements effectively directed a relevant population to our website, which resulted in new enrollees in the SPIRIT trial at a cost comparable to traditional methods. Costs were substantially greater with online recruitment when targeting under-represented populations, however. Additional research using online micro-trial tools is needed to evaluate means of more precise recruitment to improve yields in under-represented groups. Potential gains from faster recruitment speed remain to be determined.

Background/aims

Since their introduction in October 1948,¹ clinical trials continue to be the gold standard for providing high quality answers to clinical research questions.² One of the primary challenges and costs to conducting clinical trials is recruitment. It has been estimated that less than half of clinical trials meet recruitment goals without extending the length of the trial.^3–7 This often results in underpowered studies, inconclusive results, and trial failure.^8–12 More alarmingly, non-significant findings due to ineffective recruitment may increase the risk that evaluation of potentially effective interventions would end prematurely.¹³

The Internet is a promising means of recruiting participants, representing an effective tool for disseminating information¹⁴ that has been rapidly adopted by all age groups. In the United States, Internet use is now widespread, e.g. 99% among 18–29 year-olds and 64% among adults ages 65 years and greater.¹⁵ Initial experiences with websites as a means of recruiting participants into studies have been positive in terms of speed and efficacy of recruitment.^16–19 Furthermore, online recruitment efforts have been found in select studies to be more cost-effective than traditional strategies,²⁰ able to reach diverse ethnic groups,²¹ and can even be used as a randomization tool.²² This is particularly relevant for cancer trials which have traditionally low representation of ethnically/racially diverse groups.²³ Despite these reports and broad adoption of web-based marketing approaches by industry, effectiveness research of different online recruitment strategies has not been performed.

In this context, we studied the recruitment experience of the Survivorship Promotion In Reducing IGF-1 Trial (SPIRIT) to determine: (1) whether online advertisements could direct a targeted audience to the trial’s website; (2) the relative costs of online advertising versus traditional modes of clinical trial recruitment (e.g. newspapers, direct mail); and (3) the role of micro-trials to optimize recruitment participation. We hypothesized that online ads would yield trial enrollees at a comparable cost to traditional methods and that micro-trials would afford an opportunity to improve recruitment among groups with traditionally lower participation rates in clinical trials.

Methods

Study design

SPIRIT was a 3-arm parallel, randomized controlled trial funded by the Maryland Cigarette Restitution Fund. Its goal was to examine changes in insulin growth factor-1, a biomarker for cancer, among cancer survivors randomized to either a behavioral weight loss intervention, metformin, or a self-directed control. Details of the trial may be found in Supplement Table S1. Behavioral weight loss involved coach calls, initially weekly and then monthly, web utilization, and self-weighing at home. The metformin arm entailed daily compliance with metformin, a pill diary, and calls from study staff (initially weekly, then later monthly). The study duration was 1 year. In addition to their intervention requirements, participants were asked to participate in 4 study visits, which involved questionnaires, phlebotomy, stool and urine collection, as well as anthropomorphic assessments (height, weight, blood pressure). Incentives for participation included small monetary allocations at each data collection visit, gifts, and lab or physical exam findings (e.g. blood pressure). Enrollment and execution of the SPIRIT protocol (baseline and follow-up visits) took place at the Johns Hopkins ProHealth ambulatory research clinic in Woodlawn (West Baltimore), Maryland.

Recruitment population

Recruitment for SPIRIT occurred between June 2015 and December 2016. The recruitment target was 120 participants in the Baltimore metropolitan area with the goal that 50% would be African American and 50% female. Recruitment targeted participants age 18 or older, who were in remission from a prior malignancy with no active cancer treatment within the last 3 months. In addition, participants needed to be overweight (body mass index ≥25 kg/m²) with regular access to a computer with internet connection. Adults with medication-treated diabetes, an elevated blood sugar (BG ≥200 mg/dL or A1c ≥7%), or active enrollment in a weight loss program were excluded. Additional details regarding the trial’s inclusion and exclusion criteria are also found in Supplement Table S1.

Traditional recruitment strategies

The SPIRIT trial simultaneously implemented several recruitment strategies that were approved by the Johns Hopkins Institutional Review Board (IRB) (for details see Supplement Material SM1). Investigators adjusted recruitment strategies throughout the course of the trial to meet recruitment goals. For example, staff chose to target certain zip codes for mailings in order to increase the number of African American enrollees. Approved strategies included: (1) targeted mailings of invitational materials (letter and/or brochure) to lists of cancer survivors; (2) placement of brochures in physician offices in Hopkins-affiliated clinics; (3) distribution of flyers in various community settings (e.g. health fairs); (4) direct referral from study physicians; (5) word-of-mouth; and (6) advertisement in local periodicals. With regards to targeted mailings, in addition to contact information of prior study participants who requested to be contacted about new studies, new addresses were purchased from commercial vendors, who mailed study brochures to these individuals on behalf of the investigative team. The investigative team did not receive names or addresses of the people who are on the vendor’s mailing lists. Periodical advertisements were about half a page in terms of space and were limited to local newspapers in communities surrounding the ProHealth research center.

Online recruitment strategies

Investigators also received IRB approval for several online recruitment strategies about half-way through the recruitment process. A public website (domain name: spirittrial.org) was created using an online web-hosting service (Weebly, San Francisco, CA) and managed by the study team. The website was launched on September 5^th, 2015 and provided information about the trial (principal investigator, clinicaltrials.gov registration number, IRB number, contact information, and study clinic), funding source, participant eligibility criteria, instructions on how to get to the study center, and an embedded, secure form enabling interested visitors to request prescreening by the study staff (Qualtrics, Provo, UT). Text for the website was adapted from IRB-approved printed materials. The website used a responsive design, which renders an optimized page for both desktop and mobile devices. Anonymous aggregate visitor data (volume, demographic information, geographic location, and referral source) were collected by Google Analytics (Alphabet Inc, Mountain View, CA).

In addition, we received IRB approval to conduct online advertising with paid banner advertisements via Facebook (Menlo Park, CA) to attract users to the SPIRIT website. These advertisements ran from February 12^th through April 28^th and July 19^th through August 3^rd, 2016. Facebook recorded the number of times an advertisement was displayed on a user’s web browser (“impressions”), the number of unique Facebook accounts with an advertisement displayed at least once (a surrogate for individual people also called “reach”), the number of clicks on each advertisement, and the cost per click for each advertisement (total ad cost divided by clicks). Cost per click is a surrogate for advertisement performance; a lower cost per click represents greater advertisement traffic.

Facebook advertisements were designed with a goal to direct traffic to the trial’s website and were positioned on the “Desktop Right Column,” a static image to the side of a user’s newsfeed that stays in the screen (even despite scrolling) for the duration of the session. Each advertisement included one image, a “headline” title of less than 25 characters, and a brief message of less than 70 characters. The audience was selected for English language users, age ≥18 years, with select Facebook interests, which were chosen from a list of available prompts provided by Facebook (Supplement Table SM2). We employed two campaign strategies during the SPIRIT trial: (1) a general campaign, which included Facebook users within the geographic areas 20 miles around the ProHealth Clinical Research Center in Baltimore, Maryland, and 30 miles around Frederick, Maryland; and (2) a targeted campaign, restricted to specific zip codes or to men only (described below). Results from Facebook campaigns were reported for the general strategy, targeted strategy, and overall.

Facebook micro-trials

Employing a proprietary algorithm, Facebook offers the possibility to optimize ad objectives through micro-trials. We evaluated 4 strategies using this tool: targeted demographic characteristics (race/sex), alternative advertisement logo designs, alternative advertisement titles, and alternative advertisement texts. With regards to targeted demographics, we deployed advertisements (a) to zip code with a greater population of African Americans in the Baltimore area (21205, 21207, 21213, 21215, 21216, 21217, 21218, 21244, 21133), (b) to a subset of two of these zip codes (21215 & 21133) that based on staff experience had high rates of trial participation by African Americans, and (c) to male web-users only.

The evaluated logos included the Johns Hopkins Medicine logo, the SPIRIT trial logo, a combined logo (Hopkins plus SPIRIT logos), a staff group photo combined with the Johns Hopkins Medicine logo, and a staff group photo combined with the SPIRIT trial logo. The two compared titles were “Weight loss & Cancer” and “Cancer Survivor Trial.” The three evaluated text versions were: “New study for overweight, cancer survivors at Johns Hopkins ProHealth. Learn more!,” “Join SPIRIT, a new trial for overweight, cancer survivors at Johns Hopkins off I-695!,” or “SPIRIT is a weight loss trial for cancer survivors at Hopkins. Click here to learn more!”

Staff contact

Interested adults could reply to the study via phone, mailer card, email, or online. All inquiries for additional information and source of inquiry (periodical, mailed brochure, website, email) were tracked by staff in an encrypted database. Staff would then contact interested adults via telephone. After providing verbal consent, a pre-screen telephone interview was conducted over the phone. Eligible adults who remained interested were invited to attend a screening visit at the ProHealth research clinic. During the in-person screening visit, written informed consent was obtained, along with other baseline measurements and demographic information such as age, sex, race, income (<$50,000, $50,000 to 75,000, $75,000 to 100,000, and $100,000+), education level (some high school or less, high school or GED, some college, college graduate), and source of referral (Facebook, community fairs, mail/brochure, periodicals, and other). These data were maintained for those who prescreened as well as those who later enrolled in the study.

Cost analysis

All contacts and means of contact were maintained by study staff in a secure database. Persons who were prescreened and enrolled in the study were followed by the study data manager, which tracked demographic information and responses. Costs for each advertising modality were derived from Hopkins financial reports of the trial’s expenses. The cost of fairs was based on booth registration fees and did not include staff time at the fairs. The cost of periodicals was based on the aggregate cost of each periodical’s fees. Direct mailing costs were estimated using known rates for address purchases, printing, mailing, and postage reply costs. Responses and costs were aggregated by type (Facebook, community fairs, mail/brochure, periodicals, and other).

Results

Overall trial recruitment and website visitation metrics

A total of 406 adults responded to our outreach efforts (Figure 1); 6% (24 of 406) were referred from online advertisements. Initiation of online advertisements corresponded with higher webpage visitation (Figure 2, panel A) with 38% of website referrals coming from Facebook or Facebook related applications (Figure 2, panel B). The majority of online visitors had Internet protocol (IP) addresses associated with Baltimore City or Baltimore County (Figure 2, panel C).

Figure 1.

Recruitment funnel for SPIRIT trial demonstrating number of recruited participants by source.

Figure 2.

Google analytics metrics demonstrating (A) number of webpage sessions between February 1, 2016 and August 31, 2016, (B) source of webpage referrals (direct, Facebook, Google search, Hopkins Medicine webpage referral, or other), and (C) geographic density of referrals. Arrow designates the location of ProHealth, the clinical research site of the SPIRIT study.

Facebook performance

The general Facebook strategy was shown over 1.5 million times to over 88 thousand people; $2,383 was spent on this strategy which resulted in 2,618 clicks to our webpage at a rate of $0.91/click (Table 1). Strategies targeting select zip codes with a greater population of African Americans cost more per click at $1.73/click (9 zip codes) and $2.72/click (2 zip codes), respectively. Similarly, a campaign strategy targeting men only cost more than the general strategy at $1.72/click.

Table 1.

Comparison of Facebook campaign strategy results with respect to website traffic

Facebook Strategy	Number of Ads Shown	Unique People Reached	Average Times Shown Per Person	Clicks	Cost ($)	Cost per click ($)
General Campaigna	1,501,603	88,520	17	2,618	2,382.92	0.91
Zip codes with a greater population of African Americansb	762,021	24,457	31	901	1,559.90	1.73
Select zip codes with a greater number of African Americans: 21215 & 21133	252,281	6,615	38	241	655.59	2.72
Men only	534,559	29,784	18	641	1,105.34	1.72
Total	3,050,464	124,476c	25	4,401	5,703.75	1.30

^aThe general campaign was shown to both men and women and did not target specific zip codes. In contrast, two of our campaigns were restricted to zip codes with a greater population of African Americans, and one campaign targeted men alone.

^bThese zip codes were: 21205, 21207, 21213, 21215, 21216, 21217, 21218, 21244, 21133

^cNote some people were shown ads from different campaigns, such that summing the unique people reached is not equal to the total number of unique people reached

The general strategy resulted in 16 contacts with 5 undergoing prescreen and 3 enrolling in the SPIRIT trial at a cost of $794 per enrollee (Table 2). Strategies restricted to predominantly African American zip codes yielded 7 contacts, but no screenees or enrollees. A strategy targeting men alone yielded only 1 contact, who ultimately prescreened and enrolled at a cost of $1,105.

Table 2.

Comparison of Facebook strategy results with respect to trial enrollment

Strategy	Contacts	Prescreened	Enrollees	Cost ($)	Cost per contact ($)	Cost per screenee ($)	Cost per enrollee ($)
General Campaigna	16	5	3	2,383	149	477	794
Zip codes with a greater population of African Americansb	7	0	0	1,560	223	–	–
Select zip codes with a greater number of African Americans: 21215 & 21133	0	0	0	656	–	–	–
Men only	1	1	1	1,105	1,105	1,105	1,105
Total	24	6	4	5,704	238	951	1,426

^aThe general campaign was shown to both men and women and did not target specific zip codes. In contrast, two of our campaigns were restricted to zip codes with a greater population of African Americans, and one campaign targeted men alone.

^bThese zip codes were: 21205, 21207, 21213, 21215, 21216, 21217, 21218, 21244, 21133

Comparison of recruitment modality

Overall, Facebook resulted in 24 contacts, 9 screenees, and 4 enrollees. It was the most expensive modality at a cost of $1,426 per enrollee (Table 3). However, the majority of enrollees came from the general strategy that did not target African American zip codes or men alone. The cost of this general Facebook strategy (excluding strategies that targeted African Americans or men) was lower at $794 per enrollee. Fairs resulted in 10 contacts, 3 screenees, and 3 enrollees at a rate of $917 per enrollee. Mailed brochures, which represented the main recruitment strategy, was associated with 220 contacts, 81 screenees, and 65 enrollees at a rate of $799 per enrollee. Note that mailers to known contacts yielded 43 contacts at a cost of $430 ($10/enrollee). The cheapest aggregate modality was periodicals, which resulted in 84 contacts, 34 screenees, and 25 enrollees at a rate of $436/enrollee.

Table 3.

Comparison of other advertising modalities

Source	Contactsa	Screened (SV1)	Enrolled	Yield (Screened/Contact), %	Yield (Enrolled/Screened), %	Cost ($)	$ per Contact	$ per Screenee	$ per Enrollee
Facebook (overall)	24	6	4	25	67	5,704	238	951	1426
Facebook (general)	16	5	3	31	60	2,383	149	477	794
Mailed brochure	220b	81	65	37	80	51,950	236	641	799
Periodicals	84	34	25	40	74	10,906	130	321	436
Fairs	10	3	3	30	100	2,750	275	917	917

^aNote that these numbers do not total to 406 as interested adults learning about the study from other means (word of mouth, clinics, etc) are not represented in this table.

^bThere were 222 people who heard about the study via brochures. Two of these people heard about the study via brochures left in ambulatory clinics as opposed to direct mail.

Demographic characteristics of participants who completed the screening visit or were enrolled by recruitment modality may be seen in Supplement Table S2. Notably, the average age of Facebook screenees was 65 years, 83% were women, 0% were African American, income level was predominantly less than $75,000, and education level was high school graduate/GED or greater.

Online advertisement optimization

We utilized Facebook’s randomization tool to compare different features of our online advertisements to optimize yields (Table 4). While comparison of the Johns Hopkins Medicine logo and the SPIRIT trial logo were equivocal, a small change in title from “Weight Loss & Cancer” to “Cancer Survivor Trial” increased efficacy and resulted in lower costs per click, from $5.96/click to $1.76/click. Minor changes in text had little impact on click rates.

Table 4.

Micro-Trial Results Comparing Facebook Advertisement Logos, Titles, and Text

Feature	Impressions	Unique People Reached	Frequency	Clicks	Cost ($)	Cost per click ($)
Logo
Hopkins only – Total	1,312,088	117,612	11.2	2,032	2,653.93	1.31
Spirit only – Total	908,761	89,699	10.1	1,321	1,551.98	1.17
Both logos – Total	435,034	58,451	7.4	534	881.19	1.65
Group/Hopkinsa	51,460	8,790	5.9	49	75.22	1.54
Group/SPIRITa	137,171	14,964	9.2	152	233.95	1.54
Titleb
Weight loss & Cancer ‖ New study for overweight, cancer survivors at Johns Hopkins ProHealth. Learn more!	32,006	8,071	4.0	49	292.12	5.96
Cancer survivor trial ‖ New study for overweight, cancer survivors at Johns Hopkins ProHealth. Learn more!	754,834	75,782	10.0	833	1,468.81	1.76
General campaign textc
Cancer survivor trial ‖ New study for overweight, cancer survivors at Johns Hopkins ProHealth. Learn more!	579,970	62,608	9.3	981	871.92	0.89
Cancer survivor trial ‖ Join SPIRIT, a new trial for overweight, cancer survivors at Johns Hopkins off I-695!	202,865	41,323	4.9	309	302.78	0.98
Cancer survivor trial ‖ S PIRIT is a weight loss trial for cancer survivors at Hopkins. Click here to learn more!	1,803	1,179	1.5	4	4.70	1.18

^aPerformed in men only campaign

^bPerformed in the 21215 & 21133 campaign only and men only campaigns.

^cPerformed in general campaign only

Discussion

In this trial that enrolled cancer survivors, we found that online advertisements through Facebook were efficacious in directing the study’s target population to our website, which resulted in both screenees and enrollees in our clinical trial. Although targeted Facebook advertisements were not effective at recruiting African Americans or men into the trial, overall it did successfully reach eligible participants at a cost comparable to direct mail and cheaper than in-person fairs. With online tools affording an opportunity to employ micro-trials to evaluate a range of recruitment strategies, online advertising represents a means of objectively-based revisions to recruitment materials that can potentially improve yields over time. Online recruitment may also enable more agile recruitment by reducing the times between the definition of the recruitment strategy and the implementation of recruitment efforts, and that between the latter and actual enrollment. Such improved agility may be of importance to some researchers; although, their magnitude remains to be assessed.

Internet utilization has been on the rise across demographic groups.¹⁵ While there is well-known value to online advertising for commercial purposes, evidence in support of online recruitment for clinical studies is still sparse.^24,25 Our study adds to existing evidence that online recruitment efforts can be a cost-effective recruitment tool for targeting potential participants meeting several important geographic, demographic, and medical conditions of a clinical study. However, it should be noted that the absolute numbers achieved via online recruitment in our trial were low (only 6% of total trial enrollees). While this may reflect late utilization of online strategies and lower investment in online advertisements compared to other modalities (for example mailed brochure), it remains unclear whether there are plateau effects that could cap online recruitment yields regardless of greater spending. The scalability of these methods remains an important topic for future research.

Our study showed that one online recruitment strategy, right column Facebook banner advertisements, performed comparably cost-wise, to direct mail and was less expensive than in-person fairs. However, when under-represented groups (African Americans or men) were targeted using the same Facebook advertisements, Facebook was substantially more expensive and did not perform as well as other methods. These findings suggest that simply changing target audience is not enough to overcome recruitment barriers. Rather, presentation, message, and mode may be important considerations for enrolling certain groups of participants. While Facebook is the most popular social media platform,¹⁵ the use of the site among subgroups may vary. Targeting other popular social media platforms more popular among African American communities, (e.g., Instagram)²⁶ may improve efficacy. Furthermore, our study highlights a role for adopting a multi-modal recruitment strategy in order to achieve demographic study targets.

Online recruitment affords an opportunity to conduct micro-trials to refine recruitment materials and optimize yields. In our study, we found that small changes to text or logo had little impact on visitation to our study website. However, a subtle and unanticipated change in title more than tripled traffic to our website. We expect that there are many such factors unanticipated by investigators at the time recruitment materials are developed. It is possible that employing ethnographic research informed by patient interactions or community advisory boards prior to initiating recruitment campaigns might help identify and avoid ineffective recruitment materials. However, it is also possible that a completely agnostic approach that employs iterative micro-trials may be necessary to tailor recruitment messages to different subgroups. However, to implement an agnostic approach, it will be necessary to work closely with IRBs on how to secure approval for not only this approach but also messaging.

This study had limitations. First, accurate estimation of costs is inherently challenging. This issue is compounded by the fact that comparing recruitment costs was not a pre-planned outcome of the trial. As a result, we did not have information on the timeliness of response, how participants chose to respond to media, and details related to many expenses, including staff effort to design brochures, present at fairs, as well as design and maintain the banner advertisements and website. Second, our study had specific inclusion and exclusion criteria as well as study obligations that may not be readily applicable to other clinical trials or studies. Third, this study did not explore all advertising methods (television, radio, in-clinic recruitment), which may also be an important means of advertising for other studies. Fourth, our study focused on only one form of online advertisements (right column banner ads) that we thought a priori would be comparable to other traditional means of advertising. It is possible that other forms of online recruitment (blogs, news feeds, search engine advertisements, etc) could differ in terms of cost effectiveness. For example, there is growing enthusiasm for recruitment via electronic medical records, which was not explored in this study. Fifth, the implementation of our micro-trials to target African Americans or men evolved during the course of the recruitment period based on trial needs at that time. While this reflects a real-life application and advantage of online recruitment methods, it limits the robustness of our contrasts across Facebook ads. Finally, we were unable to track if respondents had seen multiple advertisements from different sources. This could be an important consideration for recruitment strategies in other trials.

This study also has many meaningful strengths. Online recruitment strategies were tested in the context of an actual community-based trial, which involved two commonly utilized interventions (pill and behavioral). Furthermore, we collected sufficient demographic information to examine the effects of different recruitment methods on participants that participate in trials less, namely, African Americans and men. Finally, the breadth of strategies employed by this study provide an opportunity to compare novel and traditional recruitment strategies, while minimizing confounding influences from different trials, temporal trends, or other target populations.

This study has important implications. Internet utilization is on the rise and patterns of engagement in traditional media are changing, such that exploration and understanding of online strategies to reach a diverse population of study participants is critical for the success and generalizability of future clinical trials. While the relevance of traditional outreach methods long-term remains unclear, adaptive behaviors to online advertising – for example, desensitization to advertisements, advertisement-blocking software, new privacy protection policies limiting accurate targeting – also raise questions about the long-term viability of online techniques. Ultimately, a multiprong strategy involving traditional and novel methods may continue to be a mainstay for meeting recruitment quotas, which represents an important topic for future research.

In conclusion, we found that Facebook advertisements directed a target population to our trial that resulted in enrollees. While general Facebook recruitment strategies were not superior to traditional methods from a cost perspective, the opportunity to optimize recruitment messaging through micro-trials offers the potential for precision recruitment with greater yields over time. Further research is needed to evaluate online randomization tools to enhance recruitment in the context of online campaigns.

Abbreviations

SPIRIT

Survivorship Promotion In Reducing IGF-1 Trial