Randomized Comparison of Online Motivational Themes in Clinical Trial Recruitment

Abstract

BACKGROUND:

Targeted, digital recruitment strategies such as tailored websites using motivational themes may improve recruitment in clinical trials, but their effectiveness remains unclear. We hypothesized that themes emphasizing community well-being, personal health benefits, or access to perks would increase engagement and prescreening sign-ups compared with a standard contribution to science message in a clinical trial focused on Black adults.

METHODS:

We implemented A/B testing of website themes for recruitment in two randomized trials testing the dietary approaches to stop hypertension diet intervention on blood pressure among Black adults. Website themes were derived from predeveloped motivational categories and included (1) contribution to science (control group), (2) community well-being, (3) personal blood pressure improvement, and (4) access to perks (groceries or cash). A/B randomization directed visitors to a theme randomly between June and December 2024. Using an open-source Web analytics platform, we captured data on 2 primary outcomes: (1) sign-up rate defined as the proportion of unique visitors who completed the trial’s prescreening form and (2) engagement defined as (a) mean pageviews per session and (b) mean time spent on site per session. We compared themes using the Welch t tests with statistical significance assessed as 2-tailed P<0.05.

RESULTS:

Among 11 484 visitors over 6 months, the themes of community well-being (13.8%), personal blood pressure improvement (14.1%), and access to perks (13.1%) all attracted higher sign-up rates than contribution to science (11.1%; P<0.05 for all comparisons). All alternative themes also led to significantly higher mean pageviews compared with the contribution to science theme (P<0.05 for all comparisons), while mean time on site was similar across all themes (range, 52–55 seconds with P>0.05 for all comparisons). There were no statistical differences noted across the 3 alternative motivational themes for these outcomes.

CONCLUSIONS:

Tailored websites with digital messages emphasizing community well-being, personal health benefits, and access to perks significantly improved engagement and prescreening sign-up rates, demonstrating that they may enhance recruitment within cardiovascular research.

REGISTRATION:

URL: https://www.clinicaltrials.gov; Unique identifiers: NCT05393232 and NCT05121337.

What is Known

• Digital recruitment strategies have the potential to improve accessibility and reach in clinical trials.

• Despite this, Black adults remain underrepresented in cardiovascular research although they bear a disproportionate burden of disease.

• Prior studies using targeted recruitment materials have shown mixed success, some improving Hispanic participation, while others found no significant effect among Black populations.

• The optimal messaging approach for equitable recruitment remains uncertain.

What the Study Adds

• This is the first randomized comparison of motivational messaging themes in digital cardiovascular trial recruitment specifically targeting Black adults.

• Personally relevant, health-focused messages (personal blood pressure improvement) achieved 27% higher sign-up rates compared with traditional altruistic appeals (contribution to science).

• All tailored motivational themes led to 71% to 74% greater website engagement than the standard message.

• These findings provide evidence-based guidance for optimizing digital recruitment messaging.

• The study offers a pragmatic A/B testing framework that other trials can use to refine recruitment strategies for historically underrepresented populations.

Despite having a higher risk of cardiovascular disease, Black adults are not well represented within clinical trials.^1,2 The lack of representation not only limits the generalizability of the study results but also leads to the development of inequitable treatment regimens.^3,4 Recently, there has been an increased effort to improve representation in clinical research through initiatives such as the All of Us Research Program, the Black Women’s Health Study, and other National Institutes of Health–supported networks.^5,6 However, systemic barriers, including distrust of the medical system, limited access to trial locations, and noninclusive recruitment strategies, discourage historically marginalized groups from participating in studies.^7,8

Digital advertisement–based recruitment is a scalable and effective way to make clinical trials more accessible.^9,10 Traditional approaches for outreach, such as physician referrals, clinic-based flyers, patient databases, and community-based advertising, are often insufficient to engage a representative audience.^11,12 Furthermore, there is increasing recognition that motivationally aligned and targeted strategies may improve engagement, particularly among underrepresented groups.¹³ Recruitment messaging themes emphasizing community well-being and personal health benefits can pique interest in trials and also improve participation.¹⁴ Strategies that are specifically tailored to represent the life experiences, values, and goals of the participants, such as community well-being or personal growth, may enhance accessibility and inclusivity within research studies.¹⁵

Theoretical frameworks that include motivational interviewing and behavioral economics have shown that framing recruitment materials to focus on intrinsic motivators, such as helping the community or getting better health, might enhance both enrollment and retention.¹⁶ Similarly, research emphasizing community benefits has shown promise in not just getting people to join but also in improving adherence.¹⁷ Such approaches may have the potential to be used in different trial settings to improve recruitment and minimize attrition.

A nuanced understanding of the factors influencing recruitment is needed to develop more effective and inclusive digital approaches, especially across diverse populations. Our research aims to address this critical knowledge gap in digital recruitment science. We hypothesized that tailored digital recruitment messaging themes would outperform conventional approaches that focused on contribution to science, specifically focusing on themes of community well-being, personal health benefits, and access to perks.

Methods

The data that support the findings of this study are available from the corresponding author upon reasonable request. As the data set includes Web analytics containing potentially sensitive metadata, it cannot be publicly shared to protect participant privacy.

Overview of GoFresh Trials

This study was undertaken within the GoFresh (Groceries For Black Residents of Boston to Stop Hypertension) trials (ClinicalTrials.gov; Unique identifiers: NCT05121337 and NCT05393232) which investigated how home-delivered healthy groceries affect blood pressure (BP) among Black residents of healthy food priority areas in Boston with untreated and treated hypertension. Participants were randomly assigned to either 12 weeks of dietary approaches to stop hypertension–patterned¹⁸ home delivery of groceries with dietitian support or a monthly stipend for self-directed shopping. After the intervention, both groups were observed for 9 additional months to identify barriers and facilitators for maintaining the dietary approaches to stop hypertension diet. The trials aimed to improve cardiovascular health among Black adults in healthy food priority areas and were designed to generate insights into effective, scalable hypertension prevention strategies, focusing on changes in systolic BP, diet quality, and barriers to maintaining the dietary approaches to stop hypertension diet.¹⁹

The GoFresh trials implemented a comprehensive digital recruitment strategy that utilized Facebook advertisements geotargeted to predominantly Black neighborhoods in greater Boston, dietary approaches to stop hypertension diet messaging with call-to-action buttons directing users to the study website. Additional offline materials (bus and subway advertisements, community flyers, and direct mail) included quick response codes or uniform resource locators to the same website. Digital advertisements used the urchin tracking module tracking parameters for source attribution though offline tracking implementation varied. Visitors from all sources were randomly assigned to 1 of 4 motivational message variants (Figure S1) through Matomo, an open-source, privacy-preserving analytics platform for Web analytics that could be locally hosted.²⁰ Use of Matomo and its A/B testing platform allowed us to ensure upstream advertising content did not determine website theme exposure. This study leveraged the digital recruitment infrastructure of GoFresh trials to evaluate whether website-based messaging with distinct motivational themes could have helped optimize participant engagement and enrollment.

We derived motivational themes from available literature on factors influencing clinical trial participation among underrepresented populations.^21–24 From this review, we identified 4 motivational themes consistently associated with GoFresh’s enrollment requirements: (1) altruistic contribution to scientific advancement (contribution to science, serving as the standard control theme), (2) community-level health benefits (community well-being), (3) personal health improvement (personal BP improvement), and (4) tangible material incentives (access to perks: groceries or cash). For each category of themes, we developed concise website header messages (Table 1) designed to be clear, actionable, and appropriate for Web-based recruitment. The messages for each theme were developed with a community advisory board.

Table 1.

Four Motivational Themes and Associated Header Messages Tested Across Recruitment Website Variants in the GoFresh (Groceries For Black Residents of Boston to Stop Hypertension) Trial A/B Study

A/B Testing Infrastructure

We developed 4 themed recruitment websites, each featuring identical content and layout but with distinct motivational messages displayed. Uniform resource locator–based randomization occurred at the visitor level through Matomo’s A/B testing module. Upon first website access, each visitor was randomly assigned to 1 of 4 thematic variants during the recruitment period from June 25 to December 31, 2024 (Figure), with assignment persisting across subsequent visits via browser cookies. Visitors accessing the website from multiple browsers or devices could potentially be assigned to different themes; we excluded 196 visits from 45 visitors (0.4% of the sample) who encountered multiple theme variants to ensure clean experimental groups. Minor group-size variations were expected due to real-world traffic fluctuations across advertising sources, rather than an imbalance in the randomization algorithm. All engagement metrics (pageviews and time on site) were aggregated per unique visitor across all their visits, and prescreening sign-ups were counted once per individual based on prescreening form (Figure S2) submissions.

Figure.

A/B randomization of website variants. Schematic of the uniform resource locator–based randomization of recruitment website visitors into 4 motivational message variants using Matomo’s built-in A/B testing module.

Data Quality and Validation

To ensure data integrity, we implemented several validation procedures. Matomo allows for the identification of unique visitors using first-party browser cookies, supplemented by device fingerprinting for cookieless identification. A unique visitor was defined as a distinct browser session identifier (cookie or device fingerprint) accessing the website during the study period. We excluded (1) visitors with IP addresses outside the United States (n=599), (2) visits originating from Matomo hosting location (n=14), and (3) visitors exposed to multiple theme variants during the study period (n=196 visits from 45 visitors). Bot traffic was filtered using Matomo’s built-in bot detection algorithm, which identifies nonhuman traffic patterns based on user-agent strings and behavioral signatures. Repeat prescreening sign-ups from the same individuals were counted once in our analysis. After all exclusions, the final analytic sample comprised 11 484 unique visitors and 1497 unique prescreening sign-ups (some visitors returned to view the confirmation page multiple times, resulting in 1723 confirmation page views from 1497 individuals).

Data Collection and Key Metrics

The Matomo tracking code was embedded across all website variants to enable real-time A/B test monitoring and data collection. Primary outcome measures included (1) sign-up rate and (2) engagement. The sign-up rate was defined as the proportion of unique visitors who completed the trial’s prescreening form with a visit to the study’s confirmation (Thank You) page, accessible only after successful form submission. Prescreening sign-ups were deduplicated by unique visitor ID (1723 confirmation page views from 1497 unique individuals), with each visitor counted once. Engagement was operationally defined through 2 metrics: (1) mean pageviews per session and (2) mean time spent on site per session. These metrics were analyzed separately to capture distinct aspects of visitor engagement. Pageviews per session were defined as the total number of pages viewed divided by the total number of sessions for each unique visitor. Secondary outcomes included the total number of visits, unique visitors, and completed prescreening forms stratified by website variant.

All metrics were collected continuously throughout the 6-month recruitment period, with Matomo’s A/B testing module providing automated visitor segmentation and performance tracking across the 4 website themes. Matomo does not collect demographic data at the website stage. Demographic information including self-reported race, ethnicity, age, and sex was collected via the prescreening form but was not linked to website analytics for this analysis.

To assess potential confounding from traffic sources, we tracked the referring campaign for each visitor using the urchin tracking module parameters embedded in advertisement uniform resource locators. Visitors were categorized as (1) theme-matched Facebook advertisements (where the ad message aligned with the randomized website theme), (2) theme-mismatched (where ad and website messages differed), and (3) neutral sources (offline recruitment materials, direct traffic, organic search, or unknown sources without urchin tracking module tracking; Table 2). This categorization allowed us to evaluate whether upstream advertising content influenced the observed theme effects.

Table 2.

Website Engagement Metrics and Sign-Up Rates by Motivational Messaging Theme for Cardiovascular Clinical Trial Recruitment (N=11 484 Unique Visitors)

Statistical Analysis

Comparative analyses between the 4 website themes were conducted using the Welch t tests to estimate mean differences in outcome metrics at the visitor level. For each unique visitor, we calculated their mean pageviews per session and mean time per session across all their visits during the study period and then compared these visitor-level aggregated means across theme groups.

Sample size was calculated prospectively using Matomo’s A/B testing calculator, which was configured to detect a minimum relative effect of 10% for the primary engagement metrics (pageviews per visitor and time on site), with 95% CIs and 80% power (α=0.05, 2-tailed) over the planned 6-month recruitment period. The calculator indicated 4754 visitors per website theme would be required (19 016 unique visitors in total). The study recruitment period yielded 2769 to 2991 unique visitors per theme group (11 484 unique visitors in total) after data cleaning and quality control procedures. The observed reference group (contribution to science; n=2769) had mean pageviews per visitor of 1.4 (SD, 0.86) and mean time on site of 53.0 (SD, 191.5) seconds. The achieved sample was smaller than the prospective target but still provided sufficient power given the large observed effects.

Sign-up rates were compared between themes using 2-proportion Z tests, with 95% CIs calculated for rate differences. The contribution to science theme served as the reference group for all comparisons. Statistical significance was defined using a 2-tailed alpha level of P<0.05. Effect sizes were calculated as percentage differences relative to the control condition.

We performed sensitivity analyses to assess whether findings were robust when excluding visitors from Facebook advertisements, which contained themed messaging that could potentially interact with website themes. Analyses were conducted separately for (1) visitors from unknown/direct sources only (n=4190; 36.5%) and (2) visitors from neutral/offline sources combined with unknown sources (n=5386; 46.9%), excluding all Facebook advertisement traffic.

Institutional Review Board (IRB) Approval and Trial Registration

The GoFresh trial and its recruitment methodologies were approved by the Beth Israel Lahey Health institutional review board. As this study involved testing of IRB-approved motivational messages within established trial recruitment pathways, additional IRB approval was not required. The parent GoFresh trials were registered on ClinicalTrials.gov (Unique identifiers: NCT05121337 and NCT05393232). This embedded Study Within A Trial (SWAT) was not separately registered or preregistered with outcomes, as it involved testing of IRB-approved recruitment materials within the established trial recruitment infrastructure and did not alter participant eligibility, intervention, or outcomes. The University of Vermont team analyzed only deidentified, aggregated website analytics data without access to individual-level demographic information. Personally identifiable information collected by Matomo (IP addresses, device identifiers, and precise geolocation data) was anonymized or deleted during data processing to ensure participant privacy.

Results

There were 11 484 visitors over 6 months. Of these visitors, 53.1% (n=6098) arrived via themed Facebook advertisements, 10.4% (n=1196) were from neutral/offline sources, and 36.5% (n=4190) had unknown referral sources (likely direct traffic, organic search, or tracking failures). The distribution of traffic sources was balanced across the 4 theme groups, indicating successful randomization at the website level independent of upstream advertising.

Prescreening sign-up rates differed significantly between the alternative motivational themes and the control group of contribution to science. The theme of personal BP improvement achieved the highest sign-up rate (14.1% [95% CI, 12.9%–15.5%]), representing a significant 3.0 percentage point increase over the contribution to science theme (11.1% [95% CI, 9.9%–12.3%]; P<0.001). Similarly, findings were noted for the community well-being theme (13.8% [95% CI, 12.7%–15.1%]; P=0.001; 2.7 percentage points over the control group) and access to perks theme (13.1% [95% CI, 11.9%–14.3%]; P=0.020; 2.0 percentage points over the control group). There were no statistical differences noted across the 3 alternative motivational themes.

Website engagement also varied significantly across motivational themes (Table S1). The access to perks theme generated the highest mean pageviews (2.4±1.38), followed closely by the community well-being theme (2.3±1.43) and personal BP improvement theme (2.3±1.31), while the contribution to science theme showed substantially lower engagement (1.4±0.86 pageviews). All 3 alternative themes demonstrated statistically significant improvements in pageviews compared with the contribution to science theme (P<0.001), with relative differences of 71.6% to 73.8%. Differences in pageviews among the top 3 performing themes were not statistically significant.

Mean time spent on the website showed no statistically significant differences across the alternative themes or the contribution to science theme. Users spent the longest time engaging with the community well-being theme (55.0±190.2 seconds; 4.3% [95% CI, −14.4% to 22.9%]; P=0.65), followed by the access to perks theme (54.0±188.2 seconds; 2.4% [95% CI, −16.3% to 21.0%]; P=0.81), contribution to science theme (53.0±191.5 seconds), and personal BP improvement theme (52.0±187.2 seconds; −2.0% [95% CI, −20.8% to 16.9%]; P=0.84).

Sensitivity analyses excluding Facebook advertisement traffic confirmed the robustness of our key findings (Table S2). Among visitors from unknown/direct sources only (n=4190) and from neutral/offline and unknown sources combined (n=5386), the alternative themes demonstrated statistically significant increases in pageviews compared with contribution to science. Similarly, among visitors from neutral/offline and unknown sources combined (n=5386), sign-up rates showed similar directional patterns across subgroups but did not reach statistical significance in sensitivity analyses due to reduced sample sizes. Time on site showed no significant differences across themes in any subgroup analysis, consistent with primary findings.

Discussion

We explored the impact of 4 motivational themes on a recruitment website for a hypertension clinical trial and found that 3 alternative themes, community well-being, personal BP improvement, and access to perks, significantly increased sign-up rates and pageviews relative to a standard contribution to science message. In contrast, average time on site showed no significant differences across themes.

Given that this study was targeted at a hypertension trial recruiting Black adults, our findings are particularly relevant in the context of prior work. Previous studies have identified significant challenges in recruiting underrepresented populations for clinical trials, with key barriers including mistrust, lack of interest, and cultural disconnection.²⁵ While culturally adapted recruitment materials have shown some success, with one study reporting Hispanic recruitment increases from 7.0% to 39.1% following targeted modifications,²⁶ results remain inconsistent. A recent trial found that culturally tailored videos did not significantly improve enrollment decisions among Black parents compared with generic messaging.²⁷ These mixed findings suggest that personal relevance may be more influential than cultural targeting alone, highlighting a critical gap in understanding effective digital messaging strategies for diverse populations.

The success of personal health–focused messaging (personal BP improvement theme) over altruistic appeals (contribution to science) in the context of hypertension clinical trials corresponds with several studies that show that personal benefits are usually the primary participation motivation, while the altruistic motivation is subsidiary.^28–31 This theme combined strong user engagement with higher conversion performance. However, a counter to this argument is that we found that the community well-being theme also showed higher engagement and sign-up rates than the control message of contribution to science, suggesting that altruism could still play a role. The disconnect between findings of engagement metrics such as time spent on the website and prescreening sign-up rates reveals that sustained attention or interest in recruitment material (such as a website in this case) does not necessarily predict actual participation or commitment. This may indicate that immediate personal relevance may play a stronger role in enrollment decisions than broader social motivations.^32–34

The contribution to science message (the broadest altruistic appeal) generated the lowest prescreening sign-up rates, indicating that researchers should give greater consideration to creating themes aligned with specific demographic considerations rather than relying on generic recruitment language in health research. Because people are motivated by different factors, researchers may achieve better performance from recruitment materials by exploring diverse messaging approaches.^35–37

These findings must be interpreted within the broader context of demographic and trust dynamics that influence recruitment effectiveness. Prior research with Black adults reveals that the relationship between health care trust and responsiveness to personal health messaging can be counterintuitive, with health care system trust levels sometimes negatively correlating with participation willingness when exposed to health-focused themes.^38–40 This could mean that the observed performance of personal BP improvement messaging may be influenced by the socioeconomic composition and trust profile of the target population.³⁷ Rather than using standardized recruitment messaging across all populations, researchers may want to develop targeted messages that align with the specific motivational priorities of groups.³⁶ Tailoring digital recruitment to emphasize community well-being, personal health benefits, or access to perks, as demonstrated in this study, may help close recruitment gaps for Black adults in clinical research. Of course, our study does not address differential rates in Black adults relative to other groups as we only evaluated our strategies in Black adults.

From a practical recruitment perspective, these findings recommend incorporating motivational themes that emphasize relevant benefits in study communications. The advantage of the personal BP improvement theme (or other alternative themes) over the contribution to science theme in sign-up rates could meaningfully improve recruitment success and study timelines. When selecting among effective messaging strategies, researchers should consider practical factors including target sample size, available resources, and alignment with study objectives. Researchers must also consider whether different messaging approaches may attract participants with varying demographics or characteristics though our current data do not directly measure such differences. Our study demonstrates that motivational messaging significantly influences initial website engagement and prescreening sign-up behavior, but these represent early stages in the recruitment funnel, and we also could not assess downstream conversion rates from prescreening to actual enrollment. It is also important to note that motivational messaging represents only one component of a broader recruitment strategy. These digital approaches are best understood as complementary to traditional, community-based, and clinical outreach methods. Combining online engagement with trusted, locally based recruitment pathways may improve both reach and representativeness.

It is worth noting that messaging emphasizing access to perks can raise ethical considerations about participant motivation. The literature presents mixed perspectives: several studies suggest that offering incentives can ethically and effectively enhance recruitment and retention, particularly in low-risk trials with direct participant benefit, without compromising voluntariness or trial integrity. Conversely, other research cautions that incentive-focused messaging may disproportionately attract individuals motivated primarily by financial gain or those experiencing economic constraints, potentially introducing selection bias or shifting participation motives away from health-related goals.^41,42 However, in the GoFresh trials, grocery support was integral to the intervention itself rather than merely a recruitment incentive. From an ethical perspective, future digital recruitment efforts should continue to balance these ethical considerations, ensuring that tangible benefits are framed to promote equitable participation and access, while avoiding undue influence on decision-making.

Our use of embedded A/B testing demonstrates how digital recruitment platforms can serve as both intervention delivery systems and experimental laboratories for optimizing enrollment strategies.⁴³ This methodology offers a practical framework for other trials to test and refine recruitment approaches continuously, enabling evidence-based messaging decisions that may improve representation of historically underrecruited populations.

Limitations and Future Research

Several additional limitations should be considered when interpreting these findings. Our focus on hypertension-related interventions and measurement of initial prescreening sign-up behavior, rather than long-term retention, limits generalizability to other health conditions and study phases. In addition, our study population’s demographics, which focus on Black adults with hypertension, do not represent all target populations for health research. As noted above, our findings could be similar or different in other groups. While we tracked traffic sources and confirmed balanced distribution across theme groups, 36.5% of visitors had unknown referral sources due to missing urchin tracking module parameters, which may have resulted from ad-blocker interference, direct uniform resource locator entry, or implementation errors. However, sensitivity analyses excluding Facebook advertisement traffic yielded consistent effect estimates (Table S2), indicating that the observed theme differences were not confounded by upstream advertising content. There is no demographic data at the website visitor stage. While the GoFresh trials targeted Black adults through geographically targeted advertisements in predominantly Black Boston neighborhoods, Matomo’s privacy-preserving architecture does not capture demographic information. Demographic characteristics were only collected after prescreening completion, preventing verification of whether all visitors were within the target demographic or assessment of whether messaging effectiveness varied by visitor characteristics. In addition, we could not determine the sign-up rates of those who ultimately enrolled in the trial. Our analysis treated each unique visitor as independent based on Matomo’s cookie-based identification. However, we could not account for potential clustering effects if multiple visits originated from the same household or if individuals accessed the site from multiple devices, which may have slightly underestimated standard errors.

Conclusions

We found that tailored digital messaging strategies significantly influence engagement and participation intentions in paired cardiovascular clinical trials recruiting Black adults. Our findings suggest that digital recruitment strategies employing targeted messages may enhance equitable participation in cardiovascular disease research by providing a better understanding of diverse motivational drivers within target populations.

Article Information

Acknowledgments

The authors would like to thank participants in the GoFresh (Groceries For Black Residents of Boston to Stop Hypertension) trials.

Sources of Funding

This work was supported by the IMPACT (Improving Participation Among Diverse Populations in Cardiovascular Clinical Trials) Science of Diversity in Clinical Trials Strategically Focused Research Network Digital Advertisements for Inclusive Trial Accrual (DAITA) award 953782. IMPACT Principal Investigator (PI): Dr Michos. DAITA PI: Dr Plante. Groceries For Black Residents of Boston to Stop Hypertension trials; were funded by the American Heart Association’s Health Equity Research Network (grant 878488) and the National Institute on Minority Health and Health Disparities (R01MD016068).

Disclosures

None.

Supplemental Material

Tables S1–S2

Figures S1–S2

Supplementary Material

hcq-18-e012945-s001.docx ^{(1.1MB, docx)}