Meeting clinical recruitment milestones in an academic center: a data-driven, visual approach

Abstract

Objectives

Describing the development of a visual dashboard leveraging available tools for efficient recruitment for patient centered clinical trials in resource constrained settings.

Materials and Methods

A real-time, visual dashboard was developed, facilitating interactive visualizations, detailed analyses, and data quality control. Daily automated REDCap data retrieval occurred via an R program using REDCap API and output was integrated into Power BI. An interrupted time series analysis was conducted evaluating effects of dashboard on clinical trial recruitment metrics.

Results

The visual dashboard displayed key recruitment metrics, including individual participant progression and recruitment trends over time. Interrupted time series analysis showed improvements in screening rates upon implementation. The mean time to study completion decreased by 19 days following implementation.

Discussion

Customizable metrics offer comprehensive view of recruitment data and granularity, identifying actionable issues, enhancing study timeliness and completion.

Conclusion

Clinical trials of all budgets can integrate dashboards for real-time monitoring and data driven improvements to promote more timely completion.

Background and significance

Nearly 25% of randomized controlled trials (RCTs) are discontinued due to poor recruitment.¹ Trial discontinuation hinders innovative discoveries and delays integration of advancements into clinical care, impeding efforts to improve patient outcomes. Effective recruitment, enabled by sufficient personnel and funding, is essential for clinical trial success.² A data informed approach leveraging available technology can assist in identifying optimal recruitment methods, leading to above average recruitment performance and RCT completion.^2–4 Among discontinued trials, just 8% reported using strategies to monitor and support recruitment, possibly due to lack of sufficient funding.^4,5 Accessible solutions to routinely monitor recruitment data are necessary.

One increasingly popular monitoring method is the visual data “dashboard.” Visualization of information can help teams optimize workflow, increase agility to adapt processes, and communicate progress. Dashboards have been explored in a variety of clinical and research settings and are associated with positive end-user feedback.^6–11 In one multisite RCT, a dashboard was developed to display participant enrollment at various hospital sites, leading to more efficient and organized study team communication. However, this dashboard required users to deploy software on their own webserver, necessitating programming experience. Additional features to this dashboard increase development time and complexity, making it infeasible for many study teams.⁶

Proprietary software products exist and may enable more complex or nuanced recruitment monitoring; however, there is generally a high initial cost along with recurring fees for ongoing use, presenting a barrier for smaller budgeted trials. These products may require duplicative data entry (ie, data entry into recruitment software and standard study data entry), costing additional personnel effort. Low-cost recruitment support tools capable of visual display of recruitment data in a timely, feasible, and affordable manner are needed to support clinical trials.³ To date, description of such non-proprietary tools is limited in the literature. Existing studies describe tools requiring extensive programming experience and time, or focus on modeling and predicting recruitment and lack customizability.^6,12,13 Here, we present a customizable dashboard leveraging widely available low- or no-cost tools to enhance RCT recruitment.

Methods

Development

The dashboard was created to support a pediatric placebo-controlled crossover RCT (NCT05509257). Figure 1 illustrates each step of the iterative dashboard development process and guiding questions. Prior to development, the study team conducted a workshop to identify metrics and goals of the dashboard. Metrics were based on core features of the recruitment workflow (candidate and participant phase) and selected by team consensus (Table 1). Candidate phase included rates of potential participants who were (1) contacted and interested, (2) could not be reached, or (3) contacted and declined for any reason (eg, not interested, unable to contact). “Potential participants” were individuals who passed prescreening (eg, identified via chart review or other IRB-approved method). The participant phase included events occurring after informed consent and included rates of (1) screening visit completion, and (2) completion of all study procedures. Initial metric goals were determined based on study team experience, target enrollment, and stretch goals. The 90-day timeframe used in metrics was a feasible estimate of expected time elapsed from candidate to study completion based on study design. Following initial development, the dashboard underwent pilot testing through review at weekly lab meetings x4 weeks to ensure metrics (1) accurately reflected current conditions, and (2) were meaningful as intended. Suggestions for minor improvements were discussed and implemented upon study team consensus. Template code and Power BI dashboard can be found on our GitHub.¹⁴

Figure 1.

Iterative workflow for the development of a visualization dashboard. The questions between steps guide quality control and process improvements.

Table 1.

Metric calculation equations.

Metric	Equation
Screening rate	$\frac{\#\mathrm{participants}\mathrm{ }\mathrm{screened}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}{\#\mathrm{candidates}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}$
Completion rate	$\frac{\#\mathrm{participants}\mathrm{ }\mathrm{completed}\mathrm{ }\mathrm{study}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}{\#\mathrm{participants}\mathrm{ }\mathrm{screened}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}$
Declination rate	$\frac{\#\mathrm{candidates}\mathrm{ }\mathrm{declined}\mathrm{ }\mathrm{for}\mathrm{ }\mathrm{any}\mathrm{ }\mathrm{reason}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}{\#\mathrm{candidates}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}$
Not interested rate	$\frac{\#\mathrm{candidates}\mathrm{ }\mathrm{explicitly}\mathrm{ }\mathrm{not}\mathrm{ }\mathrm{interested}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{study}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}{\#\mathrm{candidates}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}$
Could not contact rate	$\frac{\#\mathrm{candidates}\mathrm{ }\mathrm{unable}\mathrm{ }\mathrm{to}\mathrm{ }\mathrm{contact}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}{\#\mathrm{candidates}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}$
Interested rate	$\frac{\#\mathrm{candidates}\mathrm{ }\mathrm{explicitly}\mathrm{ }\mathrm{interested}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{study}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}{\#\mathrm{candidates}\mathrm{ }\mathrm{in}\mathrm{ }\mathrm{past}\mathrm{ }90\mathrm{ }\mathrm{days}}$

Where “candidate” is a unique individual identified through pre-screening methods and “participant” is a unique individual who has consented to study participation.

Data capture

RCT data (eg, demographics, eligibility, recruitment efforts, and study procedures) were collected using Research Electronic Data Capture (REDCap, V14.2.2).¹⁵ Recruitment effort tracking was largely by date (eg, date marked “candidate”; dates of contact; declination and/or visit dates). Details including reason for declination and recruitment source were recorded.

Data cleaning and preparation

Data were pulled, cleaned, and reshaped using an R (V4.3.3) program developed by a member of the study team.^14,16 The program uses the REDCap Application Programming Interface (API) to export specified REDCap reports (.csv), cleans and reshapes into one larger dataset, and outputs as an Excel workbook file (.xlsx) into a specified location on a secure institutional server. The “task scheduler” on a local computer was employed to run the R program daily, consistently updating the workbook file.

Data visualization and metrics

The visual dashboard was developed in Power BI (Microsoft Corporation, V2.130.930.0), a business intelligence tool that extracts, analyzes, and displays data from various sources as interactive visuals (ie, tables and graphs).¹⁷ Power BI is free to download, and the pro version is often included with institutional Microsoft Suite subscriptions. Visuals in Power BI are created by selecting the visualization type and desired fields, then formatting and filtering the visual to reach desired outcome.¹⁸ Additional background and resources are offered by Microsoft and Wark’s 2022 paper.^17–19

After connecting Power BI to the data source (ie, workbook file), visuals can be created. Recruitment metrics were date-based (Table 1), which presented challenges for advanced metric reporting in Power BI. Therefore, we disabled Power BI date auto-detection and developed custom Data Analysis Expressions (DAX)-based functions to display a variety of date-based metrics, including a date table (see Supplementary File). Rates per rolling 90-days (Table 1) were calculated.

Evaluation

A pragmatic approach was taken to evaluate dashboard impact on clinical trial recruitment. The pre-implementation period was January 2023-April 2024. Dashboard implementation date was 5/1/24, the post-implementation period May 2024-December 2024 when trial recruitment was complete. Descriptive statistics (mean ± standard deviation, SD) summarized metrics, assessed enrollment trends (eg, time to study completion), and quantified data quality control (eg, data entry errors) pre- and post-dashboard implementation and between school year (September-May) and summer (June-August). Cohen’s d measured effect size of group differences with significance level α = 0.05. An interrupted times series (ITS), using the nlme package in R, explored impact of dashboard implementation on daily recruitment metrics over time.^20–22

Results

Dashboard product

Figure 2 provides an example of key visuals and metrics used in our dashboard. Across all visuals, an individual participant can be selected to view their specific demographics, recruitment source, and study progress. Power BI Pro’s ability to integrate into Microsoft Suite applications (eg, Microsoft Teams) allows dashboard to be accessed by all team members at any time.

Figure 2.

Example of dashboard visuals. (A) Target vs Actual enrollment. (B) Consolidated Standards of Reporting Trials (CONSORT) flow diagram updated daily. Selection by individual participant allows quick visualization of where they are in the process (eg, Scheduled or Study Visit Completed). (C) Recruitment source. (D) Reason for Declination. A stacked area chart shows declination rates (Not Interested and Could Not Contact, Table 1) over time. (E) Current declination rate (Table 1). Color coded based on set target.

Cost of development

Because freely available programs were used to develop and maintain the dashboard (Microsoft Power BI, REDCap, R), the cost of dashboard development was limited to a one-time research assistant effort (5% FTE or 3 weeks). Ongoing dashboard usage and maintenance is supported by minimal research assistant effort (<0.25 hour/week). Automated dashboard updates replaced the manual effort of compiling recruitment reports, saving an estimated 1 hour per week, a net effort reduction of at least 0.75 hour/week.

Data quality control

The dashboard facilitated data quality control through real-time visualization of errors. Data entry errors (eg, missing demographic details, incorrect visit dates) are displayed with updates. Study staff promptly identify errors and implement corrections, ensuring the most valid and complete source data. Date of declination and date of consent, recorded in recruitment tracking form, were selected for data quality analysis, as each candidate should have a value for at least one. Before implementation of the dashboard, only 30% of records had a value for one of these fields. After implementation, the number increased to 72%.

Standardized recruitment

Routine assessment of latency measures prompted the development of a standardized contact schedule to reflect optimal cadence of recruitment and reduce variability (eg, baseline email, followed by call x3 until declination or scheduling). Before the contact schedule was implemented, the mean days to contact was 8.8 ± 24. After implementation of schedule, mean days to contact was 5.3 ± 2.9.

Enrollment trends

Using the dashboard, we tested the assumption that pediatric recruitment follows trends reflective of time of year (ie, higher in the summer, lower during 9-month school year). The average number of candidates per month was 11 ± 5.2 during the school year and 8.0 ± 2.6 in the summer. Though there were more candidates during the school year, the overall screening rate was 17 ± 5.9 while the summer rate was 21 ± 7.8 (Cohen’s d = 0.57, P < .001). The rate of candidates “not interested” during the school year was 30 ± 12, while the summer was 21 ± 7.0 (Cohen’s d = 1.1, P < .001). The rates of candidates explicitly “interested” were higher in the summer at 28 ± 8.5, than the school year at 24 ± 4.7, (Cohen’s d = 0.50, P < .001). The completion rate remained at 30 ± 18 through the school year and summer.

Impact on study completion

ITS analysis (Figure 3A and B) of key study metrics (screening and completion rates) showed no significant difference in slopes pre- or post-dashboard intervention (see Supplementary File for model estimates). However, a secondary model simulating performance in absence of dashboard implementation predicted lower performance. Following dashboard implementation, the mean number of days from candidate to study completion (complete cycle through study) decreased from 92 ± 55 to 73 ± 41. No study staff changes occurred throughout the evaluation period.

Figure 3.

Clinical Trial Recruitment Metrics Pre- and Post-Dashboard Implementation. (A) (screening rate, Table 1) and (B) (completion rate, Table 1) are visualizations of the interrupted time series models. The solid black line illustrates model prediction with dashboard implementation (green shading displays the 95% CI). The red dashed line shows model prediction with no dashboard implementation (red shading displays 95% CI).

Discussion

We developed a dashboard using no/low-cost tools to enable real-time monitoring, optimized recruitment strategies, and timely study completion. Additional benefits of the dashboard included ease of adherence to reporting standards (eg, CONSORT or National Institutes of Health) through automated, customizable reports. Our solution leveraged existing programs available within the institution paired with open-source software to create an efficient, accessible, and low/no-cost product that requires minimal programming experience.

Dashboard creation enabled data-driven insights for efficient recruitment and quality control. Routine monitoring of recruitment source (eg, clinic locations) informed efforts to strengthen targeted provider and clinic relationships. The ability to view trends over time (eg, summer versus school year) allowed us to test assumptions, predict ebbs and flows of recruitment, and align study staff resources accordingly. A standardized contact schedule based on latency values ensured consistent candidate follow up, reducing variability and possible bias. It is likely that these factors worked together to improve efficiency, evidenced by the decrease in days to overall study completion per participant following dashboard implementation. Data accessibility and functionality enhanced overall understanding of recruitment processes and increased collaboration among the study team on recruitment efforts. Consistent dashboard monitoring ensured any source data entry errors were quickly identified and corrected, in turn ensuring higher quality, more complete data for downstream data analysis.

The CONSORT guidelines for RCTs strongly recommend including information on participant randomization, receipt of treatment, data analysis, or loss or exclusion with the reason why to increase transparency of trial methodology and reduce biased results.²³ By transitioning our formerly static and manually updated CONSORT flow diagram to the dashboard, we have a real-time overview of RCT participant status with an added benefit of selecting a specific participant to identify their position within the flow.

Our work is limited, in part, by the pragmatic evaluation approach that combined retrospective analyses of a relatively iterative process that quickly integrated end user (eg, study team) feedback. Our ITS models evaluated daily rates which demonstrated substantial variability and limited model utility. Due to differences in duration and timing of the “school year,” enrollment trend findings may not be generalizable to all locations. Additionally, our dashboard supported a single-site clinical trial. Expansion to multiple sites requires additional considerations, however these considerations can leverage standard site coordination processes. For example, standardization of data collection across sites and protected data portability would be necessary to generate an integrated multi-site dashboard, yet these elements are already known to be key to successful multisite trial conduct.²⁴ Dashboard use for multi-site trials have the potential to enhance and clarify recruitment patterns and strategies across sites, allowing early intervention to support struggling sites and sharing best practices from most productive sites.

Conclusion

We describe the development of a custom visual dashboard using widely available resources, increasing data accessibility for clinical trials large and small. The development of a real-time dashboard optimized clinical trial recruitment by reducing overall time to study completion and facilitating data-informed resource allocation. Customizable metrics offer a comprehensive view of study data and a granular view of individual participants as they progress through trial visits. Dashboards can be integrated into clinical trial workflows for real-time monitoring and data-driven improvements, promoting successful trial completion.

Author contributions

Anna Burns, Stephani Stancil, and John Tumberger conceptualized the project. Anna Burns curated the data, developed the program, and created visuals through Power BI software. Stephani Stancil acquired funding, supervised the project, and provided resources to conduct the project. Stephani Stancil and Anna Burns prepared the original draft. All authors collected research data and reviewed, edited, and approved the final draft.

Supplementary material

Supplementary material is available at JAMIA Open online.

Funding

This work was supported by the National Institute of Mental Health [K23MH130728]. The contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.

Conflicts of interest

None declared.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author and relevant regulatory approval.