Research site mentoring: A novel approach to improving study recruitment

Abstract

Background/Aims

The VA Cooperative Studies Program's (CSP) Network of Dedicated Enrollment Sites (NODES) is a consortium of nine VA medical centers (VAMCs) with teams (nodes) dedicated to enhance performance, compliance, and management of CSP multi-site clinical trials. The West Haven CSP Coordinating Center (WH-CSPCC), study coordinating center for CSP #577, Colonoscopy Versus Fecal Immunochemical Test (FIT) in Reducing Mortality from Colorectal Cancer (CONFIRM) trial, and NODES piloted a “site mentoring” (hub-and-spoke) model. In this model, a node site would work one-on-one with a low enrolling CONFIRM site to identify and overcome barriers to recruitment. The aim was to determine the impact of a research site mentoring model on study recruitment and examine site-level characteristics that facilitate or impede it.

Results

Sites in the mentorship pilot had an average improvement of 5 ± 4 participants randomized per month (min −2.6; max 11.6; SD 4.3). Four of ten sites (40%) demonstrated continuous improvement in the average number of randomized participants per month after the pilot intervention and at three-month follow-up (post-intervention), as compared to the five-month period preceding the intervention. An additional two sites (20%) demonstrated improvement in the average number of randomized participants per month after the pilot intervention, and sustained that level of improvement at three-month follow-up (post-intervention). Additionally, six of ten sites (60%) demonstrated an increased number of participants screened for eligibility immediately following the intervention and at three-month follow-up (post-intervention). Only one site showed a decreased monthly average of randomized participants shortly after the intervention and through the three-month follow-up period.

Conclusions

The site mentoring model was successful in improving recruitment at low enrolling CONFIRM sites. An additional feasibility assessment is needed to determine if this mentoring model will be effective with other CSP trials.

Abbreviations

CBOCs

Community Based Outpatient Clinics

CONFIRM

Colonoscopy Versus Fecal Immunochemical Test in Reducing Mortality from Colorectal Cancer

CSP

Cooperative Studies Program

FIT

Fecal Immunochemical Test

NODES

Network of Dedicated Enrollment Sites

ORD

Office of Research and Development

PACT

Patient Aligned Care Team

VA

Department of Veterans Affairs

VAMCs

VA Medical Centers

WH-CSPCC

West Haven CSP Coordinating Center

1. Introduction

Clinical trials play a significant role in advancing healthcare and its delivery to patients around the world. Given their critical function in healthcare and biomedical research it is essential that study sites are able to effectively and efficiently recruit and enroll eligible participants, as defined by the study specific inclusion/exclusion criteria. A study's inability to enroll its expected number of participants presents significant challenges to obtaining an adequate sample size and providing statistical power to detect clinically meaningful effects on study outcomes [[1], [2], [3]]. These challenges may create burnout and low morale among study team members, and potentially decrease the likelihood of a study sponsor funding a particular investigator's future research proposals [4]. When considering these challenges, it is critical for clinical researchers to consider and develop effective and innovative strategies during the active recruitment phase of the clinical trial.

The Department of Veterans Affairs (VA) is the United States' largest integrated healthcare system and provides comprehensive care to more than 8.9 million Veterans each year [5]. The Cooperative Studies Program (CSP), a division of the VA Office of Research and Development (ORD), was established as a clinical research infrastructure to provide coordination for and enable cooperation on multi-site clinical trials and epidemiological studies that fall within the purview of VA [6]. The West Haven CSP Coordinating Center (WH-CSPCC) is one of five CSP coordinating centers responsible for the planning and conduct of large multi-site clinical trials in the Department of Veterans Affairs [7]. The VA Cooperative Studies Program's (CSP) Network of Dedicated Enrollment Sites (NODES) [8,9] is a consortium of nine VA medical centers (VAMCs) that have teams (nodes) in place dedicated to enhancing the overall performance, compliance, and management of CSP multi-site clinical trials. WH-CSPCC is the coordinating center responsible for CSP #577, Colonoscopy Versus Fecal Immunochemical Test (FIT) in Reducing Mortality from Colorectal Cancer (CONFIRM). CONFIRM is a large, simple, multi-site, randomized, parallel group trial directly comparing screening colonoscopy with annual FIT screening in average-risk individuals [10].

The primary aim of this pilot initiative was to determine the impact of a remote mentoring model on study recruitment at ten low enrolling CONFIRM sites. The secondary aim was to identify site-level characteristics associated with low enrollment. Results from the pilot will inform sponsors and sites on how to align resources and expectations to improve recruitment and the overall success of the clinical trial.

2. Methods

The CONFIRM study was approved by the VA Central Institutional Review Board (Protocol #: 11-03) and study participants provided informed consent either in-person or over the telephone. The study was actively recruiting in 38 VA medical facilities, had an expected weekly enrollment target of 10 study participants, and the WH-CSPCC identified ten CONFIRM sites with low study recruitment that would benefit from site-based mentoring. Eight node sites were paired with one CONFIRM site, and the ninth was paired with two CONFIRM sites. NODES management and the WH-CSPCC developed a site assessment tool (Appendix A) to gather feedback from the CONFIRM site teams. This site assessment tool was then used by the respective NODES Manager to conduct baseline phone interviews with each site team member and their Site Investigator (SI). The results of these interviews identified common themes (Fig. 1) related to site recruitment and site team performance barriers. Based on these common themes, each NODES Manager ascertained essential resources and established action items for their assigned site, including specified metrics (e.g., individual team member goals, weekly strategy or resource application reports, etc.) ancillary to those necessitated by the WH-CSPCC.

Fig. 1.

Monthly average of randomized participants trajectories.

Throughout the duration of the pilot, NODES Managers provided their assigned site teams with remote mentorship, a resource allocation assessment, and performance monitoring. Remote mentorship included frequent communication with sites through e-mails, conference calls, and Microsoft Lync^® during the intervention phase. There were an average of 14 contacts per site made during the intervention. The resource allocation assessment included review of the site infrastructure and the study teams' ability to recruit at CBOCs (Community Based Outpatient Clinics), utilize a Clinical Applications Coordinator (CAC) and Pre-Screening Algorithm, acquire electronic devices/mobile recruitment equipment, and establish access to primary care providers in Patient Aligned Care Teams (PACT). Performance monitoring included ongoing review of the standardized enrollment report and site assessment tool created for the pilot. NODES and WH-CSPCC study leadership met bi-weekly to discuss the status of each pilot site and its challenges and successes. This workgroup determined the best strategies for implementing action items identified during the initial site assessment period. The pilot was conducted over a five-month period (February 2016–June 2016) and data were reviewed, compared, and analyzed prior to the intervention (September 2015–January 2016), during the intervention and for an additional three-month follow-up period (July 2016–September 2016) to assess long-term sustainability of site improvement plans at the local level.

At the end of the pilot period, post-intervention site team interviews were conducted by the respective NODES Manager using the same site assessment tool utilized at the beginning of the pilot period. The outcomes were assessed by the WH-CSPCC and national CONFIRM study leadership teams through data and narrative reports provided by each NODES Manager, where feasibility status was determined, and/or provision of additional mentorship was provided, as needed.

3. Results

3.1. Study team, patient population, and clinic engagement summary

The NODES identified the following common themes impacting recruitment at the ten pilot CONFIRM sites at the pre-intervention phase: Adequate Staffing (N = 7), Using Pre-Screening Algorithm (N = 5), Investigator Engagement (N = 7), Adequate Training (N = 6), PACT Clinic Engagement (N = 1), CBOC Travel Ability (N = 3), Study Activity Organization (N = 3), Adequate Patient Population (N = 3), Motivation (N = 4), Supportive Team Environment (N = 3), and Delegated Responsibilities (N = 3) (Fig. 2). The NODES pilot intervention offered personalized remedies depending on the barriers denoted by each of the ten sites. NODES interventions included organizing regular team meetings, encouraging study team members to take advantage of the resources offered by the WH-CSPCC, implementing the pre-screening algorithm through the help of the local CACs, ensuring engagement of stakeholders (PACT providers), allocating an equal distribution of workload among team members, ascertaining adequate training on protocol related procedures, coordination of travel for recruitment from CBOCs, and working with the local hiring authorities to resolve staffing issues.

Fig. 2.

Common themes: Site mentorship pre-and post intervention.

The pilot intervention revealed that overcoming some barriers were more achievable than others. Some challenges such as hiring adequate staff, engaging CACs to employ the pre-screening algorithm, and engaging PACT providers for referrals were more difficult to prevail and beyond the purview of the NODES mentors as accomplishing these needed facility level support. Consequently, the NODES pilot intervention provided varied outcomes depending on the issues that respective sites encountered.

Table 1 demonstrates the changes in the monthly average of randomized participants at 10 CONFIRM sites during five months preceding NODES intervention, five months during the NODES intervention, and at three-month post-intervention. Four of ten sites (40%) demonstrated continuous improvement in average number of randomized participants per month following pilot initiation and three-month follow-up, as compared to the five-month period preceding the intervention. Additionally, six of ten sites (60%) demonstrated an increased number of participants screened for eligibility immediately following intervention and three-month follow-up (Table 2).

Table 1.

Monthly average of randomized participants per site.

Table 2.

Monthly average of participants screened for eligibility.

The highest increase in the number of randomized participants was observed for Site E. Site E's enrollment increase was attributed to both hiring a full-time study coordinator and implementing the use of the algorithm for pre-screening efforts. In comparison, Site G had reported multiple barriers including a lack of the following site characteristics: Motivation, Site Investigator (SI) Engagement, Study Activity Organization, Adequate Training, and Adequate Staffing. Although the site overcame its staffing deficit, issues with the site's team dynamics and the overall organization of study activities failed to improve despite the NODES mentoring efforts, resulting in no improvement in recruitment at this site. Sites A, B, and I reported challenges with having adequate staffing but all three sites overcame that barrier by hiring the appropriate level of staffing, and subsequently displayed continuous improvement in recruitment.

Site J reported an inadequate number of eligible patients for recruitment at the site. The NODES intervention assisted with implementing various strategies that would enhance recruitment from the limited pool of patients, such as networking with providers for referrals and increasing study awareness within the facility. Despite these efforts, this study site continued to struggle with securing support and engagement from the PACT providers and failed to show improvement in recruitment.

Sites D, F, and H showed improvement in average number of randomized participants per month following pilot initiation and sustained that improvement at the three-month post intervention follow-up. All three sites had reported barriers with adequate staffing, adequate training, and use of the pre-screening algorithm. The NODES intervention helped these sites overcome these challenges by providing them with specific guidance on how to address them. Similar barriers were documented by Site C which were addressed during the intervention phase but were not sustainable during the post NODES intervention phase.

During the post-intervention phase, most sites continued to display improvement, except for those that had issues with available staff and adequate patient population. Staffing levels at the ten pilot sites averaged 2.39 Full-Time Employment Equivalents (FTEEs) per site and ranged from 1.0 to 3.7 FTEE across the ten sites. The number of FTEEs per site did not have an impact on recruitment while the type of staff that were in place seemed to influence the level of recruitment more substantially e.g. staff training level, ability to positively influence and/or contribute to team dynamics, etc. For example, Site E had the lowest number of FTEE (1.0) among the ten pilot sites but was still able to demonstrate improvement in average number of randomized participants per month following pilot initiation and sustainment of that improvement at the three-month post-intervention follow-up. Alternatively, Site J had the highest number of FTEE (3.7) among pilot sites and demonstrated improvement in average number of participants per month following pilot initiation, but that improvement was not sustained at the three-month post-intervention follow-up.

Much of the staffing issues reported post-intervention were the result of turnover and the inability to fill personnel vacancies. When comparing pre-intervention and post-intervention, study teams reported a 25% decline in the eligible patient populations necessary to meet recruitment targets at the post interview. It could be assumed that the NODES mentorship pilot helped sites by proving them with a better understanding of the study protocol and how to organize study activities, how to engage the relevant stakeholders at the site, and how to implement and utilize the pre-screening algorithm at some sites. These themes were alleviated with structured, customized mentoring while others were not fixable, as corrective measures depended on multiple external factors including difficulty with hiring practices at the respective facilities.

3.2. Facility complexity level summary

The type of VA facility that underperforming study teams were based at was also examined as a site characteristic. VA medical facilities are categorized by complexity level which is determined by characteristics of the patient population, clinical services offered, education and research missions, and administrative complexity [11]. These complexity levels are described in greater detail in Appendix C.

Of the pilot sites that participated in this initiative, 70% were designated as Complexity Level (CL) 1a which means that they had the largest levels of volume, patient risk, teaching and research, as well as the largest number and breadth of physician specialists (Appendix C). Of these seven sites, four (57%) demonstrated a continuous improvement in average number of randomized participants per month, following the pilot initiation and at three-month follow-up, and two (29%) showed improvement after pilot initiation and sustained that level of improvement at three-month follow-up (Table 1). There was a single CL 1a site that showed no improvement in average number of randomized participants after pilot initiation or at three-month follow-up. Table 2 indicates that five of seven CL 1a sites (71%) demonstrated an increased number of participants screened for eligibility immediately following the intervention and at three-month follow-up (Table 2). The remaining two CL 1a sites (29%) had no demonstrable changes in the number of participants screened for eligibility following the intervention or at three-month follow-up.

The remaining three sites that participated in this pilot were either classified as CL 1b (n = 2) or CL 2 (n = 1). CL 1b facilities have very large levels of volume, patient risk, teaching and research, while CL 2 facilities have medium levels of teaching/research activity and patient risk (Appendix C). There was an improvement in average number of randomized participants per month and sustainment at one of the two CL 1b sites, while the other CL 1b site showed improvement after pilot initiation but was unable to sustain that level of improvement at three-month follow-up. Interestingly, the CL 1b site that displayed an improvement in average number of randomized participants per month also exhibited an increase in the number of participants screened for eligibility following the intervention and at three-month follow-up. The single CL 2 facility showed improvement in average number of randomized participants per month but was unable to sustain that level of improvement at three-month follow-up, but also demonstrated no change in the number of participants screened for eligibility following the intervention or at three-month follow-up.

4. Discussion

Clinical trials are dependent on the ability of their respective study teams to effectively and efficiently recruit and enroll eligible participants. Their benefit extends to the individual participant by establishing a broader selection of effective therapies, and to society at large by enhancing the value of health care [12,13], but their full potential is unrealized if they are unable to meet their defined enrollment targets. This pilot project suggests that the utilization of a site mentoring model was effective in increasing participant recruitment at the majority of sites that participated in the pilot (70%) and of those sites, 57% were able to sustain their improvement for a three-month follow-up period subsequent to the five-month pilot.

To date, there is a limited amount of published information on the utilization of a site mentorship approach to improve study recruitment in a large, multi-site clinical trial. Most publications on this topic address site mentorship in terms of providing educational and/or professional development opportunities to clinical research personnel but none report using the methodology described in this manuscript to improve recruitment [14,15], nor were they provided in a similar setting (e.g., large integrated healthcare system). Therefore, we are unable to compare the results of our pilot with previous studies but can address some common themes that were reported across pilot sites. These themes provide insight on site characteristics that either impede or facilitate adoption of this site mentorship model and ultimately, impact study recruitment (Fig. 2). Our findings are consistent with previous studies that have demonstrated site characteristics such as supportive team environment, clinic engagement, adequate participant population, investigator motivation/engagement, and physician referrals as being critical elements of clinical trial enrollment and overall trial success [[16], [17], [18], [19]].

This pilot also exhibited that VA medical facilities that had the largest volumes, levels of teaching and research, and largest number and breadth of physician specialists (Complexity Level 1a) had the most significant response to this site mentoring approach and improved their average number of randomized participants per month following pilot initiation and three-month follow-up, as compared to the five-month period preceding the intervention (Table 1). It could be plausible that study site teams at medical facilities with higher levels of teaching and research activities had better success in establishing the stakeholder engagement necessary to successfully conduct clinical trials e.g. clinical teams for referrals, human resource offices for staffing, etc., due to the assumed high level of support for research activities at these sites, as opposed to facilities with less support and infrastructure for research.

There are potential limitations related to the design and methodology of this pilot that may impact the generalizability of the results. First, the pilot was conducted on a relatively small number of sites (n = 10) and over a short timeframe (5 months). It is possible that both increasing the number of sites and establishing a more equal distribution of the type of site that participated in the pilot, as well as conducting the pilot over a longer time period may have yielded different results. Furthermore, the three-month post-intervention follow-up period (July 2016–September 2016) may not have been adequate to assess the long-term sustainment of the study improvement plans, and results may have varied if we examined site performance over a longer follow-up period. Lastly, the results from the pilot are only generalizable to multi-site clinical trials that are being executed in large integrated healthcare systems.

Considering the aforementioned limitations, the pilot demonstrated several key strengths. Previous studies that have examined factors impacting clinical trial enrollment offered contrasting results based on the disease or condition that the trial was focused on. Their results were also dependent on the type of medical facilities implementing the trials (e.g. academic medical centers, community-based healthcare systems, etc.) [[20], [21], [22]]. This pilot was innovative in that barriers to clinical trial enrollment were not previously examined in the context of a site mentorship model or employed in a large, integrated healthcare system comparable to the VA.

This is a critical distinction between our work and previous studies because it is possible that study sites may have been more forthcoming in terms of discussing obstacles to enrollment with individuals that they considered to be peers i.e. participating sites for the same trial, as opposed to a study sponsor or other entity that attempted to solicit this information. A key strength of NODES is the duality of being a component of the study sponsor (CSP) and a site-based consortium. NODES is able to provide insight from study sites to the CSP study coordinating centers and VA research leadership on issues that arise during the execution of clinical trials, as well as develop strategies to address them. Another key strength of this pilot is that we were able to validate the feedback received from the sites on staffing and eligible patient populations. It is likely that similar studies that examined positive and negative factors related to study enrollment in clinical trials were not able to compare the self-reported feedback from study teams to actual data related to those criteria.

5. Conclusions

In summary, the site mentoring model that we employed was effectual in increasing participant recruitment at the majority of CONFIRM sites that participated in the pilot. We believe that its success was, in large part, due to its incorporation of the overall NODES model which prioritizes collaboration and engagement of stakeholders at multiple levels within the organization. In this example, stakeholders external to the CSP at the VA site-level such as clinical trial study teams, Human Resources, Information and Technology offices, and Community-Based Outpatient Clinic administration groups were also engaged. Additional work is needed to determine the feasibility of expanding the mentoring model to additional CONFIRM sites and/or other CSP trial sites. The generalizability and sustainability of the model can be examined once the model has been implemented at other healthcare settings and upon utilization of the model throughout the lifecycle of a clinical trial.

Support

The research reported/outlined here was supported by the Department of Veterans Affairs, VA Office of Research and Development, Cooperative Studies Program (CSP).

Disclaimer

The views expressed in this article are those of the authors and do not necessarily reflect the views or policies of the US Department of Veterans Affairs, or the US government. The mention of trade names, commercial products, or organizations does not imply endorsement by the US government.

Conflicts of interest

The Authors declare that there is no conflict of interest.

Appendix A. Supplementary data

The following are the supplementary data related to this article: