Development, implementation, and dissemination of operational innovations across the trial innovation network

Abstract

Improving the quality and conduct of multi-center clinical trials is essential to the generation of generalizable knowledge about the safety and efficacy of healthcare treatments. Despite significant effort and expense, many clinical trials are unsuccessful. The National Center for Advancing Translational Science launched the Trial Innovation Network to address critical roadblocks in multi-center trials by leveraging existing infrastructure and developing operational innovations. We provide an overview of the roadblocks that led to opportunities for operational innovation, our work to develop, define, and map innovations across the network, and how we implemented and disseminated mature innovations.

Introduction

In 2015, the National Center for Advancing Translational Science (NCATS) released a funding opportunity for Innovation Centers that would leverage the strength of the existing Clinical and Translational Science Award (CTSA) Program to support the efficient and effective conduct of multi-center clinical research [1,2]. The Trial Innovation Network (TIN), launched in 2016, was the latest in a series of CTSA program initiatives to support multi-center clinical research [3]. In addition to building national clinical trial infrastructure and harmonizing processes, central to the TIN is the goal of creating a national laboratory to study, understand, and innovate. Three Trial Innovation Centers (TICs) and a Recruitment Innovation Center (RIC) were established as expert hubs where innovations could be developed and tested at scale.

Clinical trials are essential for developing and testing healthcare treatments, and multi-center clinical trials allow for larger, more accurate, and generalizable assessment of efficacy and effectiveness [4]. Adequately powered randomized controlled trials are the gold standard for testing treatments and practice patterns, but there remain significant roadblocks to trial design and conduct [5–7]. Delays in trial implementation and failure to meet recruitment and retention targets are costly wasted opportunities [8,9]. Additionally, poorly designed or underpowered trials can fail to be informative [10,11]. These roadblocks and failures in trial design and implementation indicate that innovations are needed to improve the quality and conduct of clinical trials. We frame the roadblocks as opportunities for TIN activity and innovation in Fig. 1.

Figure 1.

Roadblocks and barriers throughout the multi-center clinical trial lifecycle were opportunities for TIN activity and innovation. IRB = Institutional Review Board; TIN = Trial Innovation Network.

This brief report describes the process that the TIN went through to define, prioritize, map, share, implement, and disseminate innovations. The process was characterized by a network-wide openness to transparency and a desire to develop innovative and complementary ways to support efficient and effective multi-center clinical trials.

Defining and Prioritizing Operational Innovations

While establishing Network infrastructure, TIN leadership discussed the definition of operational innovation. The definition agreed upon was “A novel technology, method, process, or paradigm that can be used to improve the conduct of the clinical trial or study and/or its ultimate speed or likelihood of translation into practice.” The addition of the word operational is important as it makes operational innovation into an umbrella term that can include the full range of resources and activities that are developed to improve trial efficiency and function. This definition aligned thinking and served to underpin conversations about how best to develop and test operational innovations across the TICs and RIC.

At an in-person meeting of all TIC and RIC Principal Investigators (PIs) and Project Leads (PLs) in 2018, attendees were asked to identify the key activities important to improving the success of clinical trials. The conversation was captured, and the 39 responses provided were coded into seven categories: optimization of sites and site selection; acceleration of study startup; community involvement; development of technical resources; education and training; trial management; and trial design.

This began a focused discussion regarding priority innovations that continued in early 2019, with another in-person meeting that included a reminder of the categories identified at the 2018 meeting and a vote by PIs and PLs to prioritize related areas for innovation. The five areas chosen were as follows: Engaging Participants at the Operational Level of Clinical Trials; Study StartUp with Site Incentives and Gamification; Site Assessment; Pragmatic Trials with Limited Variables; and a Next Level Master Protocol.

Mapping and Sharing Operational Innovations

The 2019 meeting was also the start of a TIN Work Group that met to map operational innovations across the network in order to: position the network to review and consider ongoing work and plan future innovations; ensure innovative work was aligned across the TICs and RIC; allow the network to highlight innovations developed by the TIN internally and externally; and lay the groundwork for measuring the impact of innovations. The innovation development pipeline imagined by the Work Group is illustrated in Fig. 2.

Figure 2.

The Trial Innovation Network (TIN) innovation development pipeline began with known roadblocks and barriers. Innovations were developed, implemented in multi-center clinical trials, and then disseminated via a variety of communication pathways. Implementation and dissemination phases of the pipeline are pictured with continuous improvement cycles to indicate iterative improvement incorporating lessons learned.

The mapping exercise culminated in a repository of operational innovations derived from the literature, past experiences of TIC and RIC institutions, and work done at the TICs and the RIC and developed and/or implemented within the TIN. This repository was built for internal use to enhance transparency and ensure work across the Network was aligned and complementary. An overview of multi-center study roadblocks and barriers, TIN activities to support operational excellence in these areas, innovation categories, and innovations within each category is listed in Table 1.

Table 1.

An overview of multi-center clinical trial roadblocks and TIN activity and operational innovation to address these roadblocks

Roadblocks/Barriers	TIN Activities	Innovation Categories	Innovation Names
Flawed design	Consultations with clinical trial, disease, statistical, and recruitment experts	Study Design	1. Master Protocol
			2. Adaptive Design
			3. Design Labs [12]
			4. Expert Study Advice for Protocol Development
			5. Gaming & Incentives [20]
			6. Pragmatic Trial Design
Lack of stakeholder engagement	Specialized recruitment support	Site Selection	7. Site Assessment Survey Instrument
		Participant Identification	8. Patient Identification Improvement
Delays in study startup (e.g., IRB)	Single IRB (SIRB) support and oversight; Standard Agreement toolkits	IRB	9. Consent Builder [13]
			10. IRB Reliance Exchange (IREx) [18]
			11. One-Part vs. Two-Part consent documentation
			12. Operationalizing SIRB [17]
			13. SIRB Coordination During Study Startup
			14. Streamlining Local Context Review
			15. Faster Together [26]
		Study Startup	16. COVID trials rapid study startup in under 2 weeks
Fragmented operating models	Streamlined support to implement trials; Clinical Coordinating Center assistance	Site Startup	17. Accelerated Startup Program
			18. CTSA FDP Standard Agreements
			19. Operationalizing Study Startup
Recruitment challenges	Recruitment and retention planning support for effective enrollment	Enrollment	20. Clinician Study App
			21. Electronic Informed Consent (eConsent) [29]
			22. iConsent
			23. Informational Trial Video
			24. Key Site Startup Milestone Compensation
			25. Recruitment via Social Media
			26. Site Report Card
			27. Site Engagement/Feedback
			28. Guidelines for Developing Culturally Tailored Recruitment Materials [14]
			29. Measure of Trust in Biomedical Research in Diverse Populations [15]
High cost of data collection & management	Streamlined support to implement trials; Data Coordinating Center assistance	Risk Management	30. Impact of a study-specific training program on coordinator competency
			31. Risk assessment & risk management: the 3-3-3 approach
		Data Management	32. Global Electronic Management System
			33. Data Standardization
			34. Database Development Best Practices & Tools
			35. MyCap [25]
			36. REDCap: Clinical Data Interoperability Services

CTSA = Clinical and Translational Science Award; FDP = Federal Demonstration Partnership; TIN = Trial Innovation Network; REDCap = Research Electronic Data Capture.

To inform the development of the mapped information, a survey was distributed to 49 people including all TIN PIs, NCATS and Liaison Team Leadership, and all TIN Work Group PLs (response rate 90%). Respondents selected the term “Innovations Catalogue” to refer to the initiative, agreed on inclusion criteria and information to be contained within the catalogue, and suggested ways of incorporating the catalogue into existing TIN systems and practices. The qualitative themes that arose were: the importance of framing discussions to be inclusive of all innovations that are used to improve the conduct of clinical trials; support for doing a phased internal roll-out of the catalogue; balancing procedure and over-complexity; designing systems for implementation; focusing on TIN priority areas; and establishing a process to support innovation evaluation.

The Innovations Catalogue was implemented in a TIC- and RIC-wide accessible format in 2020 and presented to TIN leadership and PLs. In 2021, the Innovations Work Group presented again, this time on reframing the approach to innovation to be more inclusive of operational excellence. The operational innovation definition agreed to by TIN leadership was not changed; however, the Work Group wanted to be explicit in creating a more inclusive space for processes that might not be novel but had the potential to improve the conduct of clinical trials. The intention was to include valuable contributions (e.g., technologies, methods, processes, or paradigms) to clinical trial execution. While this broadened an already inclusive definition, operational innovations included in the catalogue were all focused on addressing specific barriers or inefficiencies and were tied to operational hypotheses that could be tested.

Implementing and Disseminating Operational Innovations

Operational innovations that moved from the idea stage through development and into implementation were defined as mature. Mature innovations were presented at cross-consortium Liaison Team meetings, shared with study teams in TIN consultations, published in manuscripts and white papers, and included in the publicly accessible TIN website Toolbox. A new method of identifying trials that included innovations was also developed. These trials were identified as Testing Operational Hypotheses trials and served to test innovations with the goal of improving clinical trial conduct and/or efficiency. Innovations were embedded in multi-center trials as an operational experiment, with expenses and required personnel covered by the TIC or RIC that developed the innovation.

Operational innovations were implemented either at the initial TIN consultation stage, when investigators had meetings with the TICs and RIC to discuss their support needs (e.g., Design Labs [12]) or within TIN-supported trials (e.g., Consent Builder [13]). The Catalogue included the innovation stage – design, development, or implementation – and the trials within which the innovation had been implemented.

Examples of implementation included a survey used to aid site selection (#7 in Table 1), a program to streamline informed consent across multiple sites (#9), the ability to connect trial data collection with electronic health records (#36), and systems implemented to manage risk (#31). Across time, as innovations matured, the number of operational innovations implemented within and across trials increased. Fig. 3 shows the number of innovations implemented, excluding innovations that are implemented in ways disconnected from trial design and conduct, for example, courses and templates that cannot be tracked [14,15].

Figure 3.

Split panel depiction of Trial Innovation Network (TIN) innovation implementation during the first TIN funding cycle 2016-2023. The left-hand side (3a) shows the number of innovations implemented, with lines indicating implementation at different stages of the clinical trial life cycle. The right-hand side (3b) showcases the number of Institutional Review Board (IRB) innovations as operationalizing single IRB (SIRB) was an early identified TIN objective.

Operational innovations were implemented within more than 110 TIN trials and over 40 disease areas, with some trials including a single innovation (e.g., SIRB coordination) and others including multiple innovations (e.g., accelerated startup program, gaming and incentives, one-part vs. two-part consent documentation, and streamlining local context review). While some of the innovations were implemented at the design stage pre-funding, the majority were implemented following grant submission and successful receipt of funding. Funders included NIH institutes and centers (e.g., NCI, NHLBI, NIAID, NIDDK, NINDS, and NCATS), PCORI, DOD, industry, and foundation grants.

Evaluation is ongoing, and metrics have been published for some innovations, while others are still being tested [12–26]. A TIN Work Group established to develop metrics agreed upon 10 priority metrics and data elements to measure clinical research study performance across the Network. These metrics are related to study startup, regulatory approval, the contracting process, patient accrual, and data quality. While further details are outside of the scope of this brief report, it is worth noting that the TIN is in the early stages of collecting cross-network metrics, with a goal to evaluate effectiveness of the innovations developed and implemented.

Facilitators, Barriers, and Lessons Learned

The process of defining and mapping operational innovations was instructive. An expert network of clinical trialists from five different organizations, each with their own specific environments and operational processes, came together to agree on a definition and structure for categorizing and describing operational innovations. These spanned the clinical trial life cycle, from study design through recruitment, and study conduct. It became clear that significant work was being done in complementary areas, and that alignment among them was important.

An example was work being done to support informed consent. All organizations involved in the TICs and RIC were undertaking activities related to supporting the informed consent process. These were complementary, ranging from support for streamlining the technical process of creating and updating informed consent forms, to informed consent training methods, to the development of key information sheets that are understandable and actionable [13,27]. The innovations in this area were developed into a toolkit that can be used flexibly in a fit-for-purpose way depending upon study needs [28].

The operational innovations developed and/or implemented within the TIN were designed to be broadly useful to multi-center clinical trial design and conduct. However, it is worth noting that there are elements of clinical trials that are determined by the funding source or sponsor and their requirements. This information should always be a reference point and may have an impact on the decision to implement an innovation.

Barriers to the implementation of innovations were time, cost, and concern about the innovation’s potential disruption of trial processes. The importance of early engagement with investigators to ensure their interest and support was a lesson learned. Efforts were made to guarantee operational innovations were included in a way that did not require additional PI time or expense and that innovations enhanced, rather than disrupted, trials.

The operational innovations prioritized in 2019 varied in the success of their implementation. Although study startup with site incentives/gamification and site assessment were both operationalized, implementation of a pragmatic trial with limited variables and Next Level Master Protocol has been a slower process. Engaging participants at the operational level of clinical trials was the most difficult innovation to implement given the upfront time and ongoing effort that would be required by trial investigators. Fewer foundational and more operational innovations found support and were likely to be incorporated into trials.

Future Directions

Developing and implementing innovations that address roadblocks and barriers to successful and informative clinical trials is essential to improving clinical trial quality and conduct. The development and implementation of innovations in the TIN have been extensive; however, there is still significant work to be done. Three areas for continued development are the implementation of foundational innovations, measurement of outcomes, and broad dissemination. The lessons learned in the first TIN funding cycle should inform future development in these areas.

A process for mapping innovations and tracking their implementation has been developed. These data were used to identify gaps and areas of complementarity and to prioritize future efforts. To date, implementation has been limited to operational innovations, while foundational innovations have been more difficult to implement. It may be useful to consider a separate pathway for foundational innovations that may not be easily embedded in a clinical trial. Work Groups that address the barriers and facilitators to implementation of foundational innovations, and that include stakeholders and policy makers, may help develop more successful pathways.

Establishing a national network, with aligned and efficient processes, is a significant effort. We developed processes for embedding innovations and for capturing priority metrics. However, we are only in the early stages of implementing and testing innovations at scale. We will need time and a focus on iterative process improvement to ensure that we capture the necessary information and develop appropriate statistical analysis plans.

Finally, although some pathways for dissemination have been developed, it is essential that they be expanded and that investigators are provided with support to implement the relevant innovations in their trials. This may mean an expansion of the TIN toolbox, continued work with partners at CTSA hubs to understand support needs, and internal and external meetings to highlight further areas of innovation. Dissemination and implementation methodologists could also be helpful in considering strategies for impactful and broad dissemination.

Funding statement

This content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The work was supported by the National Institutes of Health Center for Advancing Translational Sciences (NCATS) and the National Institute on Aging, in support of the Trial Innovation Network under grant numbers U24TR001579 (Vanderbilt University), U24TR001597 (University of Utah), U24TR001608 (Duke/Vanderbilt Universities), and U24TR001609 (Johns Hopkins/Tufts Universities).

Competing interests

None of the authors have competing interests related to this manuscript.