Building an infrastructure to support the development, conduct, and reporting of informative clinical studies: The Rockefeller University experience

Abstract

Introduction:

Clinical trials are a vital component of translational science, providing crucial information on the efficacy and safety of new interventions and forming the basis for regulatory approval and/or clinical adoption. At the same time, they are complex to design, conduct, monitor, and report successfully. Concerns over the last two decades about the quality of the design and the lack of completion and reporting of clinical trials, characterized as a lack of “informativeness,” highlighted by the experience during the COVID-19 pandemic, have led to several initiatives to address the serious shortcomings of the United States clinical research enterprise.

Methods and Results:

Against this background, we detail the policies, procedures, and programs that we have developed in The Rockefeller University Center for Clinical and Translational Science (CCTS), supported by a Clinical and Translational Science Award (CTSA) program grant since 2006, to support the development, conduct, and reporting of informative clinical studies.

Conclusions:

We have focused on building a data-driven infrastructure to both assist individual investigators and bring translational science to each element of the clinical investigation process, with the goal of both generating new knowledge and accelerating the uptake of that knowledge into practice.

Introduction and Background

Clinical trials are a vital component of translational science, providing crucial information on the efficacy and safety of new interventions and forming the basis for regulatory approval and/or clinical adoption. At the same time, they are complex to design, conduct, and report successfully since they must meet stringent bioethical, statistical, and regulatory requirements, and be able to recruit the required number of participants in a prescribed time period.

Concerns about the lack of completion and reporting of clinical trials led to the 2007 Food and Drug Administration Amendment Act (FDAA), which mandated registration of trials and the timely reporting on the ClinicalTrials.gov website [1]. Despite the FDAA, in 2013 Nguyen et al. reported that only 17% of cancer randomized clinical trials were either reported on ClinicalTrials.gov or published 12 months after study completion or termination [2], and Gordon et al. reported that only 23% of National Heart, Lung, and Blood Institute (NHLBI) clinical trials were published within 12 months [3]. In recognition of the challenges in designing and conducting successful clinical trials, the NIH Collaboratory, initiated in 2012 to develop pragmatic clinical trials, implemented a two-phase development program in which studies were first provided with extensive support from core working groups to refine the protocols, after which the trials were finalized and evaluated for financing [4]. Similarly, NHLBI in 2018 introduced a biphasic, milestone-driven mechanism of clinical trial approval comprising a start-up phase to refine the protocol followed by a clinical trial execution phase [5]. The CTSA program responded in 2017 by creating the Translational Innovation Network to serve as a laboratory to study and improve the clinical trial process, building into the network a series of services, consultations, and pilots studies to ensure that protocols meet high standards [6].

Zarin et al. in 2019 published a commentary entitled, “Harms From Uninformative Trials” in which they defined an uninformative trial as one in which the results are not of meaningful use for a patient, clinician, researcher, or policymaker [7]. They identified five things that are necessary for a study to be informative (Table 1), focusing on the study hypothesis, design, feasibility, analysis, and reporting. They pointed out that while IRBs assess risk/benefit, they are often unable to assess scientific merit beyond that needed to justify risk. While NIH peer review of clinical trials and the programs described above undoubtedly improve the quality of studies, NIH-supported trials represent only a small portion of all clinical trials. In fact, Zarin et al. reported that in March 2019, there were 9,484 open clinical trials registered in ClinicalTrials.gov that were enrolling over 5 million American participants and had no evidence of external funding. They therefore called on academic institutions to take on the responsibility of scientific review. Most recently, the COVID-19 pandemic raised additional concerns about uninformative trials, with Bugin and Woodcock reporting that only 5% of the clinical trials testing drugs as COVID-19 treatments were randomized and adequately powered to provide clinically meaningful data [8].

Table 1.

Conditions for trial informativeness [49–52] with Rockefeller additions of “Return of Research Results,” “Data Sharing,” and “Representative Enrollment” (column 1), Rockefeller University Center for Clinical and Translational Science (CCTS) Departments and cores (column 2) processes to support attaining the goals (column 3), metrics for assessing the outcomes (column 4) and selected performance indicators (column 5)

Conditions for trial informativeness	Departments and cores	Processes	Metrics	Performance
1. Importance: Trial hypothesis is likely to inform an important scientific, medical, or policy decision [53–58]	Research Facilitation Office Biostatistics design consultation Community engagement	TRN Required Public Health Impact Statement Epidemiology course ACCTS review CAB review	TRN Time in TRN until submission Number of protocols terminated in TRN Time from IRB submission to approval ACCTS Percent fully approved Percent approved with stipulations Number of stipulations Percent tabled IRB Time to approval Percent approved in < 30 days	TRN (2022) Median 16 days (mean 48 days) TRN terminated 6 protocols* Median 5 days (Mean 26 days) ACCTS (2018–2022) 75% fully approved 20% approved with stipulations Median: 3 4% tabled IRB (2018–2022) All protocols: mean 20 days and 80% <30 days Full review, mean 32 days and 67% <30 days
2. Design: Trial methods are likely to provide meaningful evidence related to study hypothesis [59–63]	Biostatistics Rockefeller Scientific Resource Centers TSE experts relevant to specific design	TRN R3 lectures and resources ACCTS Review
3. Feasibility: The trial is likely to be feasible [64–67]	Research Facilitation Office Community engagement Comprehensive Recruitment consultation and ongoing support	TRN ACCTS Review CAB review
4. Integrity: Trial is conducted and analyzed in a scientifically valid manner that is faithful to design [68,69]	Research Nursing Research Pharmacy Hospitalist Research coordinators Research Bionutrition Biostatistics/Bioinformatics IT oversight of data security and management Database management Clinical Research Support Office Research Facilitation Office Research Participant Advocate Good Clinical Practices Workgroup	Accrual Index CCTS early audit program Internal monitoring Real-time sharing of reported protocol deviations to TSEs IRB Continuing Review ACCTS Continuing Review
5. Reporting: Systems are in place to ensure timely, complete and accurate reporting [70,71]	Clinical Research Support Office Biostatistics Bioinformatics	Regulatory support for ClinicalTrials.gov reporting IT Database management and data queries DSMP review	Number of Problem Records on ClinicalTrials.gov register	0 records with problem status in 2023
6. Return of Results (RoR): Investigators are counseled on appropriate return of aggregate and individual results, including actionable genetic information	Clinical Research Support Office Research Facilitation Office	IRB application requires plan for return of results ICF includes return of results CRSO templates and assistance RU Informed Consent Materials for DNA sequencing research studies	RoR plan part of IRB review RoR in ICF part of IRB review	Required for IRB approval Required for IRB approval
7. Data Sharing	Biostatistics Bioinformatics IT Scientific Resources Center support	TRN – plan for good data management and reporting R3 lectures and resources ACCTS review	Study design, analysis plan required element of TRN	Required for ACCTS approval
8. Representative Enrollment	Clinical Research Support Office Research Facilitation Office Partnerships with Community-Based Organizations and Patient Advocacy Groups	TRN Recruitment consultation and support IRB review ACCTS review	Recruitment Number of consults Percent of protocols with substantive recruitment stipulations Representative enrollment	Recruitment Consults part of TRN Tracking just begun For representative enrollment data, see Table 4.

ACCTS, advisory committee for clinical and translational science; IRB, institutional review board; TSE, translational science expert/educator; CAB, community advisory board; DSMB, data safety and monitoring board; IT, information technology; TRN, translational research navigation program; R³, research rigor, reproducibility, and reporting program; DSMP, data safety moniroting plan; ICF, informed consent form; CRSO, clinical research support office; RU, Rockefeller University

The NIH General Clinical Research Center (GCRC) program, which was inaugurated in the 1960s and terminated when the Clinical and Translational Science Awards (CTSA) program began in 2006, required that recipient institutions create a GCRC Advisory Committee (GAC) to provide a scientific review of protocols to be conducted in the center and that the GAC also review a Data Safety and Monitoring Plan (DSMP) for each study. The DSMP is designed to be a quality assurance plan for the study, encompassing subject safety, data integrity, subject privacy, data confidentiality, product accountability, study documentation, and study coordination. With the advent of the CTSA program, the requirement for a GAC was eliminated, but in 2015 a CTSA Consensus Working Group recommended that institutions create Scientific Review Committees (SRCs) and specified the composition and function of such a committee [9]. Five years later, Selker et al. reported on the implementation or modification of SRCs at 10 institutions, stressing the importance of having a clear mandate from institutional leadership and at the local level, as well as clarity on integrating procedures and responsibilities with the IRB [10].

This background provides a framework for presenting the policies, procedures, and programs that we have developed in the Rockefeller University Center for Clinical and Translational Science (CCTS), supported in part by a CTSA grant since 2006, to assist in the development, conduct, and reporting of informative clinical studies.

The Rockefeller University Clinical and Translational Science Program

Overview

Rockefeller University opened as a research institute with the mission, Scientia pro bono humani generis (Science for the benefit of humanity) in 1901 and in 1910 the Rockefeller Institute Hospital opened as the first research hospital in the United States. From its beginning, the Hospital reflected the vision of its first Director, Rufus Cole, one of William Osler’s trainees at Johns Hopkins Medical School and Hospital, to create a cadre of physician-scientists who would evaluate patients at the bedside, at the autopsy table, and most importantly, at the laboratory bench. The guiding principles were that all patients were research patients, and thus not charged for their hospital or medical care, and all physicians were salaried and did not receive payment for their medical services so that they could devote themselves to the research mission. The Rockefeller Institute became The Rockefeller University in the 1960s when it began conferring PhD degrees, and in 2006, it began conferring Master’s of Clinical and Translational Science degrees when the KL2 Clinical Scholars program for junior translational scientists, was developed under the CTSA grant [11].

Today, Rockefeller University is comprised of approximately 70 separate laboratories, each headed by a senior scientist, most of whom hold PhD degrees and engage in basic science. A small number of labs are led by MD/PhD and MD investigators who variably divide their efforts between basic and clinical studies. The core educational experience for the KL2 Clinical Scholars is experiential, namely designing, conducting, and reporting a human participant protocol [11]. Many of the KL2 Clinical Scholars join basic science labs where they use their clinical skills and experience to initiate translational research programs that build on the basic science discoveries from that lab. To encourage basic scientist participation in human participant translational research and assist Clinical Scholars who have limited experience in conducting human studies, we developed a series of programs to help both groups develop, conduct, analyze, and report informative and impactful studies and to identify stakeholders and potential community-based partners to aid in study design and conduct, as well as dissemination and implementation of study results.

Translational Research Navigation (TRN) Program

Overview: This program is led by a senior core of Translational Science Experts/Educators (TSEs) who interact extensively with each other on a weekly basis to assist investigators in developing, conducting, and analyzing their studies (Table 2). TRN originally focused on protocol development [12], and was later expanded to foster standards across the full life cycle of a protocol. The TSEs reviewed the criteria proposed by Zarin et al. to ensure that protocols are informative and collectively decided to add three additional criteria: (1) “Return of results.” This was based in part on data derived from the Research Participant Perception Survey (RPPS) developed at Rockefeller [13,14], which have consistently demonstrated the high-value research participants place on receiving research results, as well as recommendations from the Secretary’s Advisory Committee on Human Research Protection, the revised Common Rule, and the landmark 2018 National Academies of Sciences, Engineering, and Medicine report recommending a paradigm shift for return of research results [15–17]. Thus, return of results, which has often been neglected, is not only a crucial element of dissemination but also an important element in showing respect for research participants as partners in the research process and demonstrating that the institution is worthy of participant and public trust [18–20]. (2) “Data Sharing.” This was based on the recognition that beyond reporting aggregate results, there is a need for sharing primary data and metadata to ensure transparency and reproducibility of research results and to facilitate the conduct of additional analyses (e.g., meta-analysis) by other investigators. (3) “Representative Enrollment.” This ensures that studies are conducted with participants who accurately reflect the target population and that individuals in the target population have access to participation.

Table 2.

Participation of Translational Science Experts/Educators (TSEs) Comprising the Translational Research Navigation Team in Activities of the Rockefeller Center for Clinical and Translational Science (CCTS).

Translational science experts/educators’ title	ACCTS meeting	Senior staff meeting	Team science leadership review	IRB
PI/PD and Physician-in-Chief	√*	√	√	√
Clinical Research Facilitation Leader	√	√	√	√
Director of Nursing	√	√	√	√
Hospital Information Manager	√	√	√	–
Director of Regulatory Affairs/HIS	–	√	√	–
*Clinical Research Officer	√*	√	√	√*
PI/PD and Hospital CEO	√*	√	√	√*
Pharmacy Director	√*	√	√	√*
Medical Director	√*	√	√	√*
Bionutrition Director	√	√	√	√
IRB Chair and KL2 Director	√*	–	√	√*
Lead Community and Collaboration	√*	√	√	–
Bioinformatics Director	√	√	√	-
Biostatistics Director	√*	√	√	√*
Chief Operating Officer/CTSA Administrator	√*	√	√	√
Basic Science Faculty members	√*	–	√	√*

^*Voting member.

ACCTS, advisory committee on clinical and translational science (monthly meeting); CEO, chief executive officer; HIS, hospital information systems; IRB, institutional review board (monthly meeting); PI/PD, CTSA principal investigator/program director; senior staff meeting (weekly meeting); TSLR, team science leadership review (periodic meetings)

These eight criteria are listed in Table 1, along with the measures we take to ensure that we address each of them. To support several of the criteria, and in recognition of the increasing importance of scientific rigor in research design, analysis, and reporting, the TSEs created a University-wide program to enhance rigor, reproducibility, and reporting, R³ [21], with active support from data management experts in the University’s library. R³ sponsors a series of presentations throughout the year and maintains a website with background materials; NIH and NSF requirements and resources; a group of available aids (guidelines, methods, and checklists); and research support resources and tools, including open science framework, navigation tools, and a data sharing wizard.

We are committed to incorporating the views and priorities of diverse, under-represented communities in our research and achieve this in part through a wide variety of outreach methods to engage special populations and our 15-year collaboration with Clinical Directors Network (CDN), an awarding-winning Practice-Based Research Network that performs clinical investigation to high standards in collaboration with Federally Qualified Community Health Centers and other safety-net primary care practices. The president of CDN, Dr Jonathan N. Tobin, serves as the Co-Lead of the Rockefeller University Community and Collaboration Core. Thus, studies performed over the past years have included vulnerable older adults from minority populations under-represented in research and engaged practicing clinicians and other staff workers in FQHCs, older adult centers, and other community-based health and social service providers. These valued research collaborators diversify the research workforce and broaden the research perspective, sensitizing investigators to service delivery concerns and the barriers and facilitators to implementation that exist in a wide range of medical care settings.

Through a variety of mechanisms, investigators at Rockefeller have also built relationships with special populations, including patients with Fanconi anemia, Down syndrome, neurodevelopmental disorders, rare immunodeficiencies, fibrolamellar hepatocellular carcinoma, hidradenitis suppurativa, several different malignancies, COVID-19, and defects in facial recognition, as well as LGBTQ persons and women in the criminal justice system at high risk of developing HIV-1 infection. Our institution also conducts studies of normal physiology in healthy individuals.

Fig. 1 provides an overview of the roles of the TSEs, TRN, and R³ in supporting the multidisciplinary teams in developing, conducting, analyzing, and reporting their studies, as well as the role of the TSEs in workforce development. Our continuous quality improvement program is driven by metrics of the performance of each element in TRN and the outcome data about the quality of the protocols as expressed by our research participants.

Figure 1.

Overall organization of the Rockefeller University Center for Clinical and Translational Science, emphasizing the vital roles of the Translational Science Team Experts/Educators (TSEs) in supporting multidisciplinary research teams through 1. translational research navigation (TRN), 2. providing services to support protocol conduct, 3. leading workforce development educational programs, and 4. using CCTS metrics and novel outcome data to drive continual process improvement.

Protocol development: The TRN process begins with a lead TSE with extensive experience in protocol development and conduct under Good Clinical Practice (GCP) exploring the scientific hypothesis with the investigator and other TSEs who provide guidance on 1. how to engage communities in all phases of the research; 2. key considerations related to patient safety, biostatistics, bioethics, research nursing support, bioinformatics, pharmaceuticals, regulatory requirements, and data management, security, and reporting; 3. how to articulate a hypothesis that operationalizes a scientific question in a clinical and/or community context; 4. alignment of aims and outcomes; 5. robust study design, including calculations of power and sample size; 6. participant recruitment feasibility and the likely time needed to complete recruitment into the study, as well as the best methods to identify eligible participants [12,22]. The TRN process also brings IRB leadership into early discussions when required, as well as the Research Hospitalist [23], a position that our Hub created to provide medical support to studies conducted by basic scientists and ensure the safety of research participants by reviewing the medical aspects of all protocols and compliance of novel research procedures and devices with infection-control and electrical engineering standards [23]. Additional multidisciplinary TRN protocol development meetings are held until the protocol is refined and judged by the investigators and TSEs, as well as the clinical research coordinator responsible for the protocol’s conduct, as ready for formal review. Investigators are encouraged but not mandated to accept recommendations from TRN staff. Protocols that do not meet high standards of validity, feasibility, bioethics, safety, or other key considerations despite TRN support are terminated before submission. Protocols that meet the standards are submitted for review through iRIS (Cayuse Data Corporation), a protocol development, review, and conduct electronic software application that the TSEs have customized over many years to support research at Rockefeller.

Protocol review: Submitted protocols undergo two rounds of review:

1. The scientific review is conducted by the Advisory Committee for Clinical and Translational Science (ACCTS), which is composed of TSEs, university faculty engaged in clinical and basic research, university legal counsel with expertise in contracting and conflict of interest policies, and the two Clinical Scholars selected as Chief Clinical Scholars (Table 2). The ACCTS starts by analyzing the hypothesis, aims, and outcomes for scientific and medical validity and novelty. The selected endpoint measures and the study design and plan for analysis are reviewed by scientists, clinicians, and the biostatistician. For studies that incorporate nucleotide sequencing and analysis, the bioinformatician is consulted. The ACCTS also reviews the data management plan; the required research nursing and research pharmacy support; funding, space, and other aspects of feasibility; and participant safety and data and safety monitoring. ACCTS also reviews resource utilization and fairly apportions resources when there are competing demands. If the scientific hypothesis is not sufficiently supported or justified, or if there is insufficient information about the relevance and validity of the measures, the sample size, and the likelihood of a meaningful finding, the protocol is either returned to the investigator with one or more stipulations that need to be addressed as a condition for later approval or tabled because the protocol needs extensive revision. In addition, some protocols that do receive full approval are returned to the investigator with non-binding recommendations for improvement. Thus, ACCTS policies and procedures meet all of the recommendations detailed in the CTSA Consortium Consensus Scientific Review Committee Working Group Report on the SRC Process [9].

2. The IRB reviews the protocol for ethical design, regulatory compliance, safety, and other aspects of human protections, with a focus on the clarity of consent. The IRB also reviews the plan for Return of Results to study participants and the community, as well as recruitment feasibility, and the plan for, and likelihood of achieving, representative enrollment. Protocols without credible recruitment plans are returned for revision, which usually involves a comprehensive recruitment consultation. The recruitment core utilizes a data-rich approach [22] to identify successful strategies, achieve representative enrollment, and predict achievable accrual [24]. The IRB also makes an independent assessment of conflict of interest issues.

The IRB chair and staff participate actively in the TRN process, meeting with investigators early during protocol development to address issues that may arise during the IRB process. The chair is also a member of the ACCTS, providing an opportunity to participate in the scientific review process and identify issues that may impact the IRB review. To ensure that the dual review process by ACCTS and the IRB does not delay the approval of protocols that are complete, the ACCTS meets the day prior to the IRB meeting. There is broad overlapping membership between ACCTS and the IRB (Table 2) and members of both committees can view the stipulations of both committees so that there is excellent communication between the committees.

Additional safeguards to ensure that approved protocols continue to meet our quality standards include, when available, assignment of experienced research coordinators to protocols; IT assistance with developing REDCap databases to record research information; early audits (i.e., after the first few participants are enrolled) of studies led by Clinical Scholars or new PhD investigators and studies that are complex or considered high risk; ongoing review of protocol recruitment progress, with the research team and recruitment staff communicating regularly to assess recruitment success and modify approaches to support timely enrollment [24]; and at least yearly review by ACCTS and the IRB. Information obtained from the above review processes informs the design of educational activities, including a bi-monthly GCP newsletter sent to all investigators that identifies emerging issues in GCP and reinforces proper clinical practice, including, where needed, systemic corrections such as changes to templates, policies, or workflow. This corrective information is then incorporated into the TRN process to proactively prevent recurrence (Fig. 1).

As indicated above, TRN was expanded beyond protocol development [12] to include Community Engaged Research Navigation [25], Protocol Initiation/Implementation Navigation, Protocol Conduct Navigation, and Protocol Completion Navigation, including assistance with reporting of results to Clinicaltrials.gov and return of results to participants. It thus provides a comprehensive structure to maintain quality control. Separate from their TRN advisory activities, as indicated in Fig. 1, TSEs also provide important services to directly support the conduct of protocols and play vital roles in educating the Clinical Scholars and assessing their progress in mastering Team Science Leadership competencies [26]. Initiatives under development include opportunities for Clinical Scholars to participate in, and lead, research teams that include practicing clinicians and other service providers, including gaining experience in training community-based collaborators in human participant research while learning about community health needs directly from clinicians and their patients. This experience will inform future implementation science studies that design and compare alternative strategies to implement, scale up, and sustain innovations developed in the laboratories at Rockefeller.

Recruitment

One of the most common reasons for studies to fail to provide informative information is the inability to recruit sufficient numbers of individuals into the study. We found that inexperienced investigators often did not begin to think in detail about their recruitment plan until their protocol was approved by the ACCTS and IRB and almost always underestimated the challenges to timely recruitment, the required budget, and time to achieve full recruitment. To address this, we instituted a comprehensive recruitment program [22] comprised of the following five elements. 1. Comprehensive Consultation early in the design of the protocol. As part of TRN, all investigators are strongly encouraged to obtain a comprehensive consultation from the members of the Recruitment Core early during protocol development. This provides an opportunity to educate the investigator about the “leaky pipe” concept, in which they can anticipate the loss of eligible participants at multiple stages in the recruitment process, and how to anticipate the likely loss at each stage based on the population being recruited, the stringency of the inclusion and exclusion criteria, the benefits and burdens of the study as assessed by potential participants, and the effectiveness of the advertising campaign. The Recruitment Core also helps investigators develop a realistic assessment of the likely time to achieve full recruitment based on the above information as well as the availability of the investigator and team members related to holidays, attendance at professional meetings, and other competing demands on their time. Finally, the Recruitment Core suggests a specific advertising campaign, including the production of advertisements that are designed to enhance equitable enrollment by ensuring that the images and wording are appropriate for a diverse group of participants, and provides a partial subsidy of the advertising. 2. Recruitment plan. Investigators are required to include a detailed recruitment plan in their protocol, including where appropriate, identifying stakeholders and collaborators who can both facilitate recruitment and support later dissemination and implementation activities. The ACCTS focuses on whether the proposed plan is realistic and the IRB focuses on whether the strategy will offer fair access to representative populations in deciding whether to approve the protocol. 3. Participant Repository. We recognized the importance of being able to contact individuals who previously volunteered to participate in a study at Rockefeller to assess their willingness to participate in a future study [22]. This is especially valuable for our CTSA hub because many of our studies involve healthy participants. In response, we developed an IRB-approved protocol to establish a Research Volunteer Repository that permits individuals who agree to add their names and some demographic data to the Repository to be recontacted about participation in future research studies. Of note, ∼95% of individuals who contact Rockefeller to inquire about their eligibility for a study agree to join the Repository, and the Repository now contains more than 10,000 individuals. One metric that highlights the value of the Repository is that 62% of all Repository members have enrolled in at least one study. 4. Centralized Contact Center. The Recruitment Core supports individual studies by providing personnel to receive contacts from potential participants who respond to advertisements for the study. The staff member engages the caller, collects basic demographic and referral information, and then assesses their eligibility for one or more studies by reviewing a list of prescreening criteria. Volunteers who meet the criteria are scheduled or referred for screening by the investigator. The staff follows an IRB-approved script to offer enrollment into the Repository, regardless of the prescreening outcome. When individuals do not meet the criteria or decline participation, the staff member records the reason and this information is compiled and shared with the investigator to inform whether it would be beneficial to modify the advertising or the protocol to improve recruitment into the study. 5. The Accrual Index. Based on the Recruitment Core’s extensive experience, a realistic estimate is made of the total time likely to be required for full accrual of the study [24]. From this, the expected fractional level of recruitment at any time during the study can be estimated. This then is converted into a dashboard that allows the Recruitment Core to monitor studies for the expected progress toward completion. When studies deviate from the expected timeline, the investigators and recruiting staff analyze the reason(s) and make appropriate modifications in the protocol or recruitment strategy, including, where appropriate, identifying additional academic and community partners.

Contracting and Conflict of Interest

A dedicated university counsel oversees clinical trial contracting, material transfer agreements, and the university’s conflict of interest program. She serves on both the ACCTS and the IRB, facilitating the flow of information on both topics to the appropriate committees.

Regulatory Requirements: Clinical Trials Registration, FDA Documents, Monitoring, and Auditing

The Clinical Research Support Office (CRSO) assists investigators in registering their trials on ClinicalTrials.gov and in reporting the required updates when they are due. They also assist investigators in interacting with the FDA to obtain and maintain INDs. The CRSO also oversees compliance with internal monitoring of studies, as well as institutional not-for-cause audits of studies, including the early auditing of studies so that corrective actions and training can be implemented as soon as possible.

Research Participant Perception Survey (RPPS)

The motivation to create the RPPS came from a desire to have an outcome measure of the quality of the investigative team’s ability to conduct clinical studies to high standards as judged from the perspective of the individuals who participated in the studies. This was in reaction to realizing that successfully completing a quiz after reading an online human research protection training module, which was the nearly universal approach adopted by institutions in response to the mandate for training in human research protections starting in the early 2000s, was at best a weak measure of quality. This led us to work with colleagues at eight NIH-funded CTSA hubs, GCRCs, and the NIH Clinical Center to develop the first version of the validated RPPS to assess the perceptions of individuals who actually participated in the study [27]. In addition to overall assessments of the participants’ experience and whether they would recommend participation to others, more detailed questions probed whether the informed consent process fully prepared the participant for what they experienced during the study, whether members of the investigative team built trust and mutual respect with the participant, and whether the investigative team was available when needed. The survey underwent extensive validation analysis including deployment in a study conducted at 15 NIH-supported institutions that reported on the responses from a diverse group of almost 5,000 participants [13,14]. A number of key conclusions came from that study, including the desire of participants to receive information about the results of the study, the central role of altruism in motivating participation, and the importance participants place on feeling that they are partners in discovery. Later research was targeted to shortening the study based on a careful analysis of the incremental value of each question in the original version and to comparing alternative methods of deployment to maximize the response rate [28–30].

The RPPS continues to be deployed at Rockefeller as a crucial outcome measure that now can be analyzed across multiple dimensions, including individual protocols, race, ethnicity, gender, age, study type, and study intensity or duration. This provides vital information for process improvement as the data are reviewed with the Senior Staff, individual investigators, and community and participant stakeholders. Specific interventions have been developed in response to data obtained from the RPPS, including enhancing educational tools to improve the informed consent process in complex studies, emphasizing clear communication, and requiring a prospective plan for return of results to participants. Thus, the RPPS facilitates a robust performance improvement cycle with the potential to monitor the impact of interventions by analysis of subsequent surveys. Most recently, Rockefeller has led an NCATS-funded 6-CTSA collaboration to develop a shared infrastructure to streamline adoption of the RPPS at other institutions and build a learning collaborative, data aggregation platform and dashboard that can be used for benchmarking [30].

Evaluation and Metrics

Our evaluation of the effectiveness of our program has undergone continual evolution and that evolution continues as we introduce new programs and appreciate subtleties in the complexity of interpreting specific metrics. We are well aware of what has been termed the tyranny of metrics [31], wherein attempts to improve the metric result in gaming the system and perverse unintended consequences, now enshrined as Goodhart’s Law (as modified by Strathern), “When a measure becomes a target, it ceases to be a good measure [32].” To address these concerns, we think the value of a metric should be judged based on the answers to the following questions. 1. To what use will the metric be put? 2. Who will use the metric and how will they use it? 3. What are the implicit assumptions underlying the metric? 4. Are those assumptions supported by evidence? 5. Are there any risks that the metric will distort decision-making in a way that would have a negative impact on optimal productivity? 6. How high is the priority of the metric proposed? 7. Is there a related metric that would have a higher priority? 8. Are there automated or minimally obtrusive methods to capture the needed data?

As a result, our primary focus is on trying to refine outcome measures that will provide us with the most valuable information on whether we have achieved our goals, and where we fall short, how we can best improve our processes and policies. We rely on process and surrogate measures when we do not have data on outcomes because they are not yet attainable, or they are not practical to obtain. And we rely on utilization measures to assess which of our services are being utilized so that we can adjust our resources to best meet the demand. Table 3 contains a list of metrics we use geared to this taxonomy and Table 1 contains a selection of these metrics and data from our experience.

Table 3.

Metrics for judging study informativeness

Outcome-Based Metrics: Improved health: 1. New FDA-approved drug, device, or diagnostic 2. Change in clinical practice based on results as judged by practice guidelines, textbooks, and review articles by authorities in the field Quality of submitted protocols: 1. Time to approval by ACCTS and IRB 2. Number and nature of stipulations and requirements from ACCTS and IRB Quality of participant recruitment plan: Adherence to expected rate of recruitment based on Accrual Index Quality of Informed Consent Process: Responses to the following questions in RPPS: Did the informed consent form prepare you for what to expect during the study? Did the information and discussion you had before participating in the research study prepare you for your experience in the study? During your discussion about the study, did you feel pressure from the research staff to join the study? Quality of Investigative Team’s ability to build trust with participants: Responses to the following questions in RPPS: Rate your overall experience in the research study (0-10). Would you recommend joining a research study to your family and friends? Did the research team members listen carefully to you? Did the research team members treat you with courtesy and respect? Did you feel like you were a valued partner in the research process? Quality of study conduct Protocol deviations Protocol violations Protocol stops Reports to regulatory agencies based on protocol deviations and violations Quality of Services provided by CCTS TSEs (Biostatistics, Bioinformatics, Navigation, Research Nursing, Research Pharmacy, Bionutrition): Yearly surveys by Clinical Scholars and other users Study Completion: Yes, before or at time predicted Yes, after time predicted No, but ongoing No, and abandoned Timely reporting of results: Reporting to ClinicalTrials.gov up-to-date or completed Reporting to scientific community via presentations and publications Reporting to research participants in accord with return of results plan Selected process-based and surrogate metrics Invention disclosures, patent application filings, patent awards, technology licenses Scientific publications and presentations Number of basic scientists who have undergone TRN and lead clinical studies Completion of required training by investigators Signing and dating of informed consent forms in accord with GCP Registration of studies on ClinicalTrials.gov prior to initiating the study Timely completion of study monitoring and auditing Utilization Metrics Number of consultations and related activities of TSEs in support of the research program

FDA, Food and Drug Administration; ACCTS, adivsory committee on clinical and translational science; IRB, institutional review board; RPPS, research participant perception survey; CCTS, center for clinical and translational science; TSEs, translational science experts/educators; GCP, good clinical practice

Outcome measures: The fundamental goal of an informative study is to improve human health; this can be measured by whether the study leads to FDA approval of a new drug, medical device, or diagnostic method, or whether it results in any measurable changes in clinical practice and public health statistics. For example, pioneering studies supported by the CTSA-funded infrastructure conducted in the Rockefeller University Hospital on the pathogenesis of psoriasis, and later the safety and efficacy of several different novel agents to inhibit T-cell activation, IL-23, and different isoforms of IL-17 family cytokines, led to the approval of multiple drugs that have dramatically improved the therapy of psoriasis and psoriatic arthritis [33–40]. These initial studies in psoriasis also led to these new drugs being used in other autoimmune diseases, including Crohn’s disease, ulcerative colitis, rheumatoid arthritis, and Type 1 diabetes [41,42]. Impact on public health statistics can be obtained from analyzing ongoing regional or national surveys, for example, those conducted by the CDC National Center for Health Statistics, state and municipal health departments, and “big data” electronic health record repositories.

Measures of the quality of a protocol include whether it is approved by the ACCTS and IRB and the time required for approval. From 2018 to 2022, 75% of protocols submitted to the ACCTS were approved on initial review and 20% received conditional approval with stipulations; all of the latter gained approval after revision and resubmission (Table 1). Fewer than 4% of protocols submitted to the ACCTS were tabled for significant deficiencies. Investigators with tabled protocols returned to the TRN process and submitted revised protocols that were granted approval within 1-3 months.

Data on the time to IRB approval from protocol submission were collected from 2012 to 2018 as part of the CTSA Common Metrics program and demonstrated median times ranging from 10 to 26 days (average 17 days; n = 108). For comparison, the median times for all of the hubs in the CTSA consortium for the years 2015–2017 ranged from 42 to 45 days. More recently, for the years 2018–2022, the mean number of days from submission to approval for all protocols at Rockefeller (expedited and full review) was 20 days, and for those undergoing full review, 67% received approval in < 30 days, with a median of 20 days and a mean of 32 days. We now also qualitatively review data on specific ACCTS and IRB stipulations and requirements for reviewed protocols and use the data to inform educational initiatives and for incorporation into the TRN to improve the program.

To assess the quality of the informed consent process and the ability of the study team to build trust with the participants, we review carefully the RPPS responses to the selected validated questions below. Also indicated below are the percentages of participants in studies conducted in 2022 selecting the indicated rating. The 142 respondents were 49% female, 21% Hispanic, 72% White, 24% Black, 4% Asian, 2% American Indian/Alaska Native, and 1% Native Hawaiian or other Pacific Islander. The response rate was 25%, which is comparable to the 26% response rate for the patient experience survey used by the Centers for Medicare and Medicaid Services [43].

Informed consent:

• Did the informed consent form prepare you for what to expect during the study? 93%, “Completely”

• Did the information and discussions you had before participating in the research study prepare you for your experience in the study? 92%, “Completely”

• During your discussion about the study, did you feel pressure from the research staff to join the study? 91%, “Never”

Building trust:

• Rate your overall experience in the research study (0–10). 85% chose grades 9 or 10

• Would you recommend joining a research study to your family and friends? 76%, “Definitely yes”

• Did the research team members listen carefully to you? 94% “Always”

• Did the research team members treat you with courtesy and respect? 97%, “Always”

• Did you feel like you were a valued partner in the research process? 92%, “Always”

To assess the representatives of recruitment, we compare the race and ethnicity of participants in our studies and in our Research Volunteer Repository over the past five years to demographics from census data for New York City in 2022 (Table 4). The sex distribution of our participants matches the New York City data, with males constituting 46.3% of our participants compared to 48.0% citywide. Since almost 23% of our participants are in the Unknown race category compared to almost 15% for New York City as a whole, all of our categories for race, which are expressed as a percentage, need to be adjusted upwards to account for this difference when comparing to New York City data. While our racial data from 2018 to 2022 generally track the New York City data, our enrollment of Black individuals is somewhat less than both the citywide data and the data from our own Repository. Similarly, the enrollment of Latino/Hispanic individuals was proportionately less than in the citywide data and data in the Repository. Since our latest data are from 2018 to 2022, it is possible that the COVID-19 pandemic, which disproportionately impacted the Black and Latino/Hispanic populations in New York City, affected their participation in our studies. This is an area under active analysis.

Table 4.

Demographics in percentages of New York City in 2023, the Rockefeller University (RU) Research Volunteer Repository, and research participants in studies conducted at Rockefeller from 2018–2022 (n = 3,857)

Race/ethnicity/sex	New York city	RU Repository	RU enrollment (2018–2022)
White	39.8	31.2	43.3
Black	23.4	35.2	18.0
Other	0.6	10.4	8.4
Asian	14.2	4.6	8.8
More than one race	7.1	3.7	5.4
Unknown	14.9	11.7	22.9
Latino/Hispanic	28.9	18.6	12.5
Male	48.0	53.6	46.3

To assess the quality of study conduct, we analyze protocol deviations, violations, stoppages, and reporting to regulatory agencies. When appropriate, full root cause analyses are performed to identify potential underlying weaknesses in policies and procedures and institute measures to prevent their recurrence. We also review protocol variance and audit data to identify issues recurring across studies that may afford opportunities to improve education, forms, policies, workflow, or policies for systemic impact.

Process and surrogate measures: Table 3 also contains a series of metrics that provide valuable information on translational science activities and the proper conduct of clinical studies. Thus, data on intellectual property, publications, and presentations give some insight into translational discoveries and where they are along the path to improving the health of communities. Process and surrogate measures of the impact of the TRN program in supporting clinical research by basic scientists at Rockefeller include the number of PhD-led laboratories that have undergone TRN for new protocols, which in 2022 was 25, and the number of human participant protocols that are or have been led by PhD principal investigators, which is 48. Other process metrics primarily chart compliance with regulatory requirements. We are planning to track study completion more systematically.

Discussion

Ensuring that clinical studies are informative is a complex process and each institution needs to customize its approach to align with its structure, culture, and goals. We recognize that Rockefeller University is unlike academic medical centers in many ways, especially with regard to size, the number of protocols, and the near-exclusive focus on early-stage studies, and so the infrastructure we have developed is unlikely to be scalable without considerable modification. At the same time, some of the fundamental principles of the approaches we have taken are likely generalizable, and these are reflected in the elements of our program designed to address the core requirements for informative trials listed in Table 1. Most important is institutional and granting agency recognition that a robust multidisciplinary infrastructure is crucial, especially in providing support for studies led by trainees and less experienced investigators. The cost of building, maintaining, and when appropriate to meet new needs, expanding the infrastructure to support informative trials is considerable and has increased dramatically during the past two decades in response to expectations about training, regulatory compliance, and informativeness. Each institution requires a plan for stable funding that meets its particular needs, recognizing that relying primarily or exclusively on chargebacks to investigators will likely discourage clinical investigation, especially for basic scientists and junior investigators with limited budgets. CTSA funding was extremely important in expanding our infrastructure when the program began in 2006, but now represents a much smaller fraction of the costs.

The most important theme that unites our approach is the application of the scientific method to each element in the process. The clinical science enterprise itself has not been the subject of rigorous research, in part because grant funds from the categorical NIH Institutes have in the past focused on disease-specific research rather than the process of performing that research, now identified as a component of translational science [44]. This began to change with the realization by the NIH Institutes that many of the clinical studies they funded were not yielding informative information in an acceptable timeframe or in a form that leads to effective dissemination and implementation. The creation of the National Center for Advancing Translational Science (NCATS) provided an opportunity to address this more directly. NCATS recently began to focus its funding on translational science rather than disease-specific translational research, but without robust funding for follow-on R-type grants, there still is not a straight path for investigators to build academic careers in studies of translational science beyond what individual CTSA hubs can fund with extremely limited uncommitted resources. Partnerships with other funding agencies, including the Centers for Disease Control and Prevention (CDC), the Agency for Healthcare Research and Quality, and the Patient-Centered Outcomes Research Institute, may provide support to examine the structure, process, and outcome of research conducted with patients, as well as provide an opportunity to broaden the research workforce by engaging individuals involved in direct care delivery settings as part of translational research teams. This will enhance workforce diversity and inject real-world lived experience perspectives and insights into the design and conduct of research. In turn, this will increase the likelihood that studies are patient-centered and designed for dissemination and that the results can be applied across diverse clinical settings [45,46].

The COVID-19 pandemic demonstrated the weaknesses in the clinical trial enterprise [8] and has stimulated both soul-searching and requests for ideas about reorganization, most recently from the White House Office of Science and Technology Policy [47,48]. The CTSA program and NCATS are ideally positioned to lead this effort, bringing rigorous scientific analysis to the clinical enterprise itself. Identifying policies and methods to ensure that clinical studies are informative is a major component of that effort and an important investment in enhancing the future of translational research.

Disclosures

The authors have no disclosures to declare.