Assessing the Impact of the NIH CTSA Program on Institutionally Sponsored Clinical Trials

Abstract

Objective

To assess the impact of the NIH CTSA program on patient enrollment in clinical trials sponsored/collaborated by CTSA consortium institutions.

Material and Methods

Using publicly available clinical trial data at ClinicalTrials.gov, we identify positive trend changes in the number of patients enrolled in clinical trials performed at CTSA consortium institutions over the years before and after their respective CTSA award dates. CTSA consortium institutions were matched with similar non‐CTSA institutions.

Results

As compared to matched non‐CTSA institutions CTSA consortium sites noted an increase in patient enrollment after the CTSA awards. In particular, we detected a change‐point, where a new enrollment trend emerged, 338 days after the CTSA award. No such trend was noted over the same period in the non‐CTSA institutions.

Conclusion

Our analysis provides evidence that the NIH CTSA funding program made a positive impact on patient enrollment.

Introduction

In an effort to speed the transfer from bench to bedside, the National Institutes of Health (NIH) created the institutional Clinical and Translational Science Award (CTSA) in 2006 and has funded 60 institutions thus far. The goal of these institutions was to provide research infrastructure support so that funded institutions could facilitate multidisciplinary team research,1 train clinical and translational investigatgors,2 and encourage research partnerships and collaborations. It was suggested that this approach would reduce the time necessary for a new idea or approach to alter patient care in the United States.3, 4, 5, 6, 7, 8

Since 2012, the CTSA program has moved into the newly created National Center for Advancing Translational Sciences (NCATS). Of the $30.7 billion total NIH budget for FY2012, NCATS's budget was $575.4 million. About 80% of it or $461.4 million was allocated to the CTSA program. While NCATS has put strong efforts into streamlining drug discovery and bringing new therapeutics, diagnostics and devices to the clinic, it maintains that a key goal of the CTSA program is to improve clinical and translational research, which we postulate would ultimately be positively reflected in the clinical trial enrollment data.9

Clinical trials are considered a cornerstone in clinical and translational research, serving as a pivotal point along the translational pipeline.7, 10, 11 Institutional efforts to address inefficiencies will positively impact clinical trial activities, such as accelerated increases in the numbers of clinical trials as well as the clinical trial patient enrollments.12 We conducted analyses over patient enrollment in the clinical trials sponsored/collaborated by the CTSA consortium institutions (available at ClinicalTrials.gov) to identify any trend pattern changes years before and after the CTSA awards in an effort to assess the CTSA impact on clinical trial activities.

Method

We investigated the trend changes in patient enrollment during the years before and after the CTSA award dates in the clinical trials sponsored/collaborated by the CTSA consortium institutions. Patient enrollment in clinical trials was used as a surrogate variable to indirectly assess the potential impact of CTSA over clinical activities and clinical and translational research in general.

ClinicalTrials.gov is the largest database and the registry of clinical trials. Run by the National Library of Medicine (NLM) at NIH, it maintains records of federally and privately supported clinical trials conducted in all 50 States in the United States and in 178 countries.

To download clinical trial records: upon connecting to ClinicalTrials.gov, choose Advanced Search for querying clinical trial records. At the field of Sponsor/Collaborators, enter the name of the institution (e.g., Harvard University) to be queried while leaving the rest fields to their default. Finally, download all the returned trial records and save in comma separated values (.csv) format. We retrieved the trial enrollments and start dates from the .csv files to conduct analyses.

In order to have sufficient comparative data past the CTSA award dates, we chose to include the 46 institutions who received their CTSA awards on or before July 14, 2009. In particular, dates in the field of Start Date in the downloaded datasets were used as the start dates of the trials to determine if a trial were to be included in the study.

We noticed that during the covered period, there were a few clinical trials that had extremely large patient enrollments (≥150,000 enrolled patients). For example, while of the 12,392 clinical trials during the period from January 1, 2003 to June 30, 2011, there were 17 (0.1% of total trials) trials with enrollments ≥150,000; enrollments for these trials were 5,270,999, which accounted for 49.1% of the total trial enrollments of 10,725,922. We removed these trials as outliers from our analyses.

Centered versus uncentered

In order to detect changes that were due solely to CTSA funding, we centered the clinical trial enrollment data by their CTSA award dates and conducted analyses over the centered enrollment data. Centering was done for each CTSA consortium institution by setting up its award date to be zero and counting the dates of its clinical trials as days relative to the award date, with negative days indicating the trial started before the award date and positive days as trials started after CTSA funding.

Matching

We collected patient enrollment data in the clinical trials sponsored/collaborated by 23 nationally recognized institutions that have not been members of the CTSA consortium. We then matched these non‐CTSA institutions with 23 out of the 46 analyzed CTSA institutions, proportionally covering the institutions in each of the four CTSA cohorts grouped by their years of awards. As Table 1 shows, we matched a pair by their geographical proximity, institution types (public/private), and the numbers of the institutionally sponsored/collaborated clinical trials. Although we made conscious efforts to match between CTSA and non‐CTSA institutions, we were well aware of the fact that all CTSA consortium members were nationally reputable and well‐funded institutions with superior capacities in conducting medical research and clinical trials than non‐CTSA institutions. As a result, there were often larger numbers of clinical trials being conducted in CTSA institutions than the matched non‐CTSA institutions. We conducted analyses collectively over these matched non‐CTSA institutions following the same protocol as we analyzed those CTSA consortium institutions. In particular, for the centered data analysis, we centered the clinical trials of each non‐CTSA institution by its matched CTSA institution award date.

Table 1.

CTSA institutions and their matched non‐CTSA institutions

CTSA institutions	Dates of award	State		Public/Private		Number of CTs		Non‐CTSA institutions
Duke	September 30, 2006	NC	NC	Private	Private	647	307	Wake forest
UC‐Davis	September 30, 2006	CA	AZ	Public	Public	265	144	U of Arizona
Pitt	September 30, 2006	PA	PA	Public	Public	753	82	Temple
U of Rochester	September 30, 2006	NY	VT	Public	Public	298	106	U of Vermont
Texas‐Houston	September 30, 2006	TX	TX	Public	Private	200	429	Baylor
Yale	September 30, 2006	CT	RI	Private	Private	468	105	Brown
Case Western	September 17, 2007	OH	DC	Private	Private	71	45	G. Washington
Johns Hopkins	September 17, 2007	MD	MD	Private	Public	564	264	Maryland
U of Michigan	September 17, 2007	MI	MI	Public	Public	579	63	Michigan State
Vanderbilt	September 17, 2007	TN	TN	Private	Public	502	80	U of Tennessee
Washington U	September 17, 2007	MO	MO	Private	Private	343	32	St. Louis U.
Cornell	September 17, 2007	NY	NJ	Private	Public	371	201	NJ Med & Den
Boston U	May 19, 2008	MA	CT	Private	Public	215	131	U‐Conn
Indiana Med	May 19, 2008	IN	MO	Public	Public	142	118	U of Missouri
Ohio State	May 19, 2008	OH	WV	Public	Public	269	26	West Virginia
UA‐Birmingham	May 19, 2008	AL	MS	Public	Public	374	65	U of Mississippi
U of Colorado	May 19, 2008	CO	NE	Public	Public	294	202	U of Nebraska
UNC‐CH	May 19, 2008	NC	VA	Public	Public	339	418	U of Virginia
Texas‐SA	May 19, 2008	TX	LA	Public	Public	91	47	LSU
Med. U of SC	September 19, 2009	SC	GA	Public	Public	221	41	GA Health Sci.
U of Arkansas	September 19, 2009	AR	OK	Public	Public	189	152	U of Oklahoma
U of Florida	September 19, 2009	FL	FL	Public	Public	442	117	U‐South Florida
Illinois‐Chicago	September 19, 2009	IL	IL	Public	Public	85	29	S. Illinois U.

Each pair of institutions was matched by their geographical proximity, types of institution (public/private) and their numbers of institutionally sponsored/collaborated clinical trials. The column of “date of award” recorded the award date for the CTSA institutions, which was also used to center the matched non‐CTSA institution in the same row. As for the columns “State,” “Public/Private” and “number of CTs,” respectively, the left subcolumns were of the CTSA institutions; while the right column were of the matched non‐CTSA institutions.

Slope comparison pre‐/post‐CTSA awards

Any apparent change in trend before and after CTSA awards provided evidence that CTSA funding influenced enrollment. We applied linear model regression to fit the data and evaluated the significance of difference in the slopes of those linear models.

Change‐point analysis

Change‐point analysis13, 14, 15, 16, 17, 18 identifies statistically significant trend pattern changes. It is an analytical method that attempts to find a point along a distribution of values where the characteristics of the values before and after the point are different.17, 19 A change‐point of an uptrend after the CTSA award would support our hypothesis that CTSA funding had a positive influence on clinical trial enrollment. We used this approach to estimate a point at which statistical properties of a sequence of observations changed.

We identified the existence of change‐point based on a hypothesis test,15 where the null hypothesis, H₀, was no change‐point. A likelihood‐ratio based method is then applied for testing the null hypothesis. The null hypothesis would be rejected if none of the likelihood ratios was significant. A change‐point was then estimated as the location where the maximum likelihood was achieved. The significance of a change‐point was measured by means of a permutation test of 5,000 permutations.

Binning

In order to be able to apply the change‐point analysis algorithm to get reliable results without many false positives, we adopted a strategy of binning. Particularly, we assigned a clinical trial to one of the equal‐sized bins according to the trial's start date. The size of bins can be chosen flexibly. In the present study, the bin size was set to be 25 days. After all the trials were assigned to a bin, for each bin, we added up the numbers of patient enrollment for all the trials in that bin. We then conducted change‐point analysis over the binned enrollment data.

Results

We visually compared the institutionally sponsored/collaborated clinical trial patient enrollment at CTSA and non‐CTSA institutions in order to identify any trend changes before and after the CTSA award dates. We considered the trials from 1,750 days (≈5 years) before and 750 days (≈2 years) after the CTSA award. We plotted, as scattered points in Figure 1, the monthly binned of centered patient enrollment at CTSA and non‐CTSA institutions, respectively. The enrollment data were normalized by their preaward average bin patient enrollment. These normalized enrollments before and after the CTSA awards were then separately fitted with linear model regressions, that is, straight lines, also drawn in Figure 1. As we can see in Figure 1, there was an apparent upward change in the slope after the award in enrollments from CTSA institutions; we however did not observe a similar change over the non‐CTSA cohort. Statistical comparison of the linear fits between pre‐ and postaward enrollments reported the p‐values of 0.0788 and 0.891 for the enrollments at CTSA and non‐CTSA institutions, respectively.

Figure 1.

Normalized patient enrollment in the clinical trials sponsored/collaborated by the CTSA institutions (red) and their matched non‐CTSA institutions (blue), centered to the CTSA award dates of the matched CTSA institutions, indicated by a black vertical line. The plotted enrollments of both CTSA and non‐CTSA institutions were binned with 30‐day bins and were normalized by their average bin enrollment prior to the awarding dates, respectively. The pre‐ and post‐award enrollments were fitted with linear models for CTSA and non‐CTSA institutions, respectively.

We then conducted change‐point analyses over the CTSA and the matched non‐CTSA institutions, respectively, using the centered patient enrollment data. As Figure 2 shows, the cumulative sum curve of the patient trial enrollment of CTSA institutions visually had a change of slope right after the CTSA award. By change‐point analysis, we identified a statistically significant change‐point 338 days after the CTSA awards with p‐value = 0.0028 over 5,000 permutations.

Figure 2.

Patient enrollment in the clinical trials sponsored/collaborated by the CTSA institutions, centered to the CTSA award date, indicated by a blue dotted vertical line. The red‐color curve was the cumulative curve during that period. The black dots were the monthly enrollment with the two red‐color line segments indicated the mean monthly enrollments during the periods before and after the change‐point. The p‐values on the top‐left indicated the significance of this change‐point, over 5,000 permutations.

In contrast, as Figure 3 shows in the case of patient clinical trial enrollment in the matched non‐CTSA institutions, our analysis identified a nonsignificant change‐point 113 days after the CTSA award, it was not considered as statistically significant with p‐value = 0.069 over 5,000 permutations. These results support the hypothesis that the positive trend changes in patient enrollment in CTSA institution‐sponsored/collaborated clinical trials were unique to the CTSA institutions and were more likely due to the impact from CTSA funding.

Figure 3.

Patient enrollment in the clinical trials sponsored/collaborated by the matched non‐CTSA institutions, centered to the CTSA award date of the matched CTSA institution, indicated by a blue dotted vertical line. The red‐color curve was the cumulative curve during that period. The black dots were the monthly enrollment with the two red‐color line segments indicated the mean monthly enrollments during the periods before and after the change‐point. The p‐values on the top left indicated the significance of this change‐point, over 5,000 permutations.

We also analyzed the uncentered patient enrollment. This was done to determine if chronological events played a role in causing the trend changes in clinical trial enrollment. We collected data with the trial starting date between January 1, 2003 and June 30, 2011. As Figure 4 shows, it appears less obvious that there are any visual changes of pattern in the trend along the cumulative sum curve of the patient trial enrollment. While the subsequent change‐point analysis detected a change‐point on May 2nd, 2009, its p‐value over 5,000 permutation tests was only 0.046, which was only borderline significant and certainly less conclusive than the results from the centered enrollment data of the 46 CTSA institutions. In fact, we anticipated a certain degree of correlation between the centered and uncentered enrollment data of the CTSA institutions due to the fact that there were only four cohorts with award dates from years of 2006 to 2009. Thus it is conceivable that the borderline significance from the un‐centered data was due to the CTSA impact as well.

Figure 4.

Patient enrollment in the clinical trials sponsored/collaborated by the CTSA institutions, during the period from January 1, 2003 to June 30, 2011. The red‐color curve was the cumulative curve during that period. The black dots were the monthly enrollment with the two red‐color line segments indicated the mean monthly enrollments during the periods before and after the change‐point. The p‐values on the top left indicated the significance of this change‐point, over 5,000 permutations.

Conclusions

These analyses confirm our hypothesis that the NIH CTSA funding program has had a positive impact on patient enrollment in the clinical trials sponsored/collaborated by the CTSA consortium institutions. Since clinical trial enrollment served as a marker for clinical research activities in general, we conclude that we identified a statistically significant correlation between the positive trend change in the institutionally sponsored/collaborated clinical trial enrollments and the time of the CTSA awards to the CTSA consortium institutions. Through matching between CTSA and non‐CTSA institutions, as well as centered/uncentered patient enrollment among CTSA consortium institutions, we removed potential confounders that may have also influenced patient enrollment and provided further evidence that such significant trend change was due to the impact of the NIH CTSA program.