Abstract
Objective
To assess the impact of the NIH CTSA program on patient enrollment in clinical trials sponsored/collaborated by CTSA consortium institutions.
Material and Methods
Using publicly available clinical trial data at ClinicalTrials.gov, we identify positive trend changes in the number of patients enrolled in clinical trials performed at CTSA consortium institutions over the years before and after their respective CTSA award dates. CTSA consortium institutions were matched with similar non‐CTSA institutions.
Results
As compared to matched non‐CTSA institutions CTSA consortium sites noted an increase in patient enrollment after the CTSA awards. In particular, we detected a change‐point, where a new enrollment trend emerged, 338 days after the CTSA award. No such trend was noted over the same period in the non‐CTSA institutions.
Conclusion
Our analysis provides evidence that the NIH CTSA funding program made a positive impact on patient enrollment.
Keywords: CTSA, clinical and translational science, clinical trials
Introduction
In an effort to speed the transfer from bench to bedside, the National Institutes of Health (NIH) created the institutional Clinical and Translational Science Award (CTSA) in 2006 and has funded 60 institutions thus far. The goal of these institutions was to provide research infrastructure support so that funded institutions could facilitate multidisciplinary team research,1 train clinical and translational investigatgors,2 and encourage research partnerships and collaborations. It was suggested that this approach would reduce the time necessary for a new idea or approach to alter patient care in the United States.3, 4, 5, 6, 7, 8
Since 2012, the CTSA program has moved into the newly created National Center for Advancing Translational Sciences (NCATS). Of the $30.7 billion total NIH budget for FY2012, NCATS's budget was $575.4 million. About 80% of it or $461.4 million was allocated to the CTSA program. While NCATS has put strong efforts into streamlining drug discovery and bringing new therapeutics, diagnostics and devices to the clinic, it maintains that a key goal of the CTSA program is to improve clinical and translational research, which we postulate would ultimately be positively reflected in the clinical trial enrollment data.9
Clinical trials are considered a cornerstone in clinical and translational research, serving as a pivotal point along the translational pipeline.7, 10, 11 Institutional efforts to address inefficiencies will positively impact clinical trial activities, such as accelerated increases in the numbers of clinical trials as well as the clinical trial patient enrollments.12 We conducted analyses over patient enrollment in the clinical trials sponsored/collaborated by the CTSA consortium institutions (available at ClinicalTrials.gov) to identify any trend pattern changes years before and after the CTSA awards in an effort to assess the CTSA impact on clinical trial activities.
Method
We investigated the trend changes in patient enrollment during the years before and after the CTSA award dates in the clinical trials sponsored/collaborated by the CTSA consortium institutions. Patient enrollment in clinical trials was used as a surrogate variable to indirectly assess the potential impact of CTSA over clinical activities and clinical and translational research in general.
ClinicalTrials.gov is the largest database and the registry of clinical trials. Run by the National Library of Medicine (NLM) at NIH, it maintains records of federally and privately supported clinical trials conducted in all 50 States in the United States and in 178 countries.
To download clinical trial records: upon connecting to ClinicalTrials.gov, choose Advanced Search for querying clinical trial records. At the field of Sponsor/Collaborators, enter the name of the institution (e.g., Harvard University) to be queried while leaving the rest fields to their default. Finally, download all the returned trial records and save in comma separated values (.csv) format. We retrieved the trial enrollments and start dates from the .csv files to conduct analyses.
In order to have sufficient comparative data past the CTSA award dates, we chose to include the 46 institutions who received their CTSA awards on or before July 14, 2009. In particular, dates in the field of Start Date in the downloaded datasets were used as the start dates of the trials to determine if a trial were to be included in the study.
We noticed that during the covered period, there were a few clinical trials that had extremely large patient enrollments (≥150,000 enrolled patients). For example, while of the 12,392 clinical trials during the period from January 1, 2003 to June 30, 2011, there were 17 (0.1% of total trials) trials with enrollments ≥150,000; enrollments for these trials were 5,270,999, which accounted for 49.1% of the total trial enrollments of 10,725,922. We removed these trials as outliers from our analyses.
Centered versus uncentered
In order to detect changes that were due solely to CTSA funding, we centered the clinical trial enrollment data by their CTSA award dates and conducted analyses over the centered enrollment data. Centering was done for each CTSA consortium institution by setting up its award date to be zero and counting the dates of its clinical trials as days relative to the award date, with negative days indicating the trial started before the award date and positive days as trials started after CTSA funding.
Matching
We collected patient enrollment data in the clinical trials sponsored/collaborated by 23 nationally recognized institutions that have not been members of the CTSA consortium. We then matched these non‐CTSA institutions with 23 out of the 46 analyzed CTSA institutions, proportionally covering the institutions in each of the four CTSA cohorts grouped by their years of awards. As Table 1 shows, we matched a pair by their geographical proximity, institution types (public/private), and the numbers of the institutionally sponsored/collaborated clinical trials. Although we made conscious efforts to match between CTSA and non‐CTSA institutions, we were well aware of the fact that all CTSA consortium members were nationally reputable and well‐funded institutions with superior capacities in conducting medical research and clinical trials than non‐CTSA institutions. As a result, there were often larger numbers of clinical trials being conducted in CTSA institutions than the matched non‐CTSA institutions. We conducted analyses collectively over these matched non‐CTSA institutions following the same protocol as we analyzed those CTSA consortium institutions. In particular, for the centered data analysis, we centered the clinical trials of each non‐CTSA institution by its matched CTSA institution award date.
Table 1.
CTSA institutions and their matched non‐CTSA institutions
CTSA institutions | Dates of award | State | Public/Private | Number of CTs | Non‐CTSA institutions | |||
---|---|---|---|---|---|---|---|---|
Duke | September 30, 2006 | NC | NC | Private | Private | 647 | 307 | Wake forest |
UC‐Davis | September 30, 2006 | CA | AZ | Public | Public | 265 | 144 | U of Arizona |
Pitt | September 30, 2006 | PA | PA | Public | Public | 753 | 82 | Temple |
U of Rochester | September 30, 2006 | NY | VT | Public | Public | 298 | 106 | U of Vermont |
Texas‐Houston | September 30, 2006 | TX | TX | Public | Private | 200 | 429 | Baylor |
Yale | September 30, 2006 | CT | RI | Private | Private | 468 | 105 | Brown |
Case Western | September 17, 2007 | OH | DC | Private | Private | 71 | 45 | G. Washington |
Johns Hopkins | September 17, 2007 | MD | MD | Private | Public | 564 | 264 | Maryland |
U of Michigan | September 17, 2007 | MI | MI | Public | Public | 579 | 63 | Michigan State |
Vanderbilt | September 17, 2007 | TN | TN | Private | Public | 502 | 80 | U of Tennessee |
Washington U | September 17, 2007 | MO | MO | Private | Private | 343 | 32 | St. Louis U. |
Cornell | September 17, 2007 | NY | NJ | Private | Public | 371 | 201 | NJ Med & Den |
Boston U | May 19, 2008 | MA | CT | Private | Public | 215 | 131 | U‐Conn |
Indiana Med | May 19, 2008 | IN | MO | Public | Public | 142 | 118 | U of Missouri |
Ohio State | May 19, 2008 | OH | WV | Public | Public | 269 | 26 | West Virginia |
UA‐Birmingham | May 19, 2008 | AL | MS | Public | Public | 374 | 65 | U of Mississippi |
U of Colorado | May 19, 2008 | CO | NE | Public | Public | 294 | 202 | U of Nebraska |
UNC‐CH | May 19, 2008 | NC | VA | Public | Public | 339 | 418 | U of Virginia |
Texas‐SA | May 19, 2008 | TX | LA | Public | Public | 91 | 47 | LSU |
Med. U of SC | September 19, 2009 | SC | GA | Public | Public | 221 | 41 | GA Health Sci. |
U of Arkansas | September 19, 2009 | AR | OK | Public | Public | 189 | 152 | U of Oklahoma |
U of Florida | September 19, 2009 | FL | FL | Public | Public | 442 | 117 | U‐South Florida |
Illinois‐Chicago | September 19, 2009 | IL | IL | Public | Public | 85 | 29 | S. Illinois U. |
Each pair of institutions was matched by their geographical proximity, types of institution (public/private) and their numbers of institutionally sponsored/collaborated clinical trials. The column of “date of award” recorded the award date for the CTSA institutions, which was also used to center the matched non‐CTSA institution in the same row. As for the columns “State,” “Public/Private” and “number of CTs,” respectively, the left subcolumns were of the CTSA institutions; while the right column were of the matched non‐CTSA institutions.
Slope comparison pre‐/post‐CTSA awards
Any apparent change in trend before and after CTSA awards provided evidence that CTSA funding influenced enrollment. We applied linear model regression to fit the data and evaluated the significance of difference in the slopes of those linear models.
Change‐point analysis
Change‐point analysis13, 14, 15, 16, 17, 18 identifies statistically significant trend pattern changes. It is an analytical method that attempts to find a point along a distribution of values where the characteristics of the values before and after the point are different.17, 19 A change‐point of an uptrend after the CTSA award would support our hypothesis that CTSA funding had a positive influence on clinical trial enrollment. We used this approach to estimate a point at which statistical properties of a sequence of observations changed.
We identified the existence of change‐point based on a hypothesis test,15 where the null hypothesis, H0, was no change‐point. A likelihood‐ratio based method is then applied for testing the null hypothesis. The null hypothesis would be rejected if none of the likelihood ratios was significant. A change‐point was then estimated as the location where the maximum likelihood was achieved. The significance of a change‐point was measured by means of a permutation test of 5,000 permutations.
Binning
In order to be able to apply the change‐point analysis algorithm to get reliable results without many false positives, we adopted a strategy of binning. Particularly, we assigned a clinical trial to one of the equal‐sized bins according to the trial's start date. The size of bins can be chosen flexibly. In the present study, the bin size was set to be 25 days. After all the trials were assigned to a bin, for each bin, we added up the numbers of patient enrollment for all the trials in that bin. We then conducted change‐point analysis over the binned enrollment data.
Results
We visually compared the institutionally sponsored/collaborated clinical trial patient enrollment at CTSA and non‐CTSA institutions in order to identify any trend changes before and after the CTSA award dates. We considered the trials from 1,750 days (≈5 years) before and 750 days (≈2 years) after the CTSA award. We plotted, as scattered points in Figure 1, the monthly binned of centered patient enrollment at CTSA and non‐CTSA institutions, respectively. The enrollment data were normalized by their preaward average bin patient enrollment. These normalized enrollments before and after the CTSA awards were then separately fitted with linear model regressions, that is, straight lines, also drawn in Figure 1. As we can see in Figure 1, there was an apparent upward change in the slope after the award in enrollments from CTSA institutions; we however did not observe a similar change over the non‐CTSA cohort. Statistical comparison of the linear fits between pre‐ and postaward enrollments reported the p‐values of 0.0788 and 0.891 for the enrollments at CTSA and non‐CTSA institutions, respectively.
Figure 1.
Normalized patient enrollment in the clinical trials sponsored/collaborated by the CTSA institutions (red) and their matched non‐CTSA institutions (blue), centered to the CTSA award dates of the matched CTSA institutions, indicated by a black vertical line. The plotted enrollments of both CTSA and non‐CTSA institutions were binned with 30‐day bins and were normalized by their average bin enrollment prior to the awarding dates, respectively. The pre‐ and post‐award enrollments were fitted with linear models for CTSA and non‐CTSA institutions, respectively.
We then conducted change‐point analyses over the CTSA and the matched non‐CTSA institutions, respectively, using the centered patient enrollment data. As Figure 2 shows, the cumulative sum curve of the patient trial enrollment of CTSA institutions visually had a change of slope right after the CTSA award. By change‐point analysis, we identified a statistically significant change‐point 338 days after the CTSA awards with p‐value = 0.0028 over 5,000 permutations.
Figure 2.
Patient enrollment in the clinical trials sponsored/collaborated by the CTSA institutions, centered to the CTSA award date, indicated by a blue dotted vertical line. The red‐color curve was the cumulative curve during that period. The black dots were the monthly enrollment with the two red‐color line segments indicated the mean monthly enrollments during the periods before and after the change‐point. The p‐values on the top‐left indicated the significance of this change‐point, over 5,000 permutations.
In contrast, as Figure 3 shows in the case of patient clinical trial enrollment in the matched non‐CTSA institutions, our analysis identified a nonsignificant change‐point 113 days after the CTSA award, it was not considered as statistically significant with p‐value = 0.069 over 5,000 permutations. These results support the hypothesis that the positive trend changes in patient enrollment in CTSA institution‐sponsored/collaborated clinical trials were unique to the CTSA institutions and were more likely due to the impact from CTSA funding.
Figure 3.
Patient enrollment in the clinical trials sponsored/collaborated by the matched non‐CTSA institutions, centered to the CTSA award date of the matched CTSA institution, indicated by a blue dotted vertical line. The red‐color curve was the cumulative curve during that period. The black dots were the monthly enrollment with the two red‐color line segments indicated the mean monthly enrollments during the periods before and after the change‐point. The p‐values on the top left indicated the significance of this change‐point, over 5,000 permutations.
We also analyzed the uncentered patient enrollment. This was done to determine if chronological events played a role in causing the trend changes in clinical trial enrollment. We collected data with the trial starting date between January 1, 2003 and June 30, 2011. As Figure 4 shows, it appears less obvious that there are any visual changes of pattern in the trend along the cumulative sum curve of the patient trial enrollment. While the subsequent change‐point analysis detected a change‐point on May 2nd, 2009, its p‐value over 5,000 permutation tests was only 0.046, which was only borderline significant and certainly less conclusive than the results from the centered enrollment data of the 46 CTSA institutions. In fact, we anticipated a certain degree of correlation between the centered and uncentered enrollment data of the CTSA institutions due to the fact that there were only four cohorts with award dates from years of 2006 to 2009. Thus it is conceivable that the borderline significance from the un‐centered data was due to the CTSA impact as well.
Figure 4.
Patient enrollment in the clinical trials sponsored/collaborated by the CTSA institutions, during the period from January 1, 2003 to June 30, 2011. The red‐color curve was the cumulative curve during that period. The black dots were the monthly enrollment with the two red‐color line segments indicated the mean monthly enrollments during the periods before and after the change‐point. The p‐values on the top left indicated the significance of this change‐point, over 5,000 permutations.
Conclusions
These analyses confirm our hypothesis that the NIH CTSA funding program has had a positive impact on patient enrollment in the clinical trials sponsored/collaborated by the CTSA consortium institutions. Since clinical trial enrollment served as a marker for clinical research activities in general, we conclude that we identified a statistically significant correlation between the positive trend change in the institutionally sponsored/collaborated clinical trial enrollments and the time of the CTSA awards to the CTSA consortium institutions. Through matching between CTSA and non‐CTSA institutions, as well as centered/uncentered patient enrollment among CTSA consortium institutions, we removed potential confounders that may have also influenced patient enrollment and provided further evidence that such significant trend change was due to the impact of the NIH CTSA program.
Acknowledgments
This publication was supported by Grant Number UL1 TR000127 from the National Center for Advancing Translational Sciences (NCATS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. The authors would like to thank Allshine Chen for his editorial contributions.
References
- 1. Selker HP, Strom BL, Ford DE, Meltzer DO, Pauker SG, Pincus HA, Rich EC, Tompkins C, Whitlock EP. White Paper on CTSA Consortium Role in Facilitating Comparative Effectiveness Research. Clin Trans Sci. 2010; 3: 29–37. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Silet KA, Asquith P, Fleming MF. A National Survey of mentoring programs for KL2 scholars. Clin Transl Sci. 2010; 3(6): 299–304. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Alving B. The Role of CTSAs in Transforming the Clinical Trials Process in the United States. Presentation at the Institute of Medicine. October 7, 2009.
- 4. Clinical and Translational Science Awards . NCRR fact sheet. Summer 2011. Bethesda, MD: National Center for Research Resources. [Google Scholar]
- 5. CTSA Progress Report 2008. Bethesda, MD: National Center for Research Resources. [Google Scholar]
- 6. CTSA Progress Report 2009. –2011. Bethesda, MD: National Center for Research Resources. [Google Scholar]
- 7. Feldman A. Does academic culture support translational research? Clin Trans Sci. 2008; 1(2): 87–88. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Hayward A. The CTSA: what they are and what they will accomplish for whom. Special Report: “The Crisis in Clinical Research.” December 2010. American Federation for Medical Research.
- 9. Wadman M. National prescription for drug development. Nat Biotechnol. 2012; 30(4): 309 –312. [DOI] [PubMed] [Google Scholar]
- 10. Gad SC. Clinical Trials Handbook. Hoboken, NJ: John Wiley & Sons; p. 118. [Google Scholar]
- 11. Goldblatt EM, Lee WH. From bench to bedside: the growing use of translational research in cancer medicine. Am J Transl Res. 2010; 2(1): 1–18. [PMC free article] [PubMed] [Google Scholar]
- 12. Heller C, Melo‐Martin I. Clinical and translational science awards: can they increase the efficiency and speed of clinical and translational research. Acad Med. 2009; 84(4): 424–432. [DOI] [PubMed] [Google Scholar]
- 13. Chen J, Gupta AK. Testing and locating variance changepoints with application to stock prices. JASA. 1997; 92: 739–747. [Google Scholar]
- 14. Chen J, Gupta AK. Parametric statistical change point analysis. Boston: Birkhouser, 2000. [Google Scholar]
- 15. Hinkley DV. Inference about the change‐point in a sequence of random variables. Biometrika. 1970; 57: 1–17. [Google Scholar]
- 16. Hsu DA. Detecting shifts of parameter in gamma sequences with applications to stock rice and air trafic flow analysis. JASA. 1979; 74: 31–40. [Google Scholar]
- 17. Killick R, Eckley I. Changepoint: an R package for changepoint analysis. R package version 0.6. http://CRAN.R‐project.org/package=changepoint, 2012.
- 18. Page ES. Continuous inspection schemes. Biometrika. 1954; 41(1–2): 100–115. [Google Scholar]
- 19. Eckley IA, Fearnhead, P , Killick R. Analysis of Changepoint Models. Bayesian Time Series Models. Cambridge, United Kingdom: Cambridge University Press; 2011. [Google Scholar]