Developing a model to predict accrual to cancer clinical trials: Data from an NCI designated cancer center

Abstract

Introduction

As cancer center funds are allocated toward several resources, clinical trial offices and the clinical trial infrastructure is constantly scrutinized. It has been shown that 20% of clinical trials fail to achieve their accrual goal and in an institutional level several trials are open with poor accrual. We sought to identify factors that are associated with clinical trial accrual and develop a model to predict clinical trial accrual

Methods and material

We identified all clinical trials from 1999 to 2015 at UT Health Cancer Center San Antonio. We included observational as well as interventional clinical trials. We collected several variables such as type of study, type of malignancy, trial phase, PI of study.

Results

In total we included 297 clinical trials. We identified several factors to be associated with clinical trial accrual (Sponsor type, trial phase, disease category, type of trial, disease state and whether the trial involved a new investigational agent). We developed a predictive model with an AUC of 0.65 that showed that observational, interventional, industry-sponsored trials and trials authored by the local PI were more likely to achieve their accrual goal.

Conclusion

We were able to identify several factors that were significantly associated with clinical trial accrual. Based on these factors we developed a prediction model for clinical trial accrual. We believe that use of this model can help improve our cancer centers clinical trial portfolio and help in fund allocation.

1. Introduction

Clinical trial accrual is a key factor in moving towards progress in the field of Medicine. However, it is estimated that only 3% of oncology patients in the United States participate in clinical trials [1,2]. Furthermore, a large number of clinical trials in oncology close early due to poor accrual. In an analysis of clinical trial for adults registered on Clinicaltrials.gov, researchers found that about 20% of the trials fail to accrue for several reasons [3]. In a study by Lara et al. [4], the most common reasons for poor clinical trial accrual were a desire for other treatment (34%), patient refusal to disclose reason (11%) and insurance denial (8%). Community based cancer center have also reported low participation in clinical trials, mostly secondary to the unavailability of appropriate clinical trials [5].

NCI- designated cancer centers spend significant resources in supporting clinical trial offices and the clinical trial operation in general. Strategic opening of clinical trials is imperative. However, in the era of molecular medicine, narrower inclusion criteria requiring tissue diagnostic tests make accruing to clinical trials more challenging. As resources in cancer centers and clinical trial offices are limited there needs to be a well-defined strategy in efficiently recruiting to clinical trials.

Several studies have attempted to develop models for predicting clinical trial accrual. In a study from a single institution Drs Tate and Cranmer built and further validated a model for predicting clinical trial accrual [6]. Based on their data, investigational drug application, disease team, number of national sites, local Institutional Review Board use, total national accrual time, accrual completed, and national enrollment goal were independently and significantly associated with accrual. When validated their model was able to predict accrual of at least 4 subjects 75% of the time. Others have evaluated data from cooperative group clinical trials in order to develop a model for clinical trial accrual [7]. However, each institution may have unique characteristics that may not allow models developed elsewhere to be generalizable to their unique infrastructure.

In this paper we seek to find the major factors affecting clinical trial accrual as well as to develop a model to predict clinical trial accrual. We performed a retrospective study involving all clinical trials opened in our institution from September 1999 to December 2015.

2. Materials and methods

This retrospective review was performed at UT Health Cancer Center San Antonio, an NCI designated Cancer Center with a catchment area of South Texas.

2.1. Data collection

Eligible studies included observational, treatment and supportive care studies between 09/1999 and 12/2015 at the Cancer Treatment Research Center now UT Cancer Center of the University of Texas Health Science Center San Antonio. We collected data on trials including sponsor type (whether this was an industry sponsored trial, a cooperative group trial etc), trial phase, disease category (what type of cancer the trial was for), author (whether the author of the trial was the institutional PI or not), clinical research category of the trial (whether the trial was interventional, observational or ancillary), multi-center study, interventional modality (whether the intervention was systemic therapy, radiotherapy or both), targeted therapy (defined as therapy targeting a cancer related pathway that does not include cytotoxic chemotherapy), metastatic disease, clinical category of the malignancy (hematologic malignancy or solid tumor), randomization, presence of placebo, number of interventions, rare cancer category (cancer that occurs in fewer than 15 out of 100,000 people each year), multiple cancers treated and new investigational agents (agents not approved by the FDA at trial initiation) which were among the major factors identified to be influencing clinical trial accrual on our internal review as well as, from literature search on factors influencing clinical trial accrual [[3], [4], [5], [6], [7], [8], [9]]. The definitions of types of clinical trials (interventional, observational etc) were taken from the NCI guide for P30 Cancer Center Support Grant. We excluded long-term studies involving ongoing biobanking specimens of the tumor tissues for future trials because the trials were designed to collect an unspecified number of samples rather than to test a clinical hypothesis. For our primary analyses we classified trials as having low accrual if their accrual was less than 50% of the target in the first year given prior evidence that few trials with less than 50% at one to two years after launch ultimately attain sufficient accrual [3,4].

2.2. Statistical methods

The primary objective was generating a model for predicting low accrual. We defined low accrual as failure to enroll 50% of the annual target for the first year. We assessed the univariate associations of each variable with low accrual using the Chi-squared or Fisher's exact test if expected cell counts fell below 5. All potential predictors of low accrual were entered into a stepwise logistic regression model selection search that used Akaike's Information Criteria (AIC). This process selects a model based upon goodness-of-fit while penalizing for additional variables. We reported the odds ratios, 95% confidence intervals (CI), and p-values for the predictors within the model with lowest AIC. We used multiple imputation method to handle missing data for some variables, this imputation method assumes that the data are missing at random (MAR), which implies that the missingness patterns are only dependent on the observed data. This method constructs 10 complete data sets by filling in the missing values with simulated predictions based upon a bootstrap expectation maximization (EMB) algorithm. The analytical methods were applied to the 10 complete datasets and results were pooled and reported. In order to evaluate the accuracy of prediction, we plotted the ROC curve and computed the area under the receiver operating curve (ROC/AUC). For the AUC, 1.0 is perfect prediction, AUC = 0.7 is good prediction, and AUC = 0.5 prediction equal to random chance. In order to avoid bias in estimation of the AUC, we split the data into training and test sets respectively with 75% and 25% of the observations (trials). Using the training set, we used multivariable logistic regression to build predictor for low accrual. In order to reduce bias due to missing data, we generated 10 imputed datasets with multiple imputation (R package Amelia) and averaged results of across imputations. Using the test set, we estimated the AUC, and we fit a calibration curve to further assess prediction accuracy. The calibration curve compares the risk predicted by the training set and the corresponding observed risk of the test set. The predictions were grouped into 5 quintiles of risk from lowest to highest and the observed risk and standard errors were computed for each quintile. Perfect calibration would be points along the diagonal where observed equals predicted risk. Additionally, we estimated the impact of Hispanic/Latino ethnicity on trial accrual. We calculated the percent of Hispanic/Latino participants that ultimately enrolled in a trial. For trials with 0 accrual (no accrued patients) we could not calculated the proportion of Hispanic/Latino and these trials were excluded from the analysis of ethnicity. We tested for the association of Hispanic/Latino enrollment by including this variable in the model selected by the stepwise AIC algorithm. All calculations were performed with R v3.3 (Vienna, Austria).

3. Results

3.1. Variables assessed and clinical trial prediction

Table 1 shows the univariate associations between low accrual and each predictor assessed. Sponsor type (industry, institutional, national and other), trial phase, disease category, type of trial (ancillary/observational, observational, drug/biologic, radiotherapy, multimodality), disease state (metastatic or not) and whether the trial involved a new investigational agent were all significantly associated with accrual. The rate of missing data for each variable is reported in Supplemental Table 1. Notably, industrial sponsored trials had a higher chance of adequate accrual than institutionally sponsored trials. Also, ancillary/correlative trials had lower rates of successful accrual than interventional trials and trials involving new agents were less likely to achieve their accrual goal. Finally, trials conducted in the metastatic setting had a better chance of achieving accrual goals.

Table 1.

Baseline characteristics.

	Level	Trials with low accrual (n = 145)	Trials with successful accrual (n = 152)	P
Sponsor type	Industrial	35 (31.0)	78 (69.0)	<0.001
	Institutional	43 (58.1)	31 (41.9)
	National	58 (65.9)	30 (34.1)
	Other Externally Peer Reviewed	9 (40.9)	13 (59.1)
Trial phase	I	23 (34.3)	44 (65.7)	0.042
	I/II	5 (29.4)	12 (70.6)
	II	31 (52.5)	28 (47.5)
	III	40 (56.3)	31 (43.7)
	Pilot	7 (53.8)	6 (46.2)
Disease category	Breast	11 (42.3)	15 (57.7)	<0.001
	Gastrointestinal	15 (57.7)	11 (42.3)
	Genitourinary	22 (46.8)	25 (53.2)
	Head & Neck	7 (53.8)	6 (46.2)
	Hematopoietic Malignancies	8 (47.1)	9 (52.9)
	Neuro-Oncology	5 (35.7)	9 (64.3)
	Pediatrics	40 (80.0)	10 (20.0)
	Phase I	22 (32.8)	45 (67.2)
	Special Populations	9 (60.0)	6 (40.0)
	Thoracic	6 (40.0)	9 (60.0)
Authored by PI	No	117 (50.2)	116 (49.8)	0.438
	Yes	28 (43.8)	36 (56.2)
Clinical research	Ancillary/Correlative	19 (82.6)	4 (17.4)	0.002
	Observational	20 (52.6)	18 (47.4)
	Intervention	106 (44.9)	130 (55.1)
Multi center trial	No	30 (46.9)	34 (53.1)	0.858
	Yes	114 (49.1)	118 (50.9)
Intervention modality	Drug or biological	80 (42.8)	107 (57.2)	0.044
	Radiotherapy	4 (44.4)	5 (55.6)
	Multimodality	15 (71.4)	6 (28.6)
Metastatic	No	36 (40.9)	52 (59.1)	0.013
	No or Yes	9 (42.9)	12 (57.1)
	Yes	41 (60.3)	27 (39.7)
	No and Yes	2 (16.7)	10 (83.3)
Disease Type	Hematologic cancers	18 (51.4)	17 (48.6)	0.276
	All cancers	11 (45.8)	13 (54.2)
	Prostate, Colon, Lung or Breast cancers	29 (46.8)	33 (53.2)
	All Solid Cancers	14 (33.3)	28 (66.7)
	All Other Solid Cancers	48 (53.9)	41 (46.1)
Randomized	No	56 (52.8)	50 (47.2)	0.110
	Yes	54 (41.5)	76 (58.5)
Placebo	No	15 (45.5)	18 (54.5)	1.000
	Yes	88 (44.9)	108 (55.1)
Number of interventions	One	45 (38.8)	71 (61.2)	0.055
	More than One	62 (52.1)	57 (47.9)
Multiple cancers treated	No	33 (41.2)	47 (58.8)	0.230
	Yes	86 (50.3)	85 (49.7)
Rare cancer	No	23 (57.5)	17 (42.5)	0.205
	Yes	91 (45.0)	111 (55.0)
New investigational agents	No	45 (36.9)	77 (63.1)	0.008
	Yes	56 (55.4)	45 (44.6)
Percent Hispanic	N	80	121	0.044
	Mean ± SD	53.74 ± 25.98	46.85 ± 21.8
	Median [Q1, Q3]	52.62 [36.27, 72.73]	44.44 [33.33, 60]
	Min, Max	0, 100	0, 100

3.2. Development of predictive model for clinical trial accrual

Table 2 shows the variables selected by the stepwise AIC procedure applied to the training data with estimates based on the fit to the full dataset. Trials authored by the local PI were 3 times (p = 0.01) more likely to have adequate accrual than those that are not. Institutional and Nationally sponsored trials had 0.22 and 0.26 times the odds (p < 0.001) for adequate accrual than industry-sponsored trials. Clinical research categories observational and interventional were about 4 times more likely to have adequate accrual compared with ancillary/correlative studies (p = 0.056 and 0.015, respectively). The ROC curve is shown in Fig. 1. The AUC is 0.65, which indicates a moderate level of predictive accuracy. The predicted risk distribution is shown in Fig. 2. The predicted risk for accrual ranged from 10% to 87%, and this prediction was bimodal with industrial trials accounting for most of the higher accrual group and national trials accounting for the group with lower accrual probability. The calibration curve is shown in Fig. 3. The predicted risk for accrual ranged from 25% to 80%, and this prediction was consistent with the observed risk ( ±1 standard err) except for the second highest risk quintile in which risk was slightly under estimated. This indicates a fair, but not excellent, level of calibration. The Homer-Lemeshow test did not identify statistically significant lack of fit (p = 0.17).

Table 2.

Logistic Regression Lowest AIC Sample Size: 297 clinical trials.

	Odds Ratio	95% CI	P-value
(Intercept)	0.51	(0.13, 1.63)	0.287
Clinical Research: Observational (ref = Ancillary/Correlative)	3.54	(1.03, 14.47)	0.056
Clinical Research: Intervention (ref = Ancillary/Correlative)	4.33	(1.46, 16.18)	0.015*
Sponsor Type: Institutional (ref = Industrial)	0.22	(0.09, 0.49)	<0.001**
Sponsor Type: National (ref = Industrial)	0.26	(0.14, 0.48)	<0.001**
Sponsor Type: Other Externally Peer Reviewed (ref = Industrial)	0.36	(0.11, 1.21)	0.098
Authored by PI: Yes (ref = No)	2.94	(1.28, 7.05)	0.013*

*p < 0.05; **p < 0.001.

Fig. 1.

Confidence intervals for specificity and sensitivity. Receiver Operating Characteristics curve. The AUC (95% CI) is 0.65 (0.52–0.77), which indicates a moderate level of predictive accuracy.

Fig. 2.

Histogram of predicted probabilites of successful accrual. The histogram of probabilties of successful accrual is shown in Fig. 1. The predicted risk ranged from 10% to 87%.

Fig. 3.

Calibration curve. The calibration curve. The predicted risk ranged from 25% to 80%, predicted risk most often overlapped ( ± 1 standard err) with the observed risk except for the second highest risk quintile. This indicates a fair, but not excellent, level of calibration.

Since 50% of patients enrolling in our clinical trials are Hispanic we assessed the effect of Hispanic/Latino enrollment on accrual probability adjusting for the variables selected in the prediction model. The Hispanic/Latino enrollment proportion was not associated with low-accrual in a multivariate logistic regression (OR = 1.0 95% CI 0.99 to 1.1, p = 0.75).

4. Discussion

In a retrospective analysis of trials conducted 1999–2015 at UT Health Cancer Center we were able to develop a model for clinical trial accrual with moderately predictive accuracy. Factors that influenced successful clinical trial accrual were interventional nature of the trial, trial authored by a local PI, as well as an industry-sponsored trial. Given the known disparity in clinical trial participation for Hispanics/Latinos, we looked at the aspect of Hispanic population affecting clinical trial accrual, and we have concluded from statistical analysis that being Hispanic/Latino did not influence clinical trial accrual.

We hypothesize that interventional clinical trials are more likely to accrue successfully due to the potential therapeutic benefit to the patient. This may in turn make it more likely for both the patient and the treating physician to consider this type of trial. A trial authored by a local PI may also be more likely to accrue because of vested interest of the PI as well as the relationship and trust that the PI has established with the patient. Finally, industry-sponsored trials may have a higher likelihood of accruing successfully due to the rigorous accrual plan, the constant monitoring and the available resources that accompany them.

Significant cancer center resources are utilized for clinical trial infrastructure. At our NCI-designated cancer center, 35–40% of the clinical trials office (CTO) budget is supplemented by either the P30 Cancer Center Core grant or philanthropic support. Studies have shown that the financial model for CTOs is unsustainable especially given the fact that cooperative group trials require long-term follow-up but do not subsidize CTOs for that activity [10]. As there is a growing number of cancer survivors on cooperative group clinical trials resources from industry-sponsored trials are not adequate to off-set the deficit created by the lack of funding from cooperative groups. Such issues highlight the importance of having a sound strategic plan when opening clinical trials so that trials can accrue quickly and efficiently. Effort tracking systems may be able to help improve the resource allocation but can be cumbersome to implement in practice [11].

In a previous publication, Bennette et al. developed a model that predicted low accrual for cooperative group clinical trials [7]. That model included 12 clinical trial risk factors and was found to have good predictive accuracy. The model by Bennette et al. has several differences and similarities compared with our model. That model was developed only on phase II or III cooperative group interventional trials, whereas our model included observational, industry, investigator-initiated trials as well as trials of all phases. In the model by Bennette et al. factors predicting low accrual were trials facing higher competition, studied diseases with lower annual incidence and trials requiring a larger enrollment fraction. As in our paper, Bennette et al. found that metastatic trials were more likely to meet target accrual, although in their multivariate model this association lost its significance. They also found that trials including non-FDA approved investigational agents were less likely to suffer from poor accrual, whereas we found the opposite in our model. This may be related to the fact that our model included phase I clinical trials whereas the model developed by Bennette et al. did not. Based on the fact that several factors used in the model from Bennette et al. (number of competing trials, tissue sample required to assess eligibility and enrollment as % of eligible population) we were not available in our database we were not able to validate their model in our dataset. However, we feel that our dataset captures single institution clinical trial scenarios more accurately than the model by Bennette et al.

Other studies have evaluated clinical trial accrual. Schroen et al. ³evaluated accrual in phase III cooperative group clinical trials and found that 29% of trials closed because of poor accrual. Korn et al. ⁸also conducted a study on phase III cooperative group clinical trials and projected that 22% of trials would have less than 90% accrual. Further analysis showed that pediatric and breast cancer related trials were more likely to have adequate accrual and all nonrandomized trials had adequate accrual. Including an investigational new drug did not seem to significantly impact accrual, however since all trials included in this study were phase III, this finding may not apply to phase I clinical trials. A study performed at MD Anderson Cancer Center showed that factors associated with low clinical trial accrual included time from trial activation to first enrolment, and national cooperative group trials [9].

In another single institution study, Tate and Cranmer evaluated factors associated with clinical trial accrual and developed a prediction model. Their model was based on accruing at least 4 patients per trial and factors that were found to be significantly associated with accrual included investigational drug application, disease team, number of national sites, local Institutional Review Board use, total national accrual time, accrual completed, and national enrollment goal [6].

Our study has several strengths and limitations. This is a single institution retrospective study. The results of it may not be generalizable to other institutions. We were also not able to capture factors that have been shown by others to significantly impact clinical trial accrual such as number of competing trials and whether tissue testing was required for accrual. However, we were able to include all trials conducted at our NCI designated cancer center, which has a very active clinical trial portfolio. Disease incidence was not included as an independent factor in our model. This is because % target accrual was used as the target variable which takes into account disease incidence. An important factor to successful clinical trial accrual is minimizing the number of competing clinical trials. This can be achieved by careful selection of clinical trials based on their inclusion criteria, making sure that the same patient population is not included in two clinical trials open at the same time in the institution. Furthermore, the time of overlap between competing trials may also influence accrual. This factor is rarely known when a clinical trial is initiated. For these reasons we have elected to not include competing trials as part of our model.

Based on the data presented here we were able to develop a model that can predict clinical trial accrual. The data that was generated from this work can be used in two ways. The first way is to understand the reasons behind factors that negatively predict clinical trial accrual. This may help our cancer center provide the necessary resources to improve on these negative predictors. For example the fact that institutional clinical trials accrue less than industry clinical trials may be due to the lack of resources that help support these clinical trials. The second way is to generate strategies that can help in improving accrual by enriching our clinical trials portfolio with trials that include positive predictors, such as interventional trials. Furthermore, providing incentives for local investigators to author clinical trials and then help support these trials internally may be another mechanism for improving clinical trial accrual, as well as, optimizing resource allocation for our cancer center. We believe that using this model at the initiation of clinical trial activation can lead to the more rapid improvement in cancer research and treatment as well as development of a more successful clinical trial portfolio which will allow for better resource allocation. We plan to prospectively validate our predictive model.

In summary, we developed a model that can help predict clinical trial accrual. Factors that were found to be significantly associated with successful clinical trial accrual were interventional nature of the trial, trial authored by a local PI, as well as an industry-sponsored trial. Even though the above data come from a single institution, they are easy to collect and therefore are easily generalizable to all cancer centers, and a retrospective analysis of these variables collected from local data could be used to tailor predictions to other centers. We believe that using this model will assist in the strategic planning on an institution's clinical trial portfolio.

Conflicts of interest

None

We have developed a model for predicting accrual to clinical trials. This model can help with the design of clinical trials and resource allocation for cancer centers.

Funding

This work is supported by Dolores Knes Fund and NIH NCI P30 CA054174.

Author contributions

Virginia G Kaklamani: conceptualization, data curation, investigation, supervision, writing, funding acquisition

Praveen Iruku: data curation, investigation, writing

Martin Goros: formal analysis, software, writing

Jonathan Gelfond: Formal analysis, software, writing

Jenny Chang: conceptualization, supervision, writing

Susan Padalecki: conceptualization, supervision, writing, funding acquisition

Ruben Mesa: conceptualization, supervision, writing, funding acquisition

Appendix A. Supplementary data

The following is the Supplementary data to this article: