Accrual Prediction Program: A Web-Based Clinical Trials Tool for Monitoring and Predicting Accrual for Early Phase Cancer Studies

Abstract

Background

Monitoring subject recruitment is key to the success of a clinical trial. Accordingly, researchers have developed accrual-monitoring tools to support the design and conduct of trials. At an institutional level, delays in identifying studies with high risk of accrual failure can lead to inefficient and costly trials with little chances of meeting study objectives. Comprehensive accrual monitoring is necessary to the success of the research enterprise.

Methods

This paper describes the design and implementation of the University of Kansas Cancer Center Accrual Prediction Program, a web-based platform was developed to support comprehensive accrual monitoring and prediction for all active clinical trials. The Accrual Prediction Program provides information on accrual, including the predicted completion date, predicted number of accrued subjects during the pre-specified accrual period, and the probability of achieving accrual targets. It relies on a Bayesian accrual prediction model to combine protocol information with real-time trial enrollment data, and disseminates results via web application.

Results

First released in 2016, the Accrual Prediction Program summarizes enrollment information for active studies categorized by various trial attributes. The web application supports real-time evidence-based decision making for strategic resource allocation and study management of over 120 ongoing clinical trials at the University of Kansas Cancer Center.

Conclusions

The Accrual Prediction Program makes accessing comprehensive accrual information manageable at an institutional level. Cancer centers or even entire institutions can reproduce the Accrual Prediction Program to achieve real-time comprehensive monitoring and prediction of subject accrual to aid investigators and administrators in the design, conduct, and management of clinical trials.

Background

According to a 2010 Institute of Medicine report, clinical trials cooperative groups must develop and implement a more systematic approach for selecting trials and improving recruitment and trial completion rates [1]. The National Cancer Institute subsequently created the National Clinical Trials Network in March 2014 with effort focused on creating a streamlined system for developing and managing cancer clinical trials [2,3]. Optimization of trial initiation and accrual rates is still needed [4–6].

Researchers in the Biostatistics and Informatics Shared Resource of the University of Kansas Cancer Center developed the Accrual Prediction Program as an interactive web-based tool for daily accrual monitoring and prediction for all active University of Kansas Cancer Center clinical trials. It combines current protocol and enrollment information retrieved from a comprehensive research information system to summarize current and projected accrual patterns to inform researchers and administrators. The Bayesian approach [7–9] is an available and applicable method that can utilize researchers/administrators’ previous experience from former trials or clinical opinion and integrate them as prior information. And the model could provide a reliable accrual prediction, by combining the prior information and the enrollment data from ongoing study. Experience with clinical trials shows that monthly accrual rates vary from month to month (apart from the rare studies that accrue very quickly) and that investigators tend to overestimate achievable accrual rates; thus, the accrual prediction would be more accurate at the late stage. The first systematic study of Bayesian accrual approach was proposed by Gajewski et al. [8]. By following this idea, Jiang et al. [9] derived a closed-form distribution for the posterior prediction of accrual, which is easily interpretable and applicable. Therefore, based on the closed form, the Accrual Prediction Program employs a Bayesian statistical model [9] to combine current trial enrollment with the targeted enrollment, trial initiation date, and targeted completion date to produce a predicted completion date, predicted number of accrued subjects during the pre-specified accrual period, and the probability of achieving accrual targets by the targeted completion date. The Accrual Prediction Program also provides prediction intervals as measures of the accuracy of its predictions.

In this paper, we describe the design and implementation of the Accrual Prediction Program, introduce a set of features accessible through the interactive web application, and conclude with a discussion of the benefits of reproducing it at other institutions. Users of the Accrual Prediction Program web application can explore the clinical trials information, including treatment target, sponsor, principal investigator, and current/projected enrollment. Most importantly, it identifies studies with a high risk of accrual failure early, supporting strategic decisions for resource allocation and study management for all active clinical trials.

Methods

Overview.

The Accrual Prediction Program is constructed via three sequential steps as presented in Figure 1. The final product—the University of Kansas Cancer Center Accrual Prediction Program web application—relies on Clinical Trials Office staff to provide current clinical trial protocol and enrollment information, and on the researchers to generate accurate accrual predictions for dissemination. This office is the centralized support organization for the University of Kansas Cancer Center, offering services that include (among others) project development and management, protocol review and monitoring, and data management coordination. The office administrators’ efforts ensure that all research activities protect patient safety and have scientific merit and adequate resources to meet their specific objectives. The Biostatistics and Informatics Shared Resource provides expertise in study design, statistical oversight and analyses, clinical research informatics and data management, electronic data collection, bioinformatics, and statistical genomics. With expertise in biostatistics, bioinformatics, and informatics—the three areas that encompass data science—the Biostatistics and Informatics Shared Resource is uniquely positioned to enhance cancer research by developing and testing new research data repositories and analytic tools.

Figure 1.

The flowchart of the Accrual Prediction Program. Figure 1A shows the Clinical Trials Office administration of each clinical trial. Figure 1B displays the process of accrual computation using R software, which starts with importing daily Clinical Trials Office data and ends with an export of the accrual report in Hypertext Markup Language (HTML) format. Figure 1C presents the Intranet operation by incorporating all the HTML files into accrual server.

To support an up-to-date and fully-functioning web application, administrators enter active trial data into the Comprehensive Research Information System. Then researchers automatically retrieve the trial information and import it into R [10] in order to run the accrual prediction model [9,11], then use rmarkdown [12] to generate an accrual report in Hypertext Markup Language (HTML) format that is incorporated into the web application. The team execute this process daily to support optimal decision making for trial management and resource allocation. Each of the steps displayed in Figure 1 is described in detail in the following sections.

Clinical Trials Office Administration of Clinical Trial Data.

The Accrual Prediction Program is refreshed once every 24 hours using protocol and accrual information entered, updated and maintained. Comprehensive Research Information System is a secure web-based clinical information management system housed within and maintained by the Biostatistics and Informatics Shared Resource. Based on our experience working with the Clinical Trials Office staff, active trial data are being entered in a timely manner because every time a participant is enrolled into a study the data feed directly into the accrual prediction program. Other programs that adopt our program will need to be aware of how data are entered and adjust if it is not done so in a timely manner. The Biostatistics and Informatics Shared Resource creates, tests, and deploys all trial-specific electronic case report forms in Comprehensive Research Information System, making it a secure, centralized location for entering and storing trial-specific data and protocol information in a standardized format compliant with National Institutes of Health reporting standards. In addition, due to the trial budget, unplanned delays or interruption, the protocol amendments might affect the trial-specific data. However, the variables that applied by the model are able to automatically retrieve the latest trial information, once the trial-specific data in Comprehensive Research Information System being updated. Table 1 displays an example of the trial-specific data entered into the system by clinical administrators. The variables include: $n$, the target sample size for the trial; $T$, the target completion time (in units defined by $StudyUnit$) to achieve the sample size goal; $m$, the current observed sample size; and $t_m,$ a system-calculated time to date defined as the time elapsed since the trial activation date. Prior to trial initiation, researcher elicits a prior probability, $P$, from the study’s principal investigator that represents his or her confidence that the trial will complete on time [8].

Table 1:

A typical example of the protocol and enrollment data for clinical trials entered into Comprehensive Research Information System by the Clinical Trials Office.

$n$	$T$	Study Unit	Activation Date	Enrolled Subjects ($m$)	Sponsor ID a	Study Number a	Disease Working Group a	Title a	Principal Investigator a
12	24	months	01/10/2017	5	ABCD 1234	STUDY 001	Lung	Title: AAAAA	AAA
25	36	months	11/25/2015	19	KUCC 1234	STUDY 002	Leukemia/Myeloid	Title: BBBBB	BBB
30	36	months	07/17/2015	28	KUCC 1001	STUDY 003	Breast	Title: CCCCC	CCC
5	12	months	12/05/2017	3	ABCD 1001	STUDY 004	Lymphoma/Myeloma	Title: DDDDD	DDD
5	18	months	04/03/2018	4	ABCD 1002	STUDY 005	Lymphoma/Myeloma	Title: EEEEE	EEE

^aThese values are blinded throughout this paper.

For example, suppose a lung cancer study that planned to recruit 12 subjects in 24 months was activated on 01/10/2017. The principal investigator was 50% confidence prior to initiating the study that it would achieve this target enrollment within the targeted enrollment period. On 05/10/2018, the study had enrolled 5 subjects. The information required for running the accrual prediction is given by: $n=12$, $T=24$, $P=0.5$, $m=5$, $t_m=16months$. The Accrual Prediction Program uses this data to make an updated prediction of the trial completion date.

Prediction of Accrual.

Once the updated enrollment information for a trial are entered into the system, Biostatistics and Informatics Shared Resource researchers export this data into R and use the “accrual” package [11] to generate accrual predictions. The “accrual” R package uses a Bayesian statistical model [9] to integrate the actual enrollment data with protocol information and the investigator’s prior belief that the trial will achieve an enrollment target within the enrollment period. It generates a predicted completion date and a predicted number of accrued subjects during the pre-specified accrual period, as well as prediction intervals that convey the accuracy of its predictions. It should be noted that the user interface of the Accrual Prediction Program, like that of the R “accrual” package it employs, was deliberately established with user-friendly clinical accrual information instead of complicated statistical terms, making it easy to use for both statisticians and clinical researchers.

Recalling the previous example, the data given by $n$, $T$, $P$, $m$, and $t_m$ are exported from Comprehensive Research Information System and imported into R where they are entered as arguments to the “accrual.T.inform” and “accrual.n.inform” functions. These functions implement the Bayesian model via Markov Chain Monte Carlo methods [13] to obtain posterior predictive distributions for completion time, $T_{Pred}$, and enrollment during the targeted enrollment period,$n_{Pred}$. In addition to numerical summaries, “accrual.T.plot” and “accrual.n.plot” functions return plots of the posterior predictive distributions.

The posterior predictive distributions produced by the Accrual Prediction Program facilitate comprehensive monitoring when applied to all active clinical trials. Trials can be identified as having slow accrual if their targeted completion time, $T$, is smaller than the lower bound of the prediction interval for $T_{Pred}$. Additionally, the model can identify studies with high risk of accrual failure by estimating the probability that a trial will achieve its accrual target late. This risk, $P_{Late}$, offers an effective way of monitoring enrollment progress and facilitates decision making for strategically allocating resources. For example, to calculate the risk for a trial, 50,000 Markov chain Monte Carlo iterations can be drawn from the Bayesian posterior predictive distribution for $T_{Pred}$. The percentage of posterior predictive draws of $T_{Pred}$ that are greater than the targeted completion date $T$ is the trial’s estimated risk of accrual failure: $P_{Late}=Pr\left(T_{Pred}>T\text{|}Data\right)$. In this way, $P_{Late}$ reflects a data-based update to the principal investigator’s prior belief that the trial will achieve its targeted enrollment.

An example of the posterior predictions produced by the “accrual” R package are given in Table 2. After all predictions for a trial are obtained, an accrual report that includes results, plots, and proper interpretation is generated in HTML format using rmarkdown [12]. All HTML reports are then incorporated into an accessible web server under a local, restricted communications network. The end result is an interactive and user-friendly web-based tool that allows for exploration of active trial information and accrual prediction results [14,15].

Table 2:

Trial accrual information and the probability of being late. (All trial information was blinded).

Study Number	Disease Working Group	Principal Investigator	Expected Completion Date $(T)$	Prediction Completion Date ($T_{Pred}$) Median	95% CI Lower Bound	95% CI Upper Bound	Probability of being Late $\left(P_{Late}\right)$
STUDY 001	Lung	AAA	01/10/2019	12/01/2019	12/17/2018	04/08/2022	96.28%
STUDY 002	Leukemia/Myeloid	BBB	11/25/2018	02/25/2019	09/09/2018	01/27/2020	81.70%
STUDY 003	Breast	CCC	07/17/2018	08/04/2018	06/11/2018	01/07/2019	68.07%
STUDY 004	Lymphoma/Myeloma	DDD	12/05/2018	09/28/2018	06/17/2018	12/18/2019	30.59%
STUDY 005	Lymphoma/Myeloma	EEE	10/03/2019	01/03/2019	07/13/2018	01/21/2021	14.47%

Results

Currently (as of 06/01/2018), 128 registered clinical trials are active and ongoing at the University of Kansas Cancer Center. The Accrual Prediction Program platform is used by the Clinical Trials Office and investigators to monitor updated enrollment information and combine it with current protocol information to predict accrual for all active trials. Take as an example Study 001 from Table 2. This trial was activated on 01/10/2017 with plans to enroll 12 subjects in 24 months. At present, 5 subjects have been enrolled in this trial. The Bayesian accrual prediction model predicts a completion date of 12/01/2019 with a 95% prediction interval ranging from as early as 12/17/2018 to as late as 04/08/2022. Additionally, this trial has a high risk for extended accrual time, with $P_{Late}=0.9628$ a reflection of the study’s slow accrual.

Authorized users within the University of Kansas Medical Center network can access the Accrual Prediction Program web application, shown in Figure 2. Individual trial accrual reports that include results interpretation and graphics can be accessed by clicking the hyperlinked study number. The clinical trial information and accrual prediction information will display in a new page [15]. Figure 3 presents an example of a Phase III study with expected 36 months duration activated on 11/25/2015. It has a predicted completion date of 02/25/2019 with a 95% prediction interval ranging from 09/09/2018 to 01/27/2020. The probability that the trial will achieve its accrual target late is 0.8170, which reflects its slow accrual: with only 8 months left 6 more subjects are required to achieve the targeted enrollment of 25 Subjects. Moreover, the plots of the posterior predictive distributions display the 95% credible interval for the number of accrued subjects in the targeted 36 months (19 – 27 subjects) and the 95% credible interval for completion time for the targeted 25 subjects (33.5 months – 50.1 months). Both intervals contain the targeted enrollment and completion time, indicating the study is on target for achieving accrual.

Figure 2.

A screenshot of the Accrual Prediction Program web application. The University of Kansas Cancer Center (KUCC) web application summarizing the clinical information and current enrollment data via the Accrual Prediction Program to support comprehensive accrual monitoring and prediction for all active clinical trials.

Figure 3.

Example of the accrual prediction and posterior predictive distribution plots for one clinical trial. Selecting a specific Study Number, for example, STUDY00002268 will open a new page displaying the results and interpretation. It contains the accrual prediction with visualization plots to demonstrate the predicted completion date with a 95% prediction interval and the posterior predictive distribution.

The Clinical Trials Office uses this information to summarize and report on the accrual progress of all trials across various indicators (e.g., by principal investigator or disease working group). Table 3 presents a summary of the risk for accrual failure across disease working groups in order by the average level of risk for trials within the working group. Of note are the brain and sarcoma/melanoma groups, whose risk is lowest among all disease working groups. Conversely, the multiple early phase clinical trials group has the largest risk.

Table 3:

The distribution of the probability of being late across disease working group ordered by mean.

Disease Working Group	Count	Mean ($P_{Late}$)	SD ($P_{Late}$)
Brain	4	56.37%	0.4060
Sarcoma/Melanoma	6	57.16%	0.3948
Multiple	19	57.85%	0.1083
Leukemia/Myeloid	24	63.52%	0.3378
Breast	13	64.20%	0.3747
Lymphoma/Myeloma	12	69.12%	0.2609
Lung	11	69.33%	0.2585
Gynecology	5	70.71%	0.2302
Gastrointestinal	9	70.85%	0.2627
Genitourinary	10	79.78%	0.2767
Head and Neck	2	86.01%	0.1399
Multiple Early Phase Clinical Trials	4	90.53%	0.1577

The Clinical Trials Office also relies on graphical summaries such as histogram and volcano plot. A histogram of the probability of being late for all active trials, relaying critical information that most of these trials have a more than 50% chance of being late meeting accrual goals. This probability indicator assists investigators and administrators in deciding whether to maintain or increase accrual efforts. Another useful graphic is the volcano plot (Figure 4), which displays a scatterplot of the ratios between the predicted completion dates $T_{Pred}$ and the expected completion dates $T$ with $\left(T_{Pred}-T\right)/T$ (x-axis) versus probabilities of being late $P_{Late}$ (y-axis) for all active trials. Trials located in the upper right panel marked by red dots have both a high probability of completing accrual late and a predicted completion date that is far away from the expected completion date—these trials can be marked for slow accrual and risk of accrual failure.

Figure 4.

Volcano plot for the ratio of predicted and expected date versus the probability of being late for each trial. The volcano plot shows a scatterplot with X-axis is the ratios between the predicted completion dates $T_{Pred}$ and the expected completion dates $T$ with $\left(T_{Pred}-T\right)/T$, and the Y-axis is the probabilities of being late $P_{Late}$ (y-axis) for each clinical trial.

In summary, the Accrual Prediction Program platform allows authorized users to search all active trials on a variety of trial features, including the principal investigator’s name, disease working group, and study title. Users are also able to sort the clinical trials based on any information that is produced by the Accrual Prediction Program platform, including the sample size, expected duration, study start date, number of enrolled subjects, and predicted completion date.

Conclusions

The Accrual Prediction Program platform is a powerful tool for clinical trial management. Investigators and administrators can access and summarize clinical accrual information easily via the Accrual Prediction Program web application. In addition, the platform provides quick and convenient information on accrual for cancer researchers, including prediction of accrual, time-to-completion, and risk of accrual failure for active clinical trials. These estimates provide the institution with vital information on accrual patterns within and across disease working groups, driving more evidence-based decision making for the design, initiation, management, and termination of clinical trials.

The plots produced by the Accrual Prediction Program platform provide vital visualization of accrual data for individual studies. The Accrual Prediction Program allows for early identification of slowly accruing trials and, more importantly, the consolidation of accrual data and trial information for all active studies with ongoing recruitment can help improve recruitment and trial completion rates for the University of Kansas Cancer Center. All plots can be generated for individual disease working groups or principal investigators by the end user.

There are some existing restrictions for our current release. The Accrual Prediction Program platform was developed for single site trials and does not allow for prediction of accrual for multi-center trials. Furthermore, the similarity among the disease/cancer types may present a similar accrual trial trend, which means the investigators or researchers could borrow previous experience from completed trials into the upcoming trials. That is a novel view to look at the subject accrual progress [16]. Future work includes incorporating Bayesian multi-center models that allow for subjects and sites recruitment into the Accrual Prediction Program platform. Advanced Bayesian model with study design factors and other qualitative data will be considered into model for accrual monitoring and prediction. In addition, complementary software like Shiny R application, for interactive prediction is in development.

In summary, the Accrual Prediction Program supports real-time evidence-based decision making for strategic resource allocation and study management of over 120 ongoing clinical trials at University of Kansas Cancer Center. The Accrual Prediction Program makes accessing comprehensive accrual information manageable at an institutional level. Cancer centers can reproduce the Accrual Prediction Program [14] to achieve real-time comprehensive monitoring and prediction of subject accrual to aid investigators and administrators in the design, conduct, and management of clinical trials to improve recruitment and trial completion rates.