A Bayesian adaptive feasibility design for rare diseases

Abstract

It is important for researchers to carefully assess the feasibility of a clinical trial prior to the launch of the study. One feasibility aspect that needs to be considered includes whether investigators can expect to successfully achieve the sample size needed for their trial. In this manuscript, we present a Bayesian design in which data collected during a pilot study is used to predict the feasibility of a planned phase III trial. Specifically, we outline a model that predicts a target sample size obtained from the Gamma-Poisson distribution. In a simulation study, we showcase the utility of the proposed design by applying it to a phase III trial designed to assess the efficacy of mycophenolate mofetil in individuals with mild systemic sclerosis. We demonstrate that the predictive nature of the proposed design is particularly useful for rare disease clinical trials and has the potential to greatly increase their efficiency.

Highlights

• •Pilot trials can be a useful way to make decisions regarding large-scale trials.
• •Recruitment is an important component of assessing trial feasibility.
• •Trials designed within a Bayesian framework allow for effective use of information.

1. Background/aims

In many medical disciplines, phase III clinical trials are critical in the assessment of potentially beneficial treatments [1]. In fact, well designed phase III trials aim to provide some of the highest-quality evidence regarding treatment effectiveness. To achieve this goal, phase III confirmatory trials often require a substantial investment of time from both patients and clinicians, as well as large financial investments from funding bodies or industry. It is, however, common for phase III trials to be inconclusive and fail to address the research questions they were designed to investigate. In many instances, phase III trials fail to recruit the target sample size which results in low power and precision in estimating the treatment effect [2]. Therefore, it is important for researchers to carefully assess the feasibility of a clinical trial prior to the launch of the study. This can be achieved through pilot clinical trials.

A pilot trial is generally a small-scale version of the target clinical trial designed to assess the feasibility of the trial on a larger scale [3]. A typical pilot trial framework includes collecting data regarding a certain design characteristic of the large-scale trial. Example features that pilot trials are designed to assess include recruitment, treatment adherence, randomization procedures, or outcome assessment [[3], [4], [5]]. The data collected during the pilot trial can then be used to make predictions about the design characteristic of interest [6]. Finally, these predictions can be used to determine if and how the investigators should continue with the design and planning of the large-scale trial, according to some prespecified criteria.

The flexibility offered by the Bayesian paradigm provides useful applications to the design of clinical trials, including pilot trials. One of the paradigm's most notable advantages is the ease with which Bayesian designs facilitate formal incorporation of information from various sources [7,8]. This feature enables researchers to effectively use available information. For instance, in situations where investigators have access to historical data, this external information can be combined with data collected in the pilot study through the use of informative priors [9]. By making predictions based on data acquired through a pilot trial, as well as previously available historical information, investigators can be more confident about the feasibility conclusions. Determining whether a phase III trial is feasible is an important decision and, thus, it is best when this decision can be based on all available information.

As already stated, pilot trials can be designed to assess a range of different design characteristics. However, in this manuscript, we will focus on pilot trials designed to assess the recruitment rate of a planned phase III trial. Especially within the context of rare diseases, recruitment rates can vary greatly between geographic locations, and even recruitment centres [10]. Thus, predicting the number of participants investigators can expect to recruit is an important aspect of the design of a phase III trial. To address this issue, we propose a Bayesian pilot design with go-no go decision rules at the end of the pilot phase that rely on the predicted target sample size obtained from a Gamma-Poisson model [11]. We will combine historical information regarding recruitment rates with data from the pilot trial to predict the recruitment rates in a planned phase III trial. The proposed design settings are tailored to a trial for Systemic Sclerosis (SSc), a rare autoimmune disease. However, the proposed methods can be applied broadly in numerous other disease settings.

In Section 1.1 of this manuscript, we first describe the motivating SSc trial. In Section 2, we provide a description of the proposed design and model. Section 3 follows with a simulation study that aims to assess the use of the proposed model within the motivating trial. We end the manuscript with a discussion in Section 4 and a conclusion in Section 5.

1.1. Motivating example

SSc is a rare and life-threatening autoimmune disease, and the rheumatic disease with the highest case-based mortality [12]. Interstitial lung-disease (ILD) is a complication found in 50 % of patients with SSc and is the leading cause of SSc-related mortality [[13], [14], [15]]. Mycophenolate mofetil (MMF) is an effective treatment for individuals with moderate to severe SSc-ILD [16]. However, it is unknown whether MMF would also be beneficial to patients with mild SSc-ILD. Given that MMF is an immunosuppressant with associated risks, the benefits in the setting of mild SSc-ILD should be demonstrated to justify those risks [17].

There is, thus, interest and need to run a phase III clinical trial to assess the efficacy of MMF in patients with mild SSc-ILD. However, given the rarity of this condition, the feasibility of this phase III trial with respect to recruitment needs to be assessed. Specifically, as patients with mild SSc-ILD represent only a subset of an already rare disease [18], investigators are concerned that they might not be able to recruit the number of participants needed to effectively test the efficacy of MMF in this population. Further, as mentioned above, since MMF is associated with potential risks [17], investigators are uncertain whether patients with only mild disease would agree to participate in a trial with this medication. Therefore, in terms of feasibility, there is concern regarding recruiting enough participants in general; but also, enough participants that will fully adhere to their assigned arm in order to obtain complete data during outcome assessment. This uncertainty provides rationale for SSc-mILD (clinicaltrials.gov NCT05785065), a pilot trial designed to assess the feasibility of a phase III trial.

Notably, the investigators have access to historical estimates for the monthly rate of recruitment at each of the clinical sites currently planned to be used during the phase III trial. These estimates come from a multi-site observational cohort [17] and, thus, do not directly apply to recruitment for a randomized controlled trial. Nonetheless, they can be informative of recruitment patterns in a range of clinical site sizes with varying recruitment rates. In this case, historical estimates are calculated from the number of individuals observed in the aforementioned observational cohort [17] that would match eligibility criteria for SSc-mILD. This information can be incorporated into the assessment of feasibility using informative priors. Given all this information, the primary objective of SSc-mILD is to determine if it is realistic to reach the sample size needed for the phase III trial with varying numbers of clinical sites of different recruitment rates.

2. Methods

2.1. Overview of design

Consider a pilot trial that runs for $m$ months at $k_s$ number of sites consisting of $s$ different sizes defined according to the expected rate of recruitment. Following recommendations proposed in the literature, the number and size of sites chosen to be included in the pilot trial should reflect the range of clinical sites that will be used during the phase III trial [4]. Similarly, the length of the pilot trial should be sufficient to reflect the timeline for initiation of recruitment and accommodate any potential delays in opening the trial sites [4]. Further, the timeline should be chosen to capture variations in patient flow as well as seasonal fluctuations. After these design characteristics are specified, each clinical site included in the pilot trial will record the number of eligible patients successfully recruited every month.

Further, suppose that historical recruitment data, obtained from an observational cohort, are available for each site size. This external information can be incorporated using prior distributions. In the next Section, we will explain in detail the construction of these priors based on historical rates.

Using a Bayesian model, the data collected through the pilot trial will be used to update the historical estimates of the monthly recruitment rate of each site size. These updated estimates will then be used to obtain posterior predictive probabilities of successfully recruiting the sample size required to run the planned phase III trial under varying scenarios.

The investigators will then compare the posterior predictive probabilities to a prespecified threshold, $t$; and following previously determined criteria, they will decide whether to continue with the phase III trial as planned or not. Following what is proposed by Hampson et al. [19], the prespecified threshold, $t\text{,}$ should be defined so that accepted probabilities of recruiting the target sample size for the planned phase III trial are high, such as above 0.8 or 0.9. Further, these posterior predictive probabilities inform decisions with respect to the number and composition of the trial sites needed to achieve the target sample size. An exemplary set of decision-making criteria are summarized in Table 1.

Table 1.

Decision rules to continue, adapt, or stop the trial, based on feasibility outcome.

Posterior predictive probability of reaching target sample size over $M_T$ number of months ($\pi )$	Decision
$\pi \ge$$t$ with 1 site of each size $s=1,\dots ,S$	Continue recruitment in the same way during planned phase III trial.
$\pi <t$ with 1 site of each size $s$, but $\pi \ge t$ with any combination of site size, up to a total of nine sites a	Begin enrolment in additional clinical sites; adapt recruitment strategies based on identified barriers.
$\pi <t$ with any combination of site size, up to a total of nine sites a	Stop trial due to lack of feasibility or add sites outside of the established network.

^aA total of nine sites is specified because this is the number of available sites that can be used in the phase III trial specified in Section 1.1.

2.2. Model

The historical information, specific to the target trial, about the monthly recruitment rate at each site size is summarized by a Gamma prior distribution. We assume the number of participants recruited per month at each site follows a Poisson distribution with a rate parameter that can vary depending on the size of the site. Therefore, to obtain posterior distributions using a closed-form solution, we have opted to use a Gamma distribution for each prior, as the Gamma distribution is conjugate to a Poisson likelihood [20]. Each prior distribution for site size $s=1,\dots ,S$ is as follows,

$${\lambda }_s\sim Gamma\left({\alpha }_s,{\beta }_s\right)\text{where}{\alpha }_s>0,{\beta }_s>0$$

where the parameters ${\alpha }_s$ and ${\beta }_s$ are specified to meet the historical estimates of the mean monthly recruitment rate as well as variation in the recruitment rates for a given site size. More specifically, values for ${\alpha }_s$ and ${\beta }_s$ are chosen such that:

• $\frac{{\alpha }_s}{{\beta }_s}=$ the mean monthly recruitment rate of the historical estimate for site type $s$; and

• $\frac{a_s}{{\beta }_s^2}=$ the variance derived from the range of the recruitment rates for site type $s$.

After the completion of the pilot trial, the data collected is modelled using a Poisson likelihood and will be used to update each prior distribution. Suppose that there are $k_s$ sites of size $s$ included in the pilot study. The result is a posterior distribution for site size $s$:

$$p\left({\lambda }_s|y_P\right)\sim Gamma\left({\alpha }_s+\sum _{j=1}^{k_s}\sum _{i=1}^m{y}_{ij},{\beta }_s+m\right)\text{where}{\alpha }_s>0,{\beta }_s>0$$

where $y_P=\left(y_{ij},m\right)$, $i=1,\dots ,m$, $j=1,\dots ,k_s\text{,}$ where $y_{ij}$ represents the number of individuals recruited in site $i$ in month $m$ , where $m$ is the length of the pilot study in months and $k_s$ is the number of sites of size $s$ in the pilot phase.

Then, following the model put forth by Anismov and Federov [11], the Gamma-Poisson distribution is used to predict the number of participants the investigators should expect to recruit at any site size given the corresponding site's pilot monthly recruitment. The posterior predictive distribution of the number of participants recruited per month is then given by:

$$p\left(N_s|y_P,K_s,M\right)=\int L\left(N_s|{\lambda }_s,K_s,M\right)\ast p\left({\lambda }_s|y_P\right)d{\lambda }_s$$

where $N_S$ represents the predicted number of recruited participants for site size $s$ in the target trial, $y_P$ represents the data collected during the pilot trial for that site size $s$, $K_s$ is the number of sites of size $s$ to be included in the target trial and $M$ denotes the length of the target trial in months. For instance, some of the decision-making criteria noted in Section 2.1 requires an estimation of the predicted number of recruited individuals with additional sites. If desired, the inclusion of any additional sites, of a specific size, is specified in this step. Next, the predictions generated for each site size are summed and compared to the target sample size, $N_T\text{,}$ needed for a planned phase III trial. Finally, Monte Carlo simulation is used to obtain the posterior predictive probability of successfully achieving the target sample size, $N_T\text{,}$ over the new conditions, $K_s$ and $M$ across all site sizes $s$. Therefore, the probability of continuing with the phase III trial, $\pi \text{,}$ is defined as follows:

$$\pi =p\left(\left[\sum _{s=1}^S{N}_s\right]>N_T|y_P,K_1,\dots ,K_S,M\right)$$

Having obtained $\pi$, we will then proceed to make decisions according to the rules described in Table 1.

3. Results

3.1. Simulation setup

We conducted a simulation study to assess the proposed feasibility design with respect to power, i.e., probability of making a decision (with varying composition of sites) to continue with the phase III trial outlined in Section 1.1 given various assumptions regarding recruitment rates. We conducted simulations across a range of scenarios, the details of which are specified in Table 2.

Table 2.

Specifications for simulation scenarios.

	Assumed ‘true’ monthly recruitment rate
Target Sample Size	Percentage (In comparison to historical rates)	Site size
		L	M	H
90	50 %	0.0835	0.165	0.25
	75 %	0.12525	0.2475	0.375
	100 %	0.165	0.33	0.5
	125 %	0.29875	0.4125	0.625
135	50 %	0.0835	0.165	0.25
	75 %	0.12525	0.2475	0.375
	100 %	0.165	0.33	0.5
	125 %	0.29875	0.4125	0.625
180	50 %	0.0835	0.165	0.25
	75 %	0.12525	0.2475	0.375
	100 %	0.165	0.33	0.5
	125 %	0.29875	0.4125	0.625
225	50 %	0.0835	0.165	0.25
	75 %	0.12525	0.2475	0.375
	100 %	0.165	0.33	0.5
	125 %	0.29875	0.4125	0.625

Without loss of generality, simulation scenarios are defined considering three site sizes: one with a low monthly recruitment rate (L), one with a moderate monthly recruitment rate (M), and one with a high monthly recruitment rate (H). This specification is based on actual sites that are currently included in the pilot study and planned to be included in the phase III trial. For each scenario, we assumed a hypothetical true monthly recruitment rate at each site size. These hypothetical rates are percentages of the historical estimates of the monthly recruitment rate for each site size. The historical estimate for each site size is based on the average number of patients recruited into the observational cohort described in Section 1.1 who would fit the trial criteria.

Under each scenario, we generated Poisson distributions with rate parameters set to the hypothetical true rates to simulate data of a pilot trial that was set to run for one year. We performed 1000 iterations for each simulation scenario. For each iteration, we fit the model specified in Section 2.2 to obtain a posterior predictive probability of successfully achieving a range of target sample sizes. These sample sizes are specified based on the sample size needed for the planned phase III trial described in Section 1.1 (∼180). Furthermore, the length of this phase III trial was set to recruit for five years. Finally, by comparing each posterior predictive probability to a prespecified threshold, $t$, we estimated the probability of continuing with the phase III trial (i.e., power of the pilot trial), under each scenario. The code is provided in the Supplementary Material.

3.2. Simulation results

We used simulation, and the model specified in Section 2.2, to estimate the power of the pilot trial under the range of simulation scenarios outlined in Table 2. For the results presented in Fig. 1, Fig. 2, Fig. 3, we used a threshold of 0.80 $\left(t=0.80\right)$ to estimate power under each scenario. In the base situation (Fig. 1), where the number of sites in the target trial cannot be increased beyond that of the pilot study (one site of each size), regardless of the assumptions regarding the recruitment rate, the pilot trial is underpowered. In fact, for the most part, the probability of continuation is estimated to be negligible. These results are expected as it is unlikely that with the site composition of the pilot study, the phase III trial will successfully achieve the target sample size.

Fig. 1.

Estimated power of the pilot trial with one each of site L, site M, and site H.

Fig. 2.

Estimated power of the pilot trial with different combinations of an additional three sites.

Fig. 3.

Estimated power of the pilot trial with different combinations of an additional six sites.

We then estimated the power of the pilot trial with the inclusion of different combinations of three additional sites. The results of these simulations are presented in Fig. 2. In these scenarios, certain combinations of an additional three sites result in satisfactory power. For instance, for a target sample size of 90, the addition of three sites with a high monthly recruitment rate result in an estimated power over 80 % across hypothetical true monthly recruitment rates of at least 75 % of the historical estimates. This indicates that under these scenarios, it is likely that the planned phase III trial would successfully achieve a target sample size of 90 over five years. However, for a target sample size of 180, none of the simulation scenarios result in sufficient power. Therefore, even with the inclusion of three additional sites of any size, it is unlikely the planned phase III trial would be able to successfully achieve the required sample size with six sites, if the true monthly recruitment rates are similar to the hypothetical ones assumed in these simulations.

Finally, we estimated the power of the pilot trial with the inclusion of different combinations of six additional sites. The results of these simulations are presented in Fig. 3. In these scenarios, six of nineteen combinations of an additional six sites result in an estimated power greater than 80 % for a target sample size of 90, across all assumed true monthly recruitment rates. This indicates that under these scenarios, it is likely that the planned phase III trial would be able to successfully recruit 90 participants over five years.

For a target sample size of 135 and 180, six of nineteen combinations of an additional six sites result in an estimated power greater than 80 %, among hypothetical monthly recruitment rates of at least 100 % of the historical estimates. This indicates that it is possible that the planned phase III trial will be able to successfully achieve its target sample size of 180. However, these results show that this is only likely with the addition of sites of mostly medium or high monthly recruitment, and if the actual monthly recruitment rates are at least 100 % of what they have been historically.

We also estimated the power of the pilot trial using a more conservative threshold ($t=0.90$) to investigate sensitivity of power estimates to the decision threshold. The power estimated using two different thresholds ($t=0.80,t=0.90$) are presented in Fig. 4. As expected, estimated power is higher when using a more permissive threshold of 0.80. Further, the difference between power estimates using the two thresholds is less with a greater number of sites. For instance, consider the differences between power estimates using the two thresholds for the situation in which the target sample size is 90 and we assume rates 125 % of the historical monthly recruitment rates. When there are only three sites, the power is improved by about 6.2 % with a more permissive threshold while the improvement is only 1.9 % when the total number of sites is six. There is a negligible difference in power when the total number of sites is nine.

Fig. 4.

Estimated power of the pilot trial using two different thresholds ($t=0.80,t=0.90$).

4. Discussion

In this manuscript, we have proposed a Bayesian pilot design together with an analysis model that uses data collected during the pilot trial to assess the feasibility of a planned phase III clinical trial. Specifically, we have described the case where monthly recruitment rates observed during a pilot trial are used to obtain an estimate of the probability of moving forward with the phase III trial (i.e., power of the pilot trial). Further, this design allows for the estimation of this probability with the addition of extra sites with varying recruitment rates. The predictive nature of the proposed model is particularly useful because it enables investigators to launch phase III clinical trials with confidence regarding achieving a target sample size. This further guarantees that a planned phase III trial will have sufficient power to answer the research question it was set out to address.

Another advantage of this design, that was not the focus of this manuscript, is the use of efficacy data collected during the pilot trial. While the pilot trial design focuses on recruitment, the trial could also randomize patients to treatment and control arms to collect data on the efficacy of the treatment that will be under investigation during the phase III trial. Then, should the pilot trial conclude feasibility of the planned phase III trial, this efficacy data can be used as prior information in the phase III trial. Even in cases where randomizing patients to the treatment is not considered in the pilot study, the data may be used to supplement the control arm of the phase III trial with appropriate adjustments. This further increases the efficiency of the phase III trial which, as previously mentioned, is extremely desirable for rare disease trials.

Using the case of SSc as a motivating example, we also used the proposed model to investigate the feasibility of a planned phase III trial that would estimate the efficacy of MMF in patients with mild SSc-ILD. As previously established, the sample size that would provide sufficient power to effectively test the efficacy of MMF in patients with mild SSc-ILD is 180. This sample size is based on a frequentist sample size calculation, which is common practice for phase III trial design. The results of our simulation study show that the three sites currently planned for the phase III trial will likely not be sufficient to achieve this target sample size, if the true monthly recruitment rates are close to what they have been historically. However, our simulations also show that high power can be achieved with certain combinations of six additional sites. Therefore, following the decision criteria outlined in Table 1 and if the true monthly recruitment rates are close to what they have been in the past, it is probable that the decision following the pilot trial will be to continue with the planned phase III trial. However, adjustments to recruitment strategies will likely be needed.

If desired, the model can be altered to accommodate for other situations akin, but not identical, to the motivating example described in this manuscript. The first situation we consider is the one in which investigators have recruitment sites that they consider to have varying importance. For instance, if among all sites researchers have a primary recruitment site in which they wish to pay close attention to. The proposed feasibility design would be specifically advantageous in this case because it considers the recruitment rate at each site separately and would allow the investigators to identify any recruitment issues in the primary site. Moreover, the decision rules could be modified accordingly and could have specific criteria regarding the posterior distribution of the recruitment rate at the primary site.

The second situation we consider is the one in which investigators opt to model the influence of geographic proximity on the recruitment rate at each site formally. This can be done by referring to methods in the work by Jarvis et al. [21]. However, it should be noted that, in the proposed feasibility design, the influence of geographic proximity would already be reflected in the recruitment rates observed at each site during the pilot trial. Therefore, while it is certainly possible to model the influence of geographic proximity formally, it would not necessarily result in a more informative decision regarding moving forward with a planned phase III trial.

The third situation we consider is the one in which investigators wish to consider dropout rates, instead of only recruitment rates. If investigators collect data on the number of dropouts at each site during the pilot trial, then this information could be used to obtain posterior distributions of the dropout rates at each site. Predicted sample sizes could then incorporate dropouts by subtracting the predicted number of dropouts from the predicted number of recruited participants. Moreover, if investigators wish to base their decision of moving forward with a planned phase III trial based on both recruitment and dropout rates, decision rules would need to be modified accordingly.

A limitation of the proposed design is that it assumes the monthly recruitment rates observed during the pilot trial do not change throughout the course of the phase III trial that follows. When using the proposed design, if the pilot trial runs over a full year, seasonal fluctuations in monthly recruitment rates should be captured in the pilot trial data, and thus, will also be considered in everything that follows [4]. However, if desired, seasonal variation can be modelled formally [22]. Other patterns in recruitment rates are also possible, for instance, if recruitment reaches a plateau at one, or more, of the sites under consideration. In the case of the simulation study performed in this manuscript, which assumes that the planned phase III trial will run over five years, a constant monthly recruitment rate assumption may be violated. It is possible that the application of methods discussed in relevant literature, such as those put forth by Turchetta et al. [23], could be used to circumvent this assumption. This is something that could be explored by future work.

5. Conclusion

We conclude by restating that pilot trial data can be a useful way to aid investigators in their decision to commit to larger-scale trials. Further, pilot trials designed within a Bayesian framework allow for easy incorporation of external information, such as historical data. The main advantage of the design proposed in this manuscript is that it formally incorporates uncertainty associated with the pilot data and historical information to present predictions with quantified uncertainty that can be used for decision making based on clearly defined decision rules. Further, the simplicity of the proposed model and design facilitates their adoption in practice.

Funding

Maureen Churipuy is supported by a trainee award from the Canadian Network for Statistical Training in Trials (CANSTAT) funded by Canadian Institutes of Health Research (CIHR) grant #262556. Other portions of this work were funded by a CIHR grant #183643 awarded to Dr. Sabrina Hoa and a McGill Interdisciplinary Initiative in Infection and Immunity grant awarded to Dr. Marie Hudson.

CRediT authorship contribution statement

Maureen M. Churipuy: Writing – review & editing, Writing – original draft, Visualization, Methodology, Investigation, Formal analysis, Data curation. Shirin Golchi: Writing – review & editing, Supervision, Methodology, Conceptualization. Marie Hudson: Writing – review & editing, Supervision, Conceptualization, Methodology. Sabrina Hoa: Funding acquisition, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article:

Data availability

R script used to generate the results in this manuscript is available is the supplementary material.