Would the Recommended Dose Have Been Different Using Novel Dose-Finding Designs? Comparing Dose-Finding Designs in Published Trials

PURPOSE

Simulation studies have shown that novel designs such as the continual reassessment method and the Bayesian optimal interval (BOIN) design outperform the 3 + 3 design by recommending the maximum tolerated dose (MTD) more often, using less patients, and allotting more patients to the MTD. However, it is not clear whether these novel designs would have yielded different results in the context of real-world dose-finding trials. This is a commonly mentioned reason for the continuous use of 3 + 3 designs for oncology trials, with investigators considering simulation studies not sufficiently convincing to warrant the additional design complexity of novel designs.

METHODS

We randomly sampled 60 published dose-finding trials to obtain 22 that used the 3 + 3 design, identified an MTD, published toxicity data, and had more than two dose levels. We compared the published MTD with the estimated MTD using the continual reassessment method and BOIN using target toxicity rates of 25% and 30% and toxicity data from the trial. Moreover, we compared patient allocation and sample size assuming that these novel designs had been implemented.

RESULTS

Model-based designs chose dose levels higher than the published MTD in about 40% of the trials, with estimated and observed toxicity rates closer to the target toxicity rates of 25% and 30%. They also assigned less patients to suboptimal doses and permitted faster dose escalation.

CONCLUSION

This study using published dose-finding trials shows that novel designs would recommend different MTDs and confirms the advantages of these designs compared with the 3 + 3 design, which were demonstrated by simulation studies.

INTRODUCTION

Phase I oncology clinical trials are designed to find the recommended dose of a treatment. For most therapeutic agents, the assumption is that a higher dose will be more efficacious although the risk of toxicity increases with higher dose. Hence, it is common to use the maximum tolerated dose (MTD) to define the recommended dose. An optimal design should maximize the frequency in the recommendation and assignment of patients to the true MTD and minimize the recommendation and assignment of patients to subtherapeutic or highly toxic doses. The challenge in dose finding is to balance simplicity in implementation, effectiveness in optimizing patient outcomes, and safety in minimizing risk.

CONTEXT

• Key Objective

• Novel dose-finding designs have been shown in previous simulation studies to outperform the 3 + 3 design in terms of identifying the maximum tolerated dose (MTD). However, it is not clear whether these novel designs would assign or recommend different doses in the context of real dose-finding trials applications. This study applied the continual reassessment method and the Bayesian optimal interval to 22 randomly selected published dose-finding studies that used the 3 + 3 design.

• Knowledge Generated

• In approximately 40% of trials, the recommended MTD using novel designs was higher than the published MTD using 3 + 3 design, with estimated and observed toxicity rates closer to the target toxicity rates of 25% and 30%. The difference was more pronounced with the increasing number of dose levels.

• Relevance

• The use of novel dose-finding methods could lead to different dose assignments and recommendations compared with the 3 + 3 design in the context of real dose-finding trials.

Traditional rule-based methods such as the 3 + 3 design¹ continue to be the most commonly used^2,3 method for finding the MTD in phase I clinical trials because of their simplicity. Criticisms of the 3 + 3 design include slow dose escalation and a rigid structure that does not allow for flexibility in defining the target toxicity rate (TTR) or incorporation of previous data.^4,5 Moreover, multiple simulation studies have shown that novel designs such as the continual reassessment method (CRM)⁶ and the Bayesian optimal interval (BOIN) design⁷ outperform the 3 + 3 design in terms of identifying the true MTD more often while using less patients, allotting more patients to the MTD, and accommodating the design to suit specific drug characteristics.^7-11

Simulation studies have been the primary tool used to compare these designs since they allow researchers to assume a true MTD and evaluate the frequency with which various designs select the correct dose under various trial scenarios. Using this approach, the 3 + 3 design gives biased estimates of the MTD by selecting a dose whose toxicity rate is lower than the TTR¹² and the CRM exposes fewer patients to highly toxic doses compared with the 3 + 3 design.¹³ Simulation studies have also demonstrated comparable average performance between the CRM and BOIN designs.¹⁴ Additionally, Conaway and Petroni¹⁵ found that using the CRM and BOIN designs in phase I yields a substantially higher proportion of effective agents in successful phase III trials compared with the 3 + 3 design.

Despite the advantages of these novel designs in simulation studies, the CRM and BOIN are yet to be fully accepted and adopted in clinical research.^2,16,17 This slow uptake could be due in part to the designs' statistical complexity or clinicians' lack of familiarity with the designs.^18,19 However, these barriers have been largely addressed through the addition of tutorial papers on implementation, modifications of designs,^19-22 new software, and acknowledgment of adaptive designs in guidance documents²³ with limited success. Another common reason mentioned by clinical investigators is that simulation studies are not sufficiently convincing to warrant the additional design complexity given that these novel designs have not been compared in the context of real dose-finding clinical trials to evaluate the trial-specific level differences in dose assignments and selected MTD. In this paper, we apply the CRM and BOIN design to published phase I oncology trials that use the 3 + 3 design and compare the estimated MTD on the basis of the observed data as well as if we had applied the novel designs for each dose assignment. This allows us to evaluate the trial-specific level benefits of using novel designs.

METHODS

To obtain a representative sample of at least 20 dose-finding trials, we randomly sampled 60 rule-based studies published from 2008 to 2014 from Chiuzan et al.³ We first sampled 30 and obtained 11 trials after excluding designs that did not reach an MTD or include toxicity data, those that did not follow the standard 3 + 3 design in practice, and those that had less than three dose levels. We then sampled another 30, which yielded 11 more eligible trials (Fig 1). Thus, a total of 22 trials were systematically reviewed to obtain the number of dose-limiting toxicities (DLTs) and patients at each dose, which we refer to as the published data for each trial. We estimated the MTD using two approaches, first applying the CRM and BOIN to the published data and second applying each design for each dose assignment and implementing design-specific characteristics, such as cohort size and starting dose on the basis of the recommendations for these designs. The MTDs were then compared with the original published MTD.

FIG 1.

Selection of dose-finding trials. DLT, dose-limiting toxicity; MTD, maximum tolerated dose.

Using Published Data

We estimated the MTD with the published toxicity data using the CRM and BOIN designs assuming the TTR of 0.25 and 0.30 without using any simulations. Rates of 0.25 and 0.30 are common targets used to compare the 3 + 3 design with the CRM and BOIN, respectively.^10,24 For the CRM, we used the Bayesian estimation in the R package “dfcrm”²¹ and selected the skeleton using the approach by Lee and Cheung,²⁵ specifying the prior guess of MTD as the median or greatest dose level below the median number of total doses. Since the CRM requires the order of dose assignment and most publications only list the number of patients assigned to each dose level, if the order was not specified, we obtained the most plausible order of dose escalation assuming a 3 + 3 design. For example, given six patients on dose level one, followed by three on dose level two, we assumed that there was one DLT in the first cohort of three and zero in the subsequent cohort on dose level one. For the BOIN, we used the R package “BOIN”²⁴ and the function select.mtd, which fit an isotonic regression to select the dose at which the fitted DLT rate is closest to the TTR. We did not include the expansion cohort data here because these cohorts were not considered for dose escalation in the 3 + 3 designs. We compared the selected MTD using the CRM and BOIN designs with the one in the original published paper.

Using Novel Methods for Actual Dose Assignments

We applied the CRM and BOIN from the start of the trial and assigned doses on the basis of the recommendations for these methods. If the dose assignment was to a dose level for which we had published data and the design used a cohort size of 3, the DLT information for the entire cohort was taken. If the design used a cohort size of one, we sampled the DLT information without replacement from the cohort. If published data were not available or already used, we generated data with a probability of toxicity equal to the observed rate of DLT at the given dose level or the isotonized DLT rate to ensure monotonicity in the dose-toxicity relationship.

For the CRM, we used the crm function from the “dfcrm” package with each new cohort and getprior function for our initial skeleton. We started each trial at the median dose level, did not allow for dose skipping, and considered cohort sizes of one, CRM(1), or three, CRM(3). The sample size was calculated on the basis of the approach by Cheung²⁶ with a probability of correction selection of 0.60 and an odds ratio of two. We chose the sample size for all designs as the closest number larger than or equal to the recommended size that was a multiple of three.

For the BOIN, we used the get.boundary function from the “BOIN” package, with 0.6 × TTR and 1.4 × TTR as the highest toxicity probability deemed subtherapeutic and overly toxic, respectively, a cutoff of 0.95 to eliminate an overly toxic dose, and an offset of 0.05²⁴ to obtain the escalation and de-escalation parameters. The select.mtd function was applied after the maximum sample size recommended for the CRM was reached. We considered an accelerated BOIN using cohorts of one until the first DLT is observed,⁷ BOIN(A), and cohorts of three throughout, BOIN(3). We started both designs at the first dose level.²⁷ For both CRM and BOIN, we repeated this process 1,000 times for each trial and obtained the following measures of comparison from each trial, design, and TTR: the proportion that each dose was selected as the MTD, the proportion and number of patients assigned to each dose, the proportion of patients with toxicities at each dose, and the proportion of patients generated.

To contrast the designs, dose assignment and DLT using the trial by Mita et al²⁸ are displayed in Figure 2 along with dose assignments had the CRM and BOIN been implemented with a sample size of 27 and a TTR of 0.25. The 3 + 3 design had six dose levels with 24 patients and recommended dose level 5 as the MTD with zero DLTs of four patients and two of six at the above dose. It also had an expansion cohort where one of 13 patients experienced a DLT. Both the CRM and BOIN designs recommended dose level 6 as the MTD instead. Starting at dose level 3, the CRMs homed in on the eventual MTD faster and assigned more patients to dose level 6 compared with the BOIN methods, starting at dose level 1.

FIG 2.

Dose assignment for one retraced trial by Mita et al²⁸ compared with original data. (A) CRM(1), (B) CRM(3), (C) BOIN(A), and (D) BOIN(3). The original data are denoted by circles, and the compared method is denoted by triangles. Solid indicates a dose-limiting toxicity. BOIN, Bayesian optimal interval; CRM, continual reassessment method.

RESULTS

Table 1 summarizes the observed DLT data from each trial of the 22 trials ordered by the number of dose levels in the trial. The median number of dose levels was four, and the median sample size was 18. Eight trials (36%) included an expansion cohort. The median sample size including expansion cohorts was 21.5.

TABLE 1.

No. of Dose-Limiting Toxicities and Patients Assigned to Each Dose Level in the Published Study

Using Published Data

The recommended MTD level in the published paper and the one obtained from applying the CRM and BOIN to the published data are shown in Table 2. The recommended MTD by the CRM, using a TTR of 0.25, matched the published MTD in 12 trials (55%), was higher by one dose level in nine trials (41%), and was lower by one dose level in one trial (4%). The estimated DLT rate for the chosen dose ranged from 0.13 to 0.31. The recommended MTD by the BOIN, using a TTR of 0.25, matched the published MTD in 14 trials (64%) and was higher by one dose level in eight trials (36%). The estimated DLT rate for the chosen dose ranged from approximately 0.01-0.40.

TABLE 2.

Comparison of Dose Level Recommended as the MTD With the Original Published Data

For the eight trials for which both the CRM and BOIN recommended higher doses, the DLT rate in the published data was between 0.18 and 0.40, whereas the observed DLT rate of the dose below, chosen by the original design, was ≤ 0.167, with a median DLT rate of 0 (Table 1). The estimated DLT rate by the CRM and BOIN for a TTR of 0.25 was below 0.31 and 0.40, respectively. There were nine trials for the CRM and one for the BOIN in which the design with a TTR of 0.30 chose a higher MTD than that chosen with a TTR of 0.25.

Using Novel Methods for Actual Dose Assignments

Comparison of the MTD selected with a TTR of 0.25 compared with the original study is displayed in Figure 3. For both cohort sizes, the MTD selected by the CRM and BOIN was similar most of the time. In general, the designs would have recommended one dose level higher than the recommended MTD in the published paper in 8 (36%) of the 22 trials most of the time and the same dose as the published paper in 14 (64%). For trials in which the designs recommended one dose higher, the observed DLT rate of the chosen dose was between 0.17 and 0.40, which was closer to the TTR of 0.25 than the dose below. In six of eight of these trials, the 3 + 3 design chose the dose level below with a DLT rate of 0. The trials for which the MTD was the same had a DLT rate of 0.17 or less with the dose above the chosen MTD having a rate ≥ 0.50. The BOIN(3) results differed from the CRM and BOIN(A) for the trials by Kunz et al³¹ and Kim et al.³³ Increasing the sample size by at least three would result in a recommended MTD that matched the other designs. The CRM(3) chose the dose level above more frequently than the CRM(1) and BOIN designs in the trial by Tevaarwerk et al⁴⁰ and Laarhoven et al.⁴⁹

FIG 3.

Selection proportion of MTD with a target toxicity rate of 0.25 compared with the original published data. Information within parentheses of column headers denotes cohort size or structure. BOIN, Bayesian optimal interval; CRM, continual reassessment method; MTD, maximum tolerated dose.

With a TTR of 0.30, shown in Appendix Figure A1, the CRM was more aggressive for a few of the trials and the BOIN(3) more closely matched the other designs. The inconsistency between BOIN(3) with the TTR of 0.25 and 0.30 is a result of slow dose escalation with a TTR of 0.25 since it de-escalates after observing a DLT rate higher than 0.30, which is more conservative than the 3 + 3 design.

For trials with less than five dose levels, the sample sizes recommended by the CRM were typically similar to those used in the 3 + 3 design without an expansion cohort. For trials with five or more dose levels, the CRM had a smaller sample size than what was typically used in 3 + 3 designs. Figure 4 shows the distribution of the proportion of patients assigned to the respective MTD of each method, excluding the expansion cohort for the 3 + 3 design. The proportions of patients assigned to the respective MTD in the BOIN(A) and both CRM designs were higher than those in the original 3 + 3, most notably with the CRM designs. For trials without an expansion cohort, or independent of their expansion cohort, the original design tended to assign a similar if not greater proportion of patients to doses with high DLT rates (> 0.50) compared with the CRM(1), CRM(3), and BOIN(3).

FIG 4.

Distribution of the proportion of patients assigned to the chosen maximum tolerated dose of each respective design across all 22 trials, with a target toxicity rate of 0.25. Information within parentheses on x-axis denotes cohort size or structure. BOIN, Bayesian optimal interval; CRM, continual reassessment method.

It should be noted that given that the CRM started at a higher dose level, approximately half of the patient DLT information was simulated; for the BOIN that started at the same dose as the original trial, about a quarter was simulated (Appendix Table A1). Finally, stopping rules can be imposed for the CRM and BOIN designs, although they were not included in this study.

DISCUSSION

Using published data from 22 dose-finding clinical trials, we demonstrated that using novel designs, the selected MTD would have been the dose above the published MTD for approximately 40% of the trials, both using the published data or if the novel design had been used for the actual dose assignments instead of the 3 + 3 design. The doses selected by the novel designs were closer to the target rate than the previous dose, which had observed rates of 0 most of the time. The CRM with a TTR of 0.30 sometimes resulted in a more aggressive MTD selection when using the published data, but matched closely to the CRM with a TTR of 0.25 when using the CRM for each dose assignment. However, we recommend using a TTR of 0.25 since a TTR of 0.30 does increase the selection of overly toxic doses. On the contrary, when applying the BOIN to published data, the recommended MTD across TTRs was similar, but when using the BOIN at each dose assignment, with a TTR of 0.25, the BOIN(3) sometimes had slow dose escalation, which allowed less patients to be evaluated at higher doses. Therefore, a TTR of 0.30 is recommended for the BOIN design or using the option of having a 3 + 3 design run with a TTR of 0.25. Applying both the CRM and BOIN to published data was for illustration purposes only, demonstrating that even fitting a model at the end using data that were produced by the 3 + 3 design would have given a different dose selection more than a third of the time.

Although some investigators may assume that the 3 + 3 design targets right below 33%, it actually overlooks any dose with an observed toxicity rate higher than 1/6, which is lower than that previous simulations have found.⁵⁰ For example, in the trial by Younes et al,⁴⁸ the 3 + 3 design selected a dose with a DLT rate of 1/12 instead of selecting the dose with a DLT rate of 0.25 (3/12). Although the 3 + 3 design is favored for its simplicity, its rule-based nature does not allow for flexibility in choosing a range of acceptable DLT rates or in making common ad hoc decisions such as altering cohort sizes or adding an intermediate dose mid-trial.

Our study allows comparison of trials with a range of dose levels. For trials with three levels, the simplicity of the 3 + 3 design is beneficial and can offer similar efficiency. As the number of dose levels increased, model-based designs in general had smaller sample sizes and treated more patients around the MTD. The 3 + 3 design is inefficient as we found 21 trials (95%) had their first DLT at a dose level greater than or equal to the median dose level and 22 trials (100%) chose an MTD at or above this dose level. The BOIN(A), CRM(1), and CRM(3) all mitigate inefficiency through faster dose escalation, starting at median dose level or both.

Our approach for comparing the 3 + 3, CRM, and BOIN designs against each other is novel in its ability to compare them in the context of real trials. However, one limitation is that we do not know the true MTD or DLT rates like in a simulation study. Thus, we can only compare the recommended MTDs against each other and make judgements on the basis of the observed and estimated DLT rates. Moreover, we were not able to connect most trials to later phase trials or approved dosages since many did not lead to successful trials. High attrition rates pose a large problem in the success of oncology trials, which could be due in part to overly conservative MTDs that do not lead to efficacy in later phases.

In summary, our study using data from 22 dose-finding trials suggests that approximately 40% of the studies would have recommended one dose higher if the CRM or BOIN with a TTR of 25%-30% were used. It also confirms the advantages of the CRM and BOIN that have been demonstrated through simulation studies mentioned earlier. Finally, model-based designs while more complex also offer more flexibility such as inclusion of ad hoc decisions, late-onset toxicities, dose combinations, and efficacy information, which are crucial for novel anticancer treatments.

APPENDIX

FIG A1.

Selection percentage of MTD with a target toxicity rate of 0.30 compared with the original published data. Proportion is the number of times that design recommends dose as MTD of 1,000 retraced trials for each of the 22 trials. Refer to Table 1 for corresponding observed dose-limiting toxicity rates. Information within parentheses of column headers denotes cohort size or structure. BOIN, Bayesian optimal interval; CRM, continual reassessment method; MTD, maximum tolerated dose.

TABLE A1.

Mean Proportion of Patient Toxicity Data Generated by Each Method With a Target Toxicity Rate of 0.25

AUTHOR CONTRIBUTIONS

Conception and design: Christina Yap, Shing M. Lee

Collection and assembly of data: Rebecca B. Silva

Data analysis and interpretation: All authors

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by the authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO's conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/po/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).