Evaluation of irrational dose assignment definitions using the continual reassessment method

Abstract

Background:

This paper studies the notion of irrational dose assignment in phase I clinical trials. This property was recently defined by Zhou and colleagues as a dose assignment that fails to de-escalate the dose when 2 of 3, 3 of 6, or 4 of 6 patients have experienced a dose-limiting toxicity event at the current dose level. The authors claimed that a drawback of the well-known continual reassessment method is that it can result in irrational dose assignments. The aim of this article is to examine this definition of irrationality more closely within the conduct of the continual reassessment method.

Methods:

Over a broad range of assumed dose-limiting toxicity probability scenarios for six study dose levels and a variety of target dose-limiting toxicity rates, we simulated 2,000 trials of n=36 patients. For each scenario, we counted the number of irrational dose assignments that were made by the continual reassessment method, according to the definitions of Zhou and colleagues. For each of the irrational decisions made, we classified the dose assignment as an under-dose assignment, a target dose assignment, or an overdose assignment based on the true dose-limiting toxicity probability at that dose.

Results:

Across eight dose-toxicity scenarios, there were a total of 181,581 dose assignments made in the simulation study. Of these assignments, 8,165 (4.5%) decisions were made when 2 of 3, 3 of 6, or 4 of 6 patients had experienced a dose-limiting toxicity at the current dose. Of these 8,165 decisions, 1,505 (18.4%) recommended staying at the current dose level and would therefore be classified as irrational by Zhou and colleagues. Among the irrational decisions, 41.2% were misclassified, meaning they were made either at the true target dose (17.9%) or at a true under-dose (23.3%). The remaining 58.8% were made at a true over-dose and therefore truly irrational. Overall, irrational dose assignments comprised <1% of the total dose assignments made during the simulation study. Similar findings are reported in simulations across 100 randomly generated dose-toxicity scenarios from a recently proposed family of curves.

Conclusion:

Zhou and colleagues argue that the behavior of the continual reassessment method is disturbing due to its ability to make irrational dose assignments. These definitions are based on rules that mimic the popular 3+3 design, which should not be the benchmark used to construct guidelines for trial conduct of modern phase I methods. Our study illustrates that these dose assignments occur very seldom in the continual reassessment method, and that even when they do occur, they can often be considered sensible when accounting for all accumulated data in the study.

Introduction

Phase I trials have served as initial safety studies, with the main objective of identifying the maximum tolerated dose, defined as the highest dose level among a range of predefined dose levels that satisfies some safety requirement. The primary toxicity endpoint of interest is most often a binary one, defined in terms of the proportion of patients who experience a dose-limiting toxicity (yes/no), based on protocol-specific adverse event definitions. The underlying assumption driving the design of a Phase I trial is that both the risk of toxicity and the probability of clinical benefit increase with dose level; thus, the maximum tolerated dose represents the most promising dose for efficacy. To avoid exposing patients to excessive risks of toxicity, dose levels are assigned to patients sequentially, with a dose level administered to a patient only if there is evidence that dose levels less than the current dose have acceptable levels of toxicity. The statistical design revolves around two questions: (1) how should doses be assigned to patients as the trial proceeds? and (2) at the end of the trial, what dose should be recommended as the maximum tolerated dose? This paper focuses on question (1). Dose-finding methods have historically been broadly categorized as “rule based,” in which the dose chosen for the current patient is based on the observed number of dose-limiting toxicities in the most recently treated cohort of patients, or “model based,” in which an assumed parametric model is used to guide dose allocation and the estimation of the maximum tolerated dose.

Researchers working on designs for early-phase clinical trials have advocated for increased use of model-based approaches, such as the continual reassessment method,¹ to efficiently and accurately address the objectives of finding appropriate doses to merit further research.² The operating characteristics of the continual reassessment method have been extensively evaluated, and compared to the 3+3 design.³ Even with evidence that the continual reassessment method is the more accurate and efficient design,^3,4 and poses no safety concerns,⁴ traditional or modified forms of the 3+3 remain the most widely used approaches in dose-finding studies.^5–7 Efforts to move away from widespread implementation of the 3+3 have been made through the recent development of a new class of “model-assisted” designs. These methods were developed with the aim of effectively balancing the trade-off between simplicity and performance, and have been receiving increased attention with clinical audiences.⁸⁻¹⁰ A recent paper by Zhou, Yuan, and Nie¹¹ compares the 3+3 design to model-assisted dose-finding methods, such as the Bayesian optimal interval method,^9,12 the modified toxicity probability interval method,⁸ and the Keyboard method,¹⁰ and to model based designs, such as the continual reassessment method,¹ Bayesian logistic regression model,¹³ and escalation with overdose control.¹⁴ This paper was the latest in a series of papers that compares model-assisted methods to model-based methods.^15–16 The take-home message from the paper is that the Bayesian optimal interval method yields comparable performance with the continual reassessment method, but that the latter method can result in irrational dose assignments. This property was defined by Zhou et al.¹¹ as a dose assignment that fails to de-escalate the dose when 2 of 3, 3 of 6, or 4 of 6 patients have experienced a dose-limiting toxicity event at the current dose level.

The definitions of Zhou et al.¹¹ classify a dose assignment as irrational only by using the data at the current dose level, ignoring data accumulated at any other dose level. These definitions reflect rules that are constructed from a 3+3 mentality, and it is unclear whether or not they should be applied broadly to all dose-finding designs in evaluating in-trial behavior. For example, in an ideal situation in which we know in advance which dose is the maximum tolerated dose, all accrued patients would be treated at the optimal dose. If the target dose-limiting toxicity rate that defines the maximum tolerated dose of the study is θ = 0.25, then there is a 14.1% chance that the dose at which we are treating patients has observed dose-limiting toxicities in 2 of 3 patients and a 13.2% chance that the dose at which we are treating patients has observed dose-limiting toxicities in 3 of 6 patients. Therefore, in situations in which we know we are treating at the optimal dose, we would expect to incur irrational dose assignments with a certain probability. Yet, such in-trial behavior would be described by Zhou et al.¹¹ as “disturbing.” In light of the conclusions made by Zhou et al,¹¹ this paper more closely examines irrational dose assignments definitions when using the continual reassessment method.

Methods

In evaluating the definitions of irrational dose assignment proposed by Zhou et al,¹¹ we simulated the continual reassessment method over a broad range of assumed dose-toxicity probability scenarios studying six dose levels. The design specifications for the continual reassessment method are the same as those described in the Supplementary Material of Zhou et al.¹¹ We assumed a one-parameter power model for the probability of dose-limiting toxicity π_j at each dose level d_j so that

$${\pi }_j=\mathrm{Pr}(\text{dose}-\text{limiting}\hspace{0.17em}\text{toxicity}\hspace{0.17em}\text{at}\hspace{0.17em}\text{dose}\hspace{0.17em}{d}_j)=p_j^{\exp (\beta )}$$

where j = 1,…,J denotes the study dose level, β is a scalar parameter to be estimated from the data, and 0 < p₁ < ⋯ < p_J are pre-specified constants, often termed the “skeleton” of the model. All skeletons were generated using the algorithm of Lee and Chueng.¹⁷ For a target dose-limiting toxicity rate of θ = 0.25, the skeleton was (p₁,…,p₆) = (0.062, 0.140, 0.250, 0.376, 0.502, 0.615). For θ = 0.20, the skeleton was (p₁,…,p₆) = (0.032, 0.095, 0.200, 0.332, 0.470, 0.596). For θ = 0.30, the skeleton was (p₁,…,p₆) = (0.095, 0.186, 0.300, 0.422, 0.540, 0.643). For each value of θ, a normal prior distribution with a mean of zero and a variance of 2 was assumed; i.e., β~N(0,2). Each simulated trial began by treating the first patient cohort at the lowest study dose level. The continual reassessment method allocates patients sequentially, with each patient cohort assigned to the dose level with the Bayesian model-based estimated probability of dose-limiting toxicity ${\widehat{\pi }}_j$ closest to the target dose-limiting toxicity rate θ. Dose-skipping was prohibited so that escalation and de-escalation movements were restricted to one dose level at a time. The following safety stopping rule was imposed: if the posterior probability that the risk of dose-limiting toxicity at the lowest study dose level exceeds θ is greater than 0.95, then the trial is terminated for safety; i.e. if Pr(π_j > θ|data) > 0.95.

We examined both individual dose assignments within single simulated trials, as well as irrational dose assignment operating characteristics over many simulated trials. For target dose-limiting toxicity rates of θ = 0.20, 0.25, and 0.30, we simulated 2,000 trials of n=36 patients. Dose assignments were made in cohorts of size 3 throughout the trial. As an illustration, we first studied the in-trial behavior of the continual reassessment method in a simulated trial example. We generated dose-limiting toxicity outcomes under an assumed set of true probabilities (0.09, 0.16, 0.23, 0.34, 0.51, 0.74; Scenario 3 in Supplemental Appendix C of Zhou et al.¹¹) for a trial of n=36 patients in cohorts of size 3. The target dose-limiting toxicity rate was θ = 0.25, indicating dose level 3 to be the true maximum tolerated dose. We counted the number of dose assignments that were made when 2 of 3, 3 of 6, or 4 of 6 patients had experienced a dose-limiting toxicity at the current dose. This value is the number of possible irrational dose assignments that are possible in any one simulated trial. We counted the number of times that the continual reassessment method recommended to stay at the current dose level in these situations, resulting in an irrational dose assignment. For each of the irrational decisions made, we classified the dose assignment as an under-dose assignment, a target dose assignment, or an overdose assignment based on the true dose-limiting toxicity at that dose. The true target dose is the dose with dose limiting toxicity probability closest to the target rate. A true under-dose is a dose with dose limiting toxicity probability below the target rate. A true over-dose is a dose with dose limiting toxicity probability above the target rate. If the irrational dose assignment was made at either a true under-dose or the true target dose, then the assignment was misclassified as irrational. If the irrational dose assignment was made at a true over-dose, then the assignment was considered truly irrational.

We began our study of operating characteristics over many simulated trials by examining irrational dose assignments made by the continual reassessment method over eight dose-toxicity scenarios given in Supplementary Appendix C for Zhou et al¹¹ (Table 1). For each scenario, the target dose-limiting toxicity rate was θ = 0.25, and the scenarios vary the location of the true maximum tolerated dose. In each simulated trial, we counted the possible number of irrational dose assignments that could be made, the number of irrational dose assignments that were actually made, and designated each irrational dose assignment as truly irrational or misclassified. We summarized these counts and calculated percentages over the 2,000 simulated trials for each scenario. For further study, we repeated the evaluation described above for six dose levels and target dose-limiting toxicity rates θ = 0.20, 0.25, and 0.30 over 100 randomly generated dose-limiting toxicity probability scenarios from the Conaway and Petroni¹⁸ class of dose-toxicity curves. The set of curves for a target rate of 25% are provided in Supplemental Material (Figure S1). The model that generates the dose-limiting toxicity responses is based on an S-shaped curve

$$f(x)=\beta +\frac{\alpha -\mu }{1+e^{-k(\text{logit}(x))}}$$

where κ > 0, α > µ, and x takes on values from 0.001 to 0.999 in increments of 0.001. In this model, µ and α represent minimum and maximum dose-limiting toxicity rates, respectively. Based on the number of dose levels J, the algorithm¹⁸ for curve generation randomly chooses J sorted values of x from the 999 possible values that ranged from 0.001 to 0.999. In order to establish adequate spacing between adjacent values, the random selection process selects one x value from each of the J intervals [(i − 1) × k, i × k], where i = 1,…, J and k = 999/J. For each simulated trial, the maximum sample size was n=36 patients, and dose assignments were made in cohorts of either size 1 or 3 throughout the trial. For each curve, we tabulated the number of irrational dose assignments made by the continual reassessment method in each simulated trial and summarized the counts and percentages over all simulated trials and dose-toxicity curves. We repeated this process for studies of four dose levels and eight dose levels.

Table 1.

Eight assumed dose-toxicity probability scenarios for a target dose-limiting toxicity rate of 0.25. Boldface denotes the true maximum tolerated dose.

Scenario	Study dose level
	1	2	3	4	5	6
	Target dose-limiting toxicity rate = 0.25
1	0.26	0.34	0.47	0.64	0.66	0.77
2	0.18	0.25	0.32	0.36	0.60	0.69
3	0.09	0.16	0.23	0.34	0.51	0.74
4	0.07	0.12	0.17	0.27	0.34	0.51
5	0.03	0.13	0.17	0.19	0.26	0.31
6	0.04	0.05	0.09	0.14	0.15	0.24
7	0.34	0.42	0.46	0.49	0.58	0.62
8	0.13	0.41	0.45	0.58	0.75	0.76

Results

The dose assignments made by the continual reassessment method in the simulated trial of n=36 patients are reported in Figure 1. The study begins on the lowest study dose level 1 and 0 of 3 patients experience a dose-limiting toxicity. At this point, the updated model indicates that dose level 3 has the dose-limiting toxicity probability estimate ${\widehat{\pi }}_j=0.223$ closest to the target rate of θ = 0.20. Pre-specified restrictions do not allow the trial to skip over an untried dose in escalation so dose level 2 is recommended for the next cohort. In this cohort, 2 of 3 patients experience a dose-limiting toxicity. At this point in the study, we observe our first possible situation in which an irrational dose assignment could be made since we have observed dose-limiting toxicities in 2 of 3 patients at the current dose. However, the continual reassessment method recommends de-escalating the dose and returning to dose level 1. At the lowest dose level, no dose-limiting toxicities are observed in each of the next two cohorts so the method recommends returning to dose level 2 with 0 of 9 patients having dose-limiting toxicity on dose level 1. Patients 13, 14, and 15 are administered dose level 2 and 1 dose-limiting toxicity is observed, bringing the observed total to 3 of 6 on dose level 2. This is the second possible decision point in the study in which an irrational dose assignment could be made, and the continual reassessment method yields model-based dose-limiting toxicity probability estimates of $({\widehat{\pi }}_1,\dots ,{\widehat{\pi }}_6)=(0.17,\hspace{0.17em}\mathbf{0.26},\hspace{0.17em}0.36,\hspace{0.17em}0.46,\hspace{0.17em}0.56,\hspace{0.17em}0.64)$. These estimates recommend that the trial stay at the current dose level, resulting in an irrational dose assignment. However, closer examination of this decision demonstrates that this assignment should not be considered irrational: (1) dose level 1 has been established to be quite safe (i.e., 0 of 9 dose-limiting toxicities), which is reflected in the estimated probabilities at each dose, (2) the posterior probability that the risk of dose-limiting toxicity at dose level 2 exceeds θ is 0.539, which is not sufficiently high; i.e. if Pr(π₂ > θ|data) = 0.539, and (3) the true probability of dose-limiting toxicity at dose level 2 is 0.16, indicating that dose level 2 is actually below the target dose-limiting toxicity rate. The definitions of Zhou et al.¹¹ would argue that we should de-escalate and further establish the safety of dose level 1, before returning to dose level 2, rather than simply staying at dose level 2. But the safety of dose level 1 has been established at this point in the trial, so it should not be required that the design try it again before deeming dose level 2 worthy of further study. Labeling this dose assignment as irrational is a myopic view of what is happening at other dose levels being studied. Ultimately, after 4 of 18 patients had dose-limiting toxicities on dose level 2, the continual reassessment method recommends escalating to dose level 3 for the final nine patients accrued to the study. At dose level 3, 1 of 9 patients experienced dose-limiting toxicities, and dose level 3 (π₃ = 0.23) was recommended as the maximum tolerated dose at the conclusion of the study, based on the final dose-limiting toxicity probability estimates of $({\widehat{\pi }}_1,\dots ,{\widehat{\pi }}_6)=(0.08,\hspace{0.17em}0148,\hspace{0.17em}\mathbf{0}\mathbf{.}\mathbf{235},\hspace{0.17em}0.336,\hspace{0.17em}0.440,\hspace{0.17em}0.540)$.

Figure 1.

Simulated trial example of the continual reassessment method for n = 36 patients under Scenario 3 in Table S2 of Zhou et al.¹¹ Red circles denote irrational dose assignments according to the definition of Zhou et al.¹¹ DLT = dose-limiting toxicity; MTD = maximum tolerated dose.

Over all eight scenarios in Table 1, there were a total of 181,581 dose assignments made in the simulation study. Of these assignments, 4,334 (2.39%) decisions were made when 2 out of 3 patients had experienced a dose-limiting toxicity at the current dose. Of these 4,334 decisions, a total of 586 (13.5%) recommended staying at the current dose level when 2 out of 3 patients had experienced a dose-limiting toxicity at the current dose. Of the 181,581 total assignments, 2965 (1.63%) decisions were made when 3 out of 6 patients had experienced a dose-limiting toxicity at the current dose. Of these 2965 decisions, a total of 863 (29.1%) recommended staying at the current dose level when 3 out of 6 patients had experienced a dose-limiting toxicity at the current dose. Of the 181,581 total assignments, 866 (0.48%) decisions were made when 4 out of 6 patients had experienced a dose-limiting toxicity at the current dose. Of these 866 decisions, a total of 56 (6.47%) recommended staying at the current dose level when 4 out of 6 patients had experienced a dose-limiting toxicity at the current dose. With an observed dose-limiting toxicity rate of 2 of 3, 32.2% of irrational decisions were either made at the true target dose or at a true under-dose, meaning they were misclassified as irrational. The remaining 67.8% were made at a true over-dose and were therefore truly irrational. With an observed dose-limiting toxicity rate of 3 of 6, 51.8% of irrational decisions were misclassified as irrational, while the remaining 48.2% were truly irrational. With an observed dose-limiting toxicity rate of 4 of 6, 49.7% of irrational decisions were misclassified, while 50.3% were truly irrational. Overall, irrational dose assignments comprised 0.829% (i.e., 1,505 of 181,581) of the total dose assignments made during the simulation study. Of the 1,505 irrational dose assignments, 619 (41.1%) occurred at a true target dose or a true under-dose (misclassified), and 886 (58.9%) were made at a true over-dose (truly irrational).

Table 2 reports the irrational dose assignment operating characteristics of the continual reassessment method in our simulation study across the eight dose-toxicity scenarios in Table 1. In each scenario, there were a total of 24,000 possible dose assignments (i.e, 12 cohorts × 2,000 simulated trials), but some trials stopped early for safety, reducing the actual number of observed dose assignments in some scenarios. In Scenarios 1 and 2, the true maximum tolerated doses are at low dose levels so it is not unexpected that all irrational dose assignments are made when the current dose level has a true dose-limiting toxicity probability above the target rate (i.e. over-doses). However, when the location of the true maximum tolerated dose is in the middle of the range of dose levels (i.e. Scenarios 3 and 4), we begin to observe dose assignments that are classified as irrational actually occurring at dose levels that are true maximum tolerated doses (i.e. target doses) or below the maximum tolerated dose (i.e., under-doses). In Scenario 3, when 3 of 6 patients had dose-limiting toxicity on the current dose and an irrational assignment was made, 83.7% of these decisions were not made at current doses with true dose-limiting toxicity probabilities above the target rate and were misclassified as irrational. In Scenario 3, when 4 of 6 patients had dose-limiting toxicity on the current dose and an irrational assignment was made, 60.0% of these decisions were misclassified. Similar findings can be observed in Scenarios 4 and 5. In Scenario 4, when 3 of 6 and 4 of 6 patients had dose-limiting toxicity on the current dose and an irrational assignment was made, 85% and 100%, respectively, of these decisions were misclassified. Finally, in Scenario 4, when 2 of 3 patients had a dose-limiting toxicity, 57.58% of irrational decisions were misclassified. In Scenario 5, when 3 of 6 and 4 of 6 patients had dose-limiting toxicity on the current dose and an irrational assignment was made, 88.8% and 100%, respectively, of these decisions were misclassified. When 2 of 3 patients had a dose-limiting toxicity in Scenario 5, 40.4% of these decisions were to retain the true maximum tolerated dose and 59.6% of these decisions were to retain a true under-dose. Even though Scenario 5 contains a dose with true dose-limiting probability above the target rate (i.e., dose level 6 with π₆ = 0.31), 100% of irrational dose assignments were misclassified.

Table 2.

A breakdown of the number of irrational dose assignments made by the continual reassessment method over eight assumed dose-toxicity curves. A misclassification is an irrational dose assignment that is made on a true under-dose or the true target dose. A truly irrational dose assignment is made on a true over-dose. DLT = dose-limiting toxicity.

	Scenario (total number of dose assignments)
	1 (21117)			2 (23449)
Observed DLT rate at current dose	2 of 3	3 of 6	4 of 6	2 of 3	3 of 6	4 of 6
Assignments made on this data	482	303	99	555	391	112
Assignments to retain current dose	15	75	3	46	122	8
% Misclassified	0%	0%	0%	0%	0%	0%
% Truly irrational	100%	100%	100%	100%	100%	100%
	3 (23968)			4 (23990)
Observed DLT rate at current dose	2 of 3	3 of 6	4 of 6	2 of 3	3 of 6	4 of 6
Assignments made on this data	647	463	121	647	445	108
Assignments to retain current dose	126	122	10	165	113	3
% Misclassified	0%	83.7%	60%	57.6%	85%	100%
% Truly irrational	100%	16.4%	40%	42.4%	15%	0%
	5 (24000)			6 (24000)
Observed DLT rate at current dose	2 of 3	3 of 6	4 of 6	2 of 3	3 of 6	4 of 6
Assignments made on this data	528	368	63	373	225	34
Assignments to retain current dose	109	107	4	97	72	3
% Misclassified	100%	88.8%	100%	100%	100%	100%
% Truly irrational	0%	11.2%	0%	0%	0%	0%
	7 (17230)			8 (23827)
Observed DLT rate at current dose	2 of 3	3 of 6	4 of 6	2 of 3	3 of 6	4 of 6
Assignments made on this data	347	192	59	755	578	270
Assignments to retain current dose	4	30	1	24	222	24
% Misclassified	0%	0%	0%	0%	0%	0%
% Truly irrational	100%	100%	100%	100%	100%	100%

Figure 2 reports the irrational dose assignment operating characteristics for the continual reassessment method over 100 randomly generated dose-toxicity curves for three different target toxicity rates. These results are consistent with those reported in the Supplemental Material of Zhou et al,¹¹ which were computed on 1,000 randomly generated dose-toxicity relations from the Clertant and O’Quigley¹⁹ family of curves. In general, the overall risk of irrational dose assignment increases as the target rate increases, and this overall risk was lower for cohorts of size 3 when compared to cohorts of size 1. Although the overall risk of irrational dose assignment was smaller using cohorts of size 3, other metrics, such as the percent of correct selection of the maximum tolerated dose and the expected number of patients treated at the maximum tolerated dose, should also be considered when considering cohort size. Additional simulation results in the Supplemental Material (Figure S2) demonstrate that cohorts of size 1 increase the probability of correctly identifying the maximum tolerated dose and increase the expected number of patients treated at this dose when compared with cohorts of size 3 for each target dose-limiting toxicity rate. As is to be expected, the lowest risk of irrational assignment occurs when 2 of 3 patients have experienced a dose-limiting toxicity, the target rate is θ = 0.20, and cohort size 3 (bottom left panel in Figure 2). Figure 2 results are consistent with those described in the previous eight scenarios in that a large percentage of dose assignments labeled as irrational were misclassified. Similar finding are reported in the Supplemental Material for four dose levels and eight dose levels (Figures S3 and S4). Overall, these results shed new light on the definition of irrational dose assignment by Zhou et al.¹¹

Figure 2.

A breakdown of the number of irrational dose assignments made by the continual reassessment method over 100 randomly generated dose-toxicity curves. A misclassification is an irrational dose assignment that is made on a true under-dose or the true target dose. A truly irrational dose assignment is made on a true over-dose.

Conclusions

In this paper, we have studied recently proposed irrational dose assignment definitions within the context of the continual reassessment method. Overall, irrational dose assignments comprised <1% of the total dose assignments made during the simulation study. Furthermore, even when irrational dose assignments are made, there is a significant probability (approximately 42%) that the decision is being made to accrue the next cohort to a dose with a true dose-limiting toxicity probability below or close to the target toxicity rate that defines the maximum tolerated dose of the study. We also briefly studied the impact of prior distribution choice on irrational dose assignment operating characteristics. For the original eight scenarios, we reran simulations using the same skeleton as before and a β~N(0,1.35), which was generated using the algorithm of Lee and Cheung²⁰ that yields the least informative normal prior in terms of which dose level is the maximum tolerated dose. This prior specification eliminated irrational dose assignments for situations in which 2 of 3 dose-limiting toxicities were observed on the current dose, and reduced the number of irrational dose assignments when ≥3 of 6 dose-limiting toxicities were observed. For the sake of brevity, we have not included these results here, but we are happy to share them with any interested reader. They illustrate that a well-calibrated prior distribution on the model parameter in the continual reassessment method can reduce the already minimal chance that the design will result in an irrational dose assignment. The operating characteristics of the continual reassessment method have been vetted for nearly thirty years and it has been consistently demonstrated as the most accurate and efficient dose-finding method, with operating characteristics near that of an optimal benchmark.²¹ Additionally, it has proved to possess desirable in-trial behavior, such as coherent dose assignment.²² As mentioned above, situations in which the behavior of the continual reassessment method has been shown to be problematic mainly occur as the result of a poorly chosen model and/or prior distribution.¹³ There has been extensive work done on appropriate design calibration^17,20 for the method, which mitigates these issues, or eliminates them altogether. The availability of easy-to-use web applications²³ that incorporate these recommended specifications should further facilitate its use in practice.

The definitions of irrational dose assignment studied in this paper were proposed within a framework that relies on local decision-making, based only on the observed data at the current dose level. This local decision making can result in questionable dose assignments for model-assisted methods, such as those described by Wages and Braun,²⁴ even though these assignments are considered rational according to the Zhou et al¹¹ criteria. General definitions for irrational dose assignment that would be applicable across the board in dose-finding trials are difficult to propose. Such definitions are not necessary, and perhaps counterproductive. Common metrics to evaluate the operating characteristics of dose-finding designs, such as the percentage of maximum tolerated dose selection at and around the target dose, the expected number of patients treated at and around the target dose, the percentage of trials in which an overdose was recommended as the maximum tolerated dose, the expected number of patients treated at doses above the target dose, and the expected number of dose-limiting toxicities at each dose level have been used for many years and they should continue to be the criterion for assessing performance. The introduction of new metrics based on a 3+3 mentality can cause confusion in the clinical trial community as to which designs are better, which could ultimately lead investigators to fall back to the easily understood 3+3. We hope this current study will encourage investigators to more closely examine the in-trial behavior of dose-finding methods, rather than defaulting to guidelines that mimic the antiquated 3+3 design.