Coherence principles in interval-based dose-finding

Abstract

This paper studies the notion of coherence in interval-based dose-finding methods. An incoherent decision is either: (i) a recommendation to escalate the dose following an observed dose-limiting toxicity, or (ii) a recommendation to de-escalate the dose following a non-dose-limiting toxicity. In a simulated example, we illustrate that the Bayesian optimal interval method and the Keyboard method are not coherent. We generated dose-limiting toxicity outcomes under an assumed set of true probabilities for a trial of n = 36 patients in cohorts of size 1, and we counted the number of incoherent dosing decisions that were made throughout this simulated trial. Each of the methods studied resulted in 13/36 (36%) incoherent decisions in the simulated trial. Additionally, for two different target dose-limiting toxicity rates, 20% and 30%, and a sample size of n = 30 patients, we randomly generated 100 dose-toxicity curves and tabulated the number of incoherent decisions made by each method in 1000 simulated trials under each curve. For each method studied, the probability of incurring at least one incoherent decision during the conduct of a single trial is greater than 75%. Coherency is an important principle in the conduct of dose-finding trials. Interval-based methods violate this principle for cohorts of size 1 and require additional modifications to overcome this shortcoming. Researchers need take a closer look at the dose assignment behavior of interval-based methods when using them to plan dose-finding studies.

1 |. INTRODUCTION

This paper studies the principle of coherence in interval-based dose-finding methods. Consider a dose-finding trial studying J discrete dose levels d₁, … , d_J, and suppose patient i accrued to the study receives dose x_i ∈ {d₁, … , d_J }. Let Y_i denote the binary indicator of dose-limiting toxicity for the ith patient, which is typically the primary study endpoint. The primary objective of the study is to identify the maximum tolerated dose, defined as the dose d_j with a probability of dose-limiting toxicity π_j closest to a pre-specified target dose-limiting toxicity rate θ. Dose-finding designs are outcome-adaptive in that they sequentially assign each accruing patient cohort to one of the study dose levels based on the results of the most current cohort, or possibly all previous cohorts. At the time of each dose assignment, one of three possible decisions is made for the next cohort: (1) de-escalate from the current dose; i.e., x_i < x_i−1, (2) stay at the current dose; i.e., x_i = x_i−1, or (3) escalate from the current dose; i.e., x_i > x_i−1. Cheung¹ defines an escalation decision for patient i to be coherent only when Y_i−1 = 0. That is, a dose-finding design is said to be coherent in escalation if

$$\text{Pr}\left(x_i>x_{i-1}|Y_{i-1}=1\right)=0$$

for all i. Similarly, a design is said to be coherent in de-escalation if

$$\text{Pr}\left(x_i<x_{i-1}|Y_{i-1}=0\right)=0$$

for all i. Incoherent decisions are defined as either escalations immediately after a dose-limiting toxicity outcome or de-escalations after a non-dose-limiting toxicity outcome. This property was formally defined by Cheung¹, although it had been previously discussed as a desirable property of the continual reassessment method by O’Quigley, Pepe and Fisher² and O’Quigley and Shen³.

Until recently this single clear definition for a design’s coherency has been in use. Liu and Yuan⁴, in their introduction to the Bayesian optimal interval method, refer to Cheung’s definition as short-term memory coherence and distinguish it from a new property, which the authors define as long-term memory coherence. The authors acknowledge that their method is not short-term memory coherent, as a result of the fact that interval-based methods rely on decision-making based on the observed data at the current dose level only. Consider a trial in progress in which the current dose level is d_j, and suppose t_j dose-limiting toxicities have been observed in n_j patients treated so far at d_j. Let D and ε represent the actions of de-escalating the dose and escalating the dose, respectively. Based on the accumulated data Ω_j = (t_j, n_j) and the estimated dose-limiting toxicity rate ${\widehat{p}}_j=t_j/n_j$ at d_j, a dose assignment is said to be long-term memory coherent in escalation if

$$\text{Pr}\left(\epsilon |{\widehat{p}}_j>\theta ,{\Omega }_j\right)=0.$$

Similarly, a dose assignment is said to be long-term memory coherent in de-escalation if

$$\text{Pr}\left(D|{\widehat{p}}_j<\theta ,{\Omega }_j\right)=0.$$

A long-term memory coherent design does not induce any long-term memory incoherent dose assignments. This latter definition is a necessary construction for interval-based methods^4–7 due to the fact that these methods do not meet the commonly accepted definition of coherence highlighted by Cheung¹. Coherency demonstrates an important principle in which dose escalation for the next patient is appropriate only when the dose administered to the most recent patient is deemed safe (i.e. > 0 dose-limiting toxicities result in staying at the same dose or de-escalating). Conversely, it protects against the counter-intuitive allocation decision of reducing the dose for the next patient when the dose administered to the most recent patient was well-tolerated (i.e.0 dose-limiting toxicities results in staying at the same dose or escalating). Interval-based methods have received considerable attention recently, with investigators arguing that they provide a simpler alternative to the continual reassessment method^5–8. The aim of this article is to examine dose assignment behavior of interval-based dose-finding methods in order to illustrate the impact of a design failing to be coherent. Throughout the manuscript, the use of the term ‘coherent’ refers to the original definition.

2 |. METHODS

2.1 |. Interval-based dose-finding

We studied the trial conduct of the Bayesian optimal interval and the Keyboard⁵ methods. The Keyboard method was developed as an improvement to the modified toxicity probability interval method⁶ and is equivalent to the modified toxicity probability interval-2 method⁹. Therefore, we chose not to separately examine the modified toxicity probability interval method since we are studying an equivalent method to the modified toxicity probability interval-2 method. The details of the Bayesian optimal interval and the Keyboard methods are provided elsewhere, so we only briefly recall them here.

The Bayesian optimal interval method⁴ begins by deriving pre-specified dose escalation and de-escalation boundaries, λ_e and λ_d, respectively. The boundaries are derived by specifying the largest dose-limiting toxicity probability θ₁ that is considered to be underdosing (i.e., the design should escalate), and the smallest dose-limiting toxicity probability θ₂ that is considered to be overdosing (i.e., the design should de-escalate). Based on these values, the escalation and de-escalation boundaries are

$${\lambda }_e=\frac{\text{log }\left(\frac{1-{\theta }_1}{1-\theta }\right)}{\text{log }\left(\frac{\theta \left(1-{\theta }_1\right)}{{\theta }_1(1-\theta )}\right)},{\lambda }_d=\frac{\text{log }\left(\frac{1-\theta }{1-{\theta }_2}\right)}{\text{log }\left(\frac{{\theta }_2(1-\theta )}{\theta \left(1-{\theta }_2\right)}\right)}.$$ (1)

Detailed derivations of λ_e and λ_d are given in Liu and Yuan⁴. Dosing decisions are made by comparing the observed dose-limiting toxicity rate ${\widehat{p}}_j=t_j/n_j$ at the current dose level d_j with the decision boundaries. If ${\widehat{p}}_j\le {\lambda }_e$, escalate the dose level for the next cohort. If ${\widehat{p}}_j\ge {\lambda }_d$, de-escalate the dose level for the next cohort. Otherwise, retain the current dose level.

The Keyboard method assumes a standard beta-binomial model

$$t_j|{\pi }_j~\text{ Binomial}\left({\pi }_j\right)$$

$${\pi }_j~\text{Beta}(\alpha ,\beta ),$$

where Beta(α, β) is a beta distribution with parameters α and β. Based on the accumulated data Ω_j = (t_j, n_j) at the current dose level d_j, the posterior distribution of π_j follows a beta distribution

$${\pi }_j|{\Omega }_j~\text{Beta}\left(\alpha +t_j,\beta +n_j-t_j\right).$$

The dose-finding algorithm partitions the interval (0, 1) into a set of equal-width dosing intervals (keys). The method begins by specifying a dosing interval C_θ, termed the target key, around the target dose-limiting toxicity rate θ. This interval is formed by specifying ϵ₁ and ϵ₂ that define a target interval around θ so that C_θ = (θ − ϵ₁, θ + ϵ₂). Both below and above the target key, the remainder of the (0, 1) interval is divided into a set of intervals of equal length to C_θ. For each interval (c₁, c₂), the Keyboard method calculates the posterior probability

$$\begin{gathered} \text{Pr}\left({\pi }_j\in \left(c_1,c_2\right)|{\Omega }_j\right)=\mathcal{B}\left(c_2;\alpha +t_j,\beta +n_j-t_j\right) \\ -\mathcal{B}\left(c_1;\alpha +t_j,\beta +n_j-t_j\right), \end{gathered}$$

where $\mathcal{B}\left(c;\alpha +t_j,\beta +n_j-t_j\right)$ is the cumulative density function of the beta distribution with parameters α + t_j and β + n_j − t_j, evaluated at c. Dose assignments are based on which key has the highest posterior probability of dose-limiting toxicity, which is termed the strongest key. If the strongest key is below the target key, the dose level for the next cohort is escalated from the current level. If the strongest key is above the target key, the dose level for the next cohort is de-escalated from the current level. If the strongest key is the target key, the current dose level is retained. For both methods, after accrual of the maximum sample size, isotonic regression¹⁰ is applied to the observed dose-limiting toxicity rates ${\widehat{p}}_j$ at each dose level and the dose with an estimated dose-limiting toxicity rate closest to the target dose-limiting toxicity rate θ is selected as the maximum tolerated dose. As an illustration of incoherency, we examined both individual dose assignment decisions of the Bayesian optimal interval method and the Keyboard method within a single simulated trial, as well coherency operating characteristics of these methods over many simulated trials. User-friendly R code is available at http://faculty.virginia.edu/model-based_dose-finding/ for (1) randomly generating dose-toxicity curves according to the algorithm of Clertant and O’Quigley¹¹, (2) simulating single trials using each method and observing the number of incoherent dose assignments, and (3) generating coherency operating characteristics for each method. The .csv files that contain the decision tables for both methods, as well as the randomly generated dose-toxicity curves, are also available at this site.

2.2 |. Single trial illustration

We generated dose-limiting toxicity outcomes under an assumed set of true probabilities (0.09, 0.16, 0.23, 0.34, 0.51, 0.74; Scenario 3 in Supplemental Appendix C of Zhou et al.⁸) for a trial of n = 36 patients in cohorts of size 1. In simulating the data, the tolerance of each patient can be considered a uniformly distributed random variable on the interval [0, 1], which we term a patient’s latent toxicity tolerance and denote u_i for the ith entered patient¹². At the dose (d_j) assigned to patient i, if the tolerance is less than or equal to its true dose-limiting toxicity probability (i.e. u_i ≤ π_j), then patient i has a dose-limiting toxicity; otherwise the patient has a non-dose-limiting toxicity outcome. Of course, in a real trial, it is impossible to observe a patient’s latent tolerance, but it is a useful tool in simulation and can be used to compare the operating characteristics of different designs within a single trial. Based on the same latent tolerance sequence, the allocation algorithms of the Bayesian optimal interval method and the Keyboard method can be evaluated using the same patients. The target dose-limiting toxicity rate was θ = 25%, indicating dose level 3 to be the true maximum tolerated dose. For the Bayesian optimal interval method, Liu and Yuan⁴ recommend θ₁ = 0.6 θ and θ₂ = 1.4 θ. Based on these values, the dose escalation and de-escalation boundaries are calculated to be 19.7% and 29.8%, respectively, using Equation (1). For the Keyboard method, ϵ₁ = ϵ₂ = 0.05 are recommended⁵, so that the target key is (0.25−0.05, 0.25+0.05) = (0.2, 0.3). We counted the number of incoherent dosing decisions, defined by Cheung¹, that were made throughout the conduct of this simulated trial.

2.3 |. Operating characteristics

In order to demonstrate that incoherency is not an isolated case present in the simulated example described above, we examined coherency behavior of the two methods over many simulated trials under different dose-toxicity scenarios. We studied two different target dose-limiting toxicity rates, θ = 20% and θ = 30%, for trials investigating six study dose levels. For target toxicity rate 20%, the Bayesian optimal interval method uses dose escalation and de-escalation boundaries of 15.7% and 23.8%, respectively, using the default values for θ₁ and θ₂ and Equation (1). For the Keyboard method, the target key is (0.15, 0.25) for target rate 20%. For target toxicity rate 30%, the Bayesian optimal interval method uses dose escalation and de-escalation boundaries of 23.6% and 35.8%, respectively. For the Keyboard method, the target key is (0.25, 0.35) for target rate 30%. For each target toxicity rate, we simulated 1000 trials for each of 100 assumed dose-toxicity scenarios, randomly generated from the Clertant and O’Quigley¹¹ class of dose-toxicity curves (Figure 1). The following algorithm generates the curves:

1. Choose the maximum tolerated dose, denoted j, at random from one of the J study dose levels.

2. Sample $\mathit{M}~\mathcal{B}(\text{max}(\mathit{J}-j),0.5,1)$ and set an upper bound B = θ+(1−θ)×M for the dose-limiting toxicity probabilities.

3. Repeatedly sample J dose-limiting toxicity probabilities uniformly on [0, B] until these correspond to a scenario in which j is the maximum tolerated dose.

For each simulated trial, the maximum sample size was n = 30 patients, and allocation decisions were made in cohorts of size 1 throughout the trial. For each curve, we tabulated the number of incoherent decisions made by each method in each simulated trial and summarized the percentage of trials that incurred at least each observed number of incoherent decisions.

FIGURE 1.

100 randomly generated dose-toxicity curves from the Clertant and O’Quigley¹¹ class for target toxicity rates of 20% and 30%.

3 |. RESULTS

3.1 |. Single trial illustration

The allocation decisions of the Bayesian optimal interval method and the Keyboard method for the simulated trial are identical and are reported in Figure 2. Of particular interest are decisions such as the escalation from dose level 2 to dose level 3 after the dose-limiting toxicity outcome observed on patient 14. Both the Bayesian optimal interval method and Keyboard methods recommend returning to dose level 3 immediately after a dose-limiting toxicity outcome is observed (patient 14) at dose level 2, illustrating incoherence. For Bayesian optimal interval, the observed dose-limiting toxicity rate at dose level 2 is ${\widehat{p}}_2=1/7=14.3\%$, which falls below the escalation boundary λ_e = 19.7%. For the Keyboard method, the intervals (c₁, c₂) that partition (0, 1) are {(0.00, 0.10), (0.10, 0.20), (0.20, 0.30), (0.30, 0.40), (0.40, 0.50), (0.50, 0.60), (0.60, 0.70), (0.70, 0.80), (0.80, 0.90), (0.90, 1.00)}. For each of these intervals, the posterior probabilities in Equation (2) are calculated based on the data Ω₂ = (t₂ = 1, n₂ = 7) and a uniform prior α = β = 1. The posterior density for p₂ after patient 14 is given in Figure 3 (a), illustrating that the strongest key (i.e., maximum posterior density) for dose level 2 falls below the target key and that the dose should be escalated. However, the posterior density of p₃ in Figure 3 (b) illustrates that the strongest key for dose level 3 is above the target key based on the current data at dose level 3. This information is ignored due to the lack of borrowing across dose levels, and patient 15 is administered dose level 3 despite information indicating that it is too toxic to administer.

FIGURE 2.

Simulated trial example of the Bayesian optimal interval method and the Keyboard method for n = 36 patients under Scenario 3 in Table S2 of Zhou et al⁸. Red circles denote incoherent dose assignments. DLT = dose-limiting toxicity; MTD = maximum tolerated dose.

FIGURE 3.

The posterior distribution of p₂ and p₃ at the time patient 15 is accrued to the simulated trial (Figure 2). DLT = dose-limiting toxicity.

Long-term memory coherence would argue that it is the right decision to return to dose level 3 after this dose-limiting toxicity since the overall observed rate at dose level 2 is 1/7 = 14.3%. This argument, however, ignores the sequential nature of the dose-finding problem. That is to say, it matters what data has been observed at other levels in order for the design to arrive at dose level 2 at this point. The assignment of patient 14 to dose level 2 was a de-escalation from dose level 3 (patient 13). The current estimate of the best dose to give patient 14 is level 2, but, after observing a dose-limiting toxicity on patient 14, dose level 2 is considered “too safe” and the dose must be escalated to level 3. Similar counter-intuitive assignments can be observed after non-dose-limiting toxicities. Patient 17 receives dose level 3, and after a non-dose-limiting toxicity, it is recommended that the trial de-escalate to level 2. Again, long-term memory coherence would myopically focus on the observed 3/8 = 37.5% rate at level 3 to justify this decision. This decision is implying that, after patient 16, it was a safe to try dose level 3. But after observing a non-dose-limiting toxicity outcome, it is no longer safe to be at dose level 3, and that the trial must de-escalate to dose level 2. Overall, data observed for 13 of 36 (36%) patients accrued to the study (patients 7, 9, 11, 13, 14, 17, 19, 21, 25, 26, 29, 31, and 33) resulted in an incoherent dose assignment for the following patient accrued to the study for both methods. Incoherent dose assignments are indicated in red in Figure 2. While this simulated trial constitutes only a single example of a behavior of these methods that would not be clinically acceptable, we hope that it will encourage researchers to take a closer look at in-trial behavior of these interval-based Phase I methods.

3.2 |. Operating characteristics

Figure 4 reports the number of incoherent decisions observed and the percentage of simulated trials with at least each observed number of incoherent decisions for both methods for the two target toxicity rates studied. For the Bayesian optimal interval method and target rate of 20%, 79.2% of simulated trials resulted in at least one incoherent decision. The probability of observing more than one incoherent decision was 58.9%, and the probability of observing at least five incoherent decisions was 27.9%. The maximum number of incoherent decisions in any one simulated trial was 13 decisions, but this occurred in very few trials. Incoherent de-escalation decisions tended to occur much more frequently than incoherent escalation decisions. The maximum number of incoherent de-escalation decisions in any one simulated trial was 11 decisions, and the maximum number of incoherent escalation decisions made in any one simulated trial was 2 decisions. For the Bayesian optimal interval method and target rate of 30%, 77.0% of simulated trials resulted in at least one incoherent decision. The probability of observing more than one incoherent decision was 56%, and the probability of observing at least five incoherent decisions was 18%. The maximum number of incoherent decisions in any one simulated trial was 12 decisions. The maximum number of incoherent de-escalation decisions in any one simulated trial was 9 decisions, and the maximum number of incoherent escalation decisions made in any one simulated trial was 3 decisions. For the Keyboard method and a target rate of 20%, the percentage of simulated trials with at least one incoherent decision was 78.4%. The probability of observing more than one incoherent decision was 58.1%, and the probability of observing at least five incoherent decisions was 24.6%. The maximum number of incoherent decisions in any one simulated trial was 13 decisions. Like the Bayesian optimal interval method, the maximum number of incoherent de-escalation decisions in any one simulated trial was 11 decisions, and the maximum number of incoherent escalation decisions made in any one simulated trial was 2 decisions. For the Keyboard method and target rate of 30%, 78.0% of simulated trials resulted in at least one incoherent decision. The probability of observing more than one incoherent decision was 57%, and the probability of observing at least five incoherent decisions was 20%. The maximum number of incoherent decisions in any one simulated trial was 12 decisions. The maximum number of incoherent de-escalation decisions in any one simulated trial was 9 decisions, and the maximum number of incoherent escalation decisions made in any one simulated trial was 3 decisions. Overall, these results illustrate that, when using interval-based methods, there is a substantial probability (≈77%−79%) that investigators will encounter an occurrence in which the method will recommend an incoherent decision, there is a greater than 50% probability that this will occur more than once in a single 30 patient phase I trial, and this could potentially be faced in as many as 13 dose assignment decisions in such a trial.

FIGURE 4.

The percentage of simulated trials that incurred at least each observed number of incoherent decisions for the Bayesian optimal interval method and the Keyboard method. For target dose-limiting toxicity rates 20% and 30%, 100 dose-toxicity scenarios were randomly generated from the Clertant and O’Quigley (2017) class of curves. DLT = dose-limiting toxicity; BOIN = Bayesian optimal interval.

4 |. CONCLUSIONS

In this article, we have studied the coherency behavior of interval-based dose-finding methods. Coherency, as originally defined by Cheung¹, is an important consideration in the conduct of phase I clinical trials, a property that ensures clinical acceptability of the method being used¹³. Interval-based designs fail this principle, and would require additional modifications to overcome this shortcoming. One advantage of the interval-based designs are their ability to output pre-tabulated escalation and de-escalation decision rules, similar to a 3+3 design. This perceived advantage over the continual reassessment method is diminished by the recent emergence of available software for generating dose transition pathways¹⁴, which allow model-based dose assignments to be anticipated in one or more cohorts. These projections can be included in the protocol so as to ensure clinically sensible dose recommendations are made and expedite the protocol review process¹⁵. They also allow for the creation of simple trial reports throughout the study, which augment communication between clinical and statistical members of the study team. Moreover, if additional dose assignment rules are required to ensure coherence of interval-based methods, this advantage is further mitigated since it would be difficult to assemble all of this information into a single clear table. Overriding a single dose assignment due to incoherence may not have a tremendous effect on the operating characteristics of a dose-finding method, but we do not know the impact on performance of having to do so several times throughout a small study. Operating characteristics of interval-based methods with additional rules to guarantee coherence remains a topic for future study. The advantage of simplicity for this class of dose-finding methods is accompanied by the questionable trade-off of accepting the considerable probability of making an incoherent decision.

The continual reassessment method, whether likelihood-based or Bayesian, is coherent as long as the model-based dose recommendation is followed¹. The two-stage, likelihood-based version of the continual reassessment method may be incoherent at the transition point from the algorithmic-based stage one to the model-based stage two. However, the initial escalation design can be calibrated in order to create a algorithmic-based stage that is compatible with the model-based stage, thus avoiding incoherency at the seam of the two stages¹⁶. The continual reassessment method demonstrates more accurate maximum tolerated dose selection than interval-based methods¹⁷, and it is always coherent when the model-based dose-recommendation is followed. The availability of easy-to-use software¹⁸ should facilitate its use in practice. The semi-parametric method of Clertant and O’Quigley¹¹ was also proved to be coherent, and escalation with overdose control methods have been shown, under certain conditions, to be coherent^19–21.

We have not studied the notion of group-coherence in this investigation. That is to say, it may appear that the drawbacks highlighted in interval-based methods could be avoided by including larger cohort sizes at each allocation decision. Cheung¹ defines a method as being group-coherent in escalation if it escalates only when the observed toxicity rate in the most recently accrued cohort is strictly less than the target toxicity rate. Group-coherence differs from long-term memory coherence in that group-coherence is evaluated using data only from the most recently accrued cohort, whereas long-term memory coherence is assessed using data from all patients accrued to the current dose level throughout the study. Cheung¹ proves group-coherence for the Bayesian continual reassessment method in the case where doses are assigned according to the posterior mode of the model parameter with a unimodal prior. The reason we chose to study the principle of coherence only in cohorts of size 1 is that, as pointed out by Cheung²², the notion of group-coherence only makes sense in the case in which no information is available between the arrival of the first patient in a cohort and when the last patient in the cohort is treated. There are cases where patients within a cohort are not treated simultaneously, and toxicity outcomes may occur at some random time within the toxicity observation period. Consequently, extensive theoretical developments are needed and it is beyond the scope of this work since, even if established, it may be difficult to interpret. The purpose of this manuscript is to help clinical investigators better understand in trial behavior of interval-based Phase I methods and encourage researchers to take a closer look at the theoretical properties of these designs.