Bayesian designs to account for patient heterogeneity in phase II clinical trials

Abstract

Purpose of review

Between-patient heterogeneity is very common in clinical trials. This complicates treatment evaluation, due to known prognostic subgroup effects or potential treatment-subgroup interactions. We review two Bayesian phase II clinical trial designs that account explicitly for patient heterogeneity. The first design uses analysis of covariance to assess treatment effects in subgroups known to have different prognoses. The second design uses a hierarchical model for settings where, a priori, the experimental treatment effects in the subgroups are assumed to be exchangeable.

Recent findings

Compared to simpler designs that ignore patient heterogeneity, each design provides substantial improvements by reducing both false positive and false negative rates and focusing resources on subgroups where an experimental treatment is more likely to provide an advance over standard therapy. In either case, accounting for potential treatment-subgroup interactions is extremely important.

Summary

Due to the rapidly increasing number of potential new treatments to be evaluated clinically, increasing costs and limited resources, it is critically important to perform early phase clinical trials efficiently. The new Bayesian methods described here, and other related methods, provide efficient, broadly applicable tools to address these problems.

Introduction

Although between-patient heterogeneity is common in clinical trials, most phase II designs assume patients are homogeneous [1–6]. This may lead to severe errors when evaluating an experimental treatment, E. A design that ignores known differences between prognostic subgroups runs substantial risks of reaching erroneous conclusions about the effects of E within subgroups. Alternatively if, a priori, there is no known difference between subgroups, assuming one overall response rate does not accommodate the possibility that the effectiveness of E may differ between subgroups. We will describe two Bayesian designs [7, 8] that address these settings by accounting for patient heterogeneity.

Some Bayesian concepts

Since both of the designs we will discuss are Bayesian, we first review some Bayesian concepts. The two main objects in a statistical analysis are the observable data and the model parameter, θ. Parameters are not observable but rather are conceptual objects that characterize important aspects of the phenomenon being studied, such as the probability of tumor response, median survival time, or effects of patient characteristics on clinical outcome. Often, θ is a vector including several such quantities. In the Bayesian paradigm [9–11] θ is considered random. To reflect one’ s knowledge and uncertainty about θ before an experiment, a probability distribution, prior(θ), is assumed. For example, when asked to specify the probability of tumor response with standard therapy, S, for a particular disease, an oncologist might reply “Somewhere between 16% and 35%.” Denoting the response probability with S by θ_S, the assumption that θ_S follows a beta(20, 60) prior distribution yields the equation Pr(.16 < θ_S < .35) = .95, represented by the shaded area under the solid curve in Figure 1A. That is, under this probability distribution, [.16 – .35 ] is a Bayesian “95% credible interval” for θ_S, which formalizes the oncologist’s statement. Equivalently, a beta(20, 60) distribution corresponds to having previously observed 20 responders and 60 non-responders in 80 patients treated with S. Suppose that θ_E is the response probability with an experimental treatment, E, in a planned phase II trial. To reflect substantial uncertainty about E one might assume that θ_E follows a beta(.25, .75) prior, represented by the dotted curve in Figure 1A. These two distributions have the same mean, .25, but the beta(.25, .75) has effective sample size of 1 patient whereas the beta(20, 60) has effective sample size of 80 patients, reflecting almost no knowledge about θ_E and substantial knowledge about θ_S.

Fig. 1.

Fig. 1A. Illustration of beta distributions. A beta(20,60) is shown by the solid line, with its 95% credible interval shown by the shaded region under the curve. A beta(.25, .75) distribution is shown by the dotted line.

Fig. 1B. Illustration of beta distributions. A beta(.25, .75) prior is shown by the solid line, and the posterior distributions that would be obtained if 3/10, 8/20 or 20/40 responses were observed are given by dashed lines.

Once data have been observed, Bayes Law is applied to compute a new probability distribution, posterior(θ | data), that quantifies what has been learned about θ from the data. The vertical bar can be read to mean “given that one knows” or “given that one has observed.” The posterior is used to make statistical inferences about θ. In contrast, classical “frequentist” statistics considers θ to be fixed but unknown. Advances in computational methods have greatly accelerated practical application of Bayesian methods in recent years, especially in biostatistics [12] and clinical trials [13–16]. The Bayesian paradigm provides a natural framework for making decisions sequentially based on accumulating data during a clinical trial, as is done in a phase II trial with interim monitoring rules, since Bayes’ Law may be applied repeatedly by using the posterior obtained after each stage as the prior for the next stage. In the above example, a Bayesian rule to stop the trial of E due to futility might take the form Pr(θ_S + .15 < θ_E | data) < .05, which says to stop accrual if a .15 improvement of E over S in response probability is unlikely, given the observed data. Figure 1B gives the posteriors of θ_E that would result if 3 out of 10 (30%), 8 out of 20 (40%), or 20 out of 40 (50%) responses were observed on E in a phase II trial. For these three outcomes, the respective values of the decision criterion probability Pr(θ_S + .15 < θ_E | data) are .23, .47 and .85.

Phase II trials with known prognostic subgroup effects

Most designs for phase II trials make the simplifying assumption that a single response probability applies to all patients, i.e., that patients are homogenous. While technically convenient, this assumption may lead to highly flawed conclusions. We will discuss two cases. In the first case, there are known prognostic differences between patient subgroups. We will show that a statistical design that ignores prognosis by assuming one response probability, θ_E, with E for all patients has a substantial risk of producing false positive or false negative conclusions about the effects of E within subgroups.

For simplicity, we will illustrate the first design in the case of two known prognostic subgroups, good (G) and bad (B). The design may easily be applied more generally in settings with more than two subgroups, although the per-subgroup sample size must be large enough to obtain reasonably reliable results. In this example, to keep track of the four treatment-subgroup combinations, we will denote the subgroup-specific tumor response probabilities by

$$\begin{array}{l}{\mathrm{\theta }}_{\mathrm{S},\mathrm{G}}=\text{Prob}(\text{response}\mid \text{treated}\text{with}\mathrm{S}\text{in}\text{subgroup}\mathrm{G}),\\ {\mathrm{\theta }}_{\mathrm{S},\mathrm{B}}=\text{Prob}(\text{response}\mid \text{treated}\text{with}\mathrm{S}\text{in}\text{subgroup}\mathrm{B}),\\ {\mathrm{\theta }}_{\mathrm{E},\mathrm{G}}=\text{Prob}(\text{response}\mid \text{treated}\text{with}\mathrm{E}\text{in}\text{subgroup}\mathrm{G}),\\ {\mathrm{\theta }}_{\mathrm{E},\mathrm{B}}=\text{Prob}(\text{response}\mid \text{treated}\text{with}\mathrm{E}\text{in}\text{subgroup}\mathrm{B}).\end{array}$$

The method of Wathen et al. [7] is based on a Bayesian analysis of covariance (ANCOVA) model for these response probabilities that incorporates the historical subgroup effects seen with S and also allows the subgroups to have different E-versus-S treatment effects. The model expresses each logit (θ) = log{θ/(1−θ)} as

$$[\text{treatment}\text{effect}]+[\text{subgroup}\text{effect}]+[\text{treatment}\times \text{subgroup}\text{interaction}].$$

For example, logit(θ_E,G) = [treatment E effect] +[subgroup G effect] + [E × G interaction]. Because the four probabilities given above share common parameters and are correlated under this model, it allows one to “borrow strength” between subgroups. Using the Bayesian formalism, an informative prior on the [subgroup effect] parameters is derived from historical data on S, while non-informative priors are assumed for all other parameters since they involve as yet unknown effects of E. Essentially, the data from the trial are used to learn about parameters involving effects of E. The design allows the possibility that E may be more effective than S in one subgroup but equivalent or inferior to S in another. For example, if an improvement of .15 in response probability is targeted within each subgroup then the criterion probability Pr(θ_S,G + .15 < θ_E,G | data) is used to decide whether to stop or continue in the G subgroup and, similarly, Pr(θ_S,B + .15 < θ_E,B | data) is used in the B subgroup. That is, the method uses subgroup-specific futility stopping rules, with the important provision that each rule is constructed so that the false negative rate (FNR) each subgroup is controlled below a specified small value, such as .10. Given a maximum total sample size, M, if accrual to the B subgroup is stopped early but the G subgroup is not stopped, then accrual of G patients is continued until a total of M patients have been enrolled in the trial. In this way, the design spends limited resources in subgroups where E is more likely to provide an advance over S.

As an example, suppose it is known that, with S, the probability of tumor response is .45 in patients with good prognosis and .25 in patients with bad prognosis. The most common approach that accounts for prognosis is to simply conduct two separate trials, one in G patients and the other in B patients. Using either this approach or the Wathen et al. ANCOVA-based design [7], a response rate of .45+.15 = .60 is considered “promising” in the G subgroup whereas .45 is not promising. Similarly, a response rate of .25+.15 = .40 is considered “promising” in the B subgroup whereas .25 is not promising. A third approach, which is the most common, is to ignore prognosis by assuming one overall response rate with S, the average value θ_S = .35, and one corresponding response rate θ_E with E. For a statistical design based on these simplifying assumptions that targets an improvement of .15 over S, θ_E = .50 corresponds to E being considered promising whereas θ_E = .35 or smaller implies E is not promising.

Table 1 gives computer simulation results comparing these three designs for a 100 patient trial of E monitored in cohorts of size 10. In Case 1, the null case where E provides no improvement over S in either subgroup, all three designs have reasonably high early stopping probabilities. The design that ignores subgroups appears to be the best approach of the three because it has the highest probability of correctly rejecting E, .86. Equivalently, it has the smallest false positive rate, FPR = 1 – .86 = .14. The approach of conducting separate trials within subgroups has smaller probabilities of rejecting E than the ANCOVA-based design in each subgroup because it does not borrow strength between subgroups. In Case 2, where E achieves the desired the .15 improvement over S in both subgroups, all three designs control the FNP at .10. In Case 3, there is treatment-subgroup interaction since E provides the desired improvement in the B subgroup, from .25 to .40, but not in the G subgroup. As in Case 1, the design that conducts separate trials within subgroups has a lower probability of rejecting E in the G subgroup compared to the ANCOVA-based design. However, the design that ignores subgroups has extremely poor properties, since the probability of stopping early and rejecting E equals .41 in both subgroups. This translates into FPR = 1 – .41 = .59 in the G patient subgroup and FNR = .41 in the B subgroup. While these error rates may seem surprisingly large, they illustrate the severe risks one runs if known patient heterogeneity is ignored. The flaw in ignoring the prognostic subgroups in this example can be seen by noticing that, accounting for patient heterogeneity, since a response rate of .40 is promising for the B subgroup and a response rate of .60 is promising for the G subgroup, the single value .50 targeted by the simple design is actually too small for G patients and too large for B patients. Unfortunately, the simple but error-prone approach of ignoring prognostic subgroups is that most commonly used in phase II trials.

Table 1.

Computer simulation results for a trial with up to 100 patients, monitoring in cohorts of size 10. Each design targets an improvement of .15 over each assumed null response probability, .25 in the bad (B) subgroup, .45 in the good (G) subgroup, or the single average null value .35 in the design that ignores subgroups.

			Pr(Reject E) (Average No. patients)
Case	Prognostic Subgroup	True Pr(Response)	Account for Subgroups Using ANCOVA	Conduct Separate Trials Within Subgroups	Ignore Subgroups
1	Good	.45	0.76 (38)	0.65 (34)	0.86 (24)
	Bad	.25	0.78 (37)	0.65 (33)	0.86 (24)
2	Good	.60	0.10 (50)	0.10 (47)	0.10 (47)
	Bad	.40	0.10 (50)	0.10 (47)	0.10 (47)
3	Good	.45	0.72 (33)	0.65 (34)	0.41 (38)
	Bad	.40	0.10 (64)	0.10 (47)	0.41 (38)

For comparability, all three of these designs considered above had maximum sample size 100 and monitoring schedule using cohorts of size 10. If one wishes to also consider the commonly used Simon optimal two-stage design [3], one must assume homogeneity (ignore subgroups) in order to apply this design. In a similar setting where there is a treatment-subgroup interaction, a Simon design with up to 43 patients and nominal FNR and FPR both set equal to .10 may have an actual FNR as high as .75 (Wathen et al., [7, Table III]). This large error rate has two causes, the first being that known prognostic differences are ignored, as in the above example. The second cause arises from the fact that, in general, a two-stage design is far less efficient than a design that monitors the accruing data more frequently.

Phase II trials starting with exchangeable subgroup effects

The second setting is that where, a priori, there is no known prognostic difference between subgroups and it is desired to conduct one trial including all subgroups. To simplify the illustration, we will consider a phase IIA trial where there is no standard treatment with anti-disease activity in any subgroup. The usual approach is to target a small fixed positive response probability, p, usually in the range.10 to .30. A Gehan design [1] is often used, or a Bayesian rule [17] that stops the trial if Pr(θ > p | data) is below a fixed small cut-off. Suppose that one wishes to allow the possibility that the effectiveness of E may differ between subgroups, but one also desires a statistical method in which a response observed in one subgroup leads to a higher estimated response rate in the other subgroups, that is, one wishes to borrow strength between subgroups. Neither assuming one θ for all subgroups nor conducting separate trials, one in each subgroup, provides a solution to this problem.

The second design [8] achieves both of the above goals by assuming a Bayesian hierarchical model. As a simple illustration, we will assume that there are 4 subgroups. Under the hierarchical model, there are four subgroup-specific response probabilities, θ₁, θ₂, θ₃, θ₄, that are generated from a common prior, the parameters of which themselves have second level prior, known as a a “hyper-prior,” which induces correlation among the four probabilities. Consequently, for example, data in subgroup 1 affect not only the posterior of θ₁, but also the posteriors of θ₂, θ₃, and θ₄. With this design, each subgroup has its own stopping criterion, Pr(θ₁ > p | data) for subgroup 1, Pr(θ₂ > p | data) for subgroup 2, etc., where, importantly, the “data” in each posterior probability includes the data from all 4 subgroups.

To see how this works in practice, suppose that a response probability .30 is targeted in each subgroup, we stop accrual to subgroup j if Pr(θ_j >.30 | data) < 0.05 and the prior correlation among the response probabilities corresponds to Pr(θ₁>.30) = .45 whereas Pr(θ₂>.30 | 2 responses in 6 pts. observed in subgroup 2) = .51 and Pr(θ₁>.30 | 2 responses in 6 pts. observed in subgroup 2) = .48. This is the type of information elicited from the oncologists planning the trial that is used to establish the hyperprior [8, section 4]. Table 2 illustrates how the decisions made conducting one trial under this hierarchical model may differ from those that would be made if, instead, 4 completely separate trials were conducted assuming an independent prior for each θ_j. In Case 1, under the hierarchical model the higher empirical response rates of .50, .67 and .83 observed in subgroups 2, 3 and 4 pull the posterior of θ₁ upward so that, despite the low empirical response rate of .17 in subgroup 1, it is not stopped early. In contrast, if trials are conducted independently in each subgroup using 4 separate, non-hierarchical models these data would cause subgroup1 to be terminated. Case 2 shows the same effect but with very different sample sizes within subgroups. Case 3 shows how this might work in the opposite direction, with the poor results in subgroups1, 2 and 3 pulling θ₄ downward.

Table 2.

Illustration of decisions made by futility stopping rules with one trial conducted under the hierarchical model, compared to conducting separate trials within the four subgroups. Each trial targets a .20 response rate in each subgroup. Each observed data entry is [# responses]/[# patients evaluated].

		Subgroup
Case		1	2	3	4
1	Observed Data	5/30	15/30	20/30	25/30
	Hierarchical Model	Continue	Continue	Continue	Continue
	Four Separate Trials	Stop	Continue	Continue	Continue
2	Observed Data	0/5	3/10	18/20	25/40
	Hierarchical Model	Continue	Continue	Continue	Continue
	Four Separate Trials	Stop	Continue	Continue	Continue
3	Observed Data	1/30	1/30	2/30	6/30
	Hierarchical Model	Stop	Stop	Stop	Stop
	Four Separate Trials	Stop	Stop	Stop	Continue

Figure 2 illustrates the effects of the assumed correlation among the four probabilities for a hypothetical trial taken from Case 3, with dashed lines showing the posteriors that would be obtained if θ₁, θ₂, θ₃, θ₄, were assumed to be independent and solid lines the corresponding posteriors under the hierarchical model. Note that the posteriors for θ₁ and θ₂ are shrunk more because subgroups 1 and 2 have smaller observed sample sizes of 5 and 10, whereas the posteriors for θ₃ and θ₄ are shrunk less because their sample sizes of 40 and 20 are much larger. This illustrates how the prior correlation induced by the hierarchical model affects the posteriors, and also the fact that, as with any reasonable Bayesian model, the accumulating data eventually overcome any prior assumptions.

Fig. 2.

Illustration of the within-subgroup posteriors from a hypothetical trial under either a simple model with four independent response probabilities (dashed lines) or under the hierarchical model (solid lines).

Conclusions

We have briefly reviewed two general Bayesian methods that account for heterogeneity in phase II. More complex, application-specific Bayesian methods that account for patient covariates using regression models have been described [18–20]. A general conclusion is that treatment-subgroup interactions may cause simple phase II methods to have extremely large false positive and false negative error rates within subgroups. Consequently, not only accounting for patient subgroups, but also accounting for the possibility of treatment-subgroup interactions appear to be very important. Alternatively, if an investigator feels strongly that subgroup effects and treatment-subgroup interactions are highly unlikely, then a conventional design may be used.

A frequentist, hypothesis test-based approach to phase II trials that accounts for heterogeneity also has been proposed [21], but this method does not allow the possibility of treatment-subgroup interactions and thus suffers from potentially very large error rates within subgroups.

We have not addressed the related but very different problem of identifying new prognostic subgroups, which usually is not feasible in phase II due to limited sample sizes [22].