The Need for Combined Assessment of Multiple Outcomes in Noninferiority Trials in Oncology

Abstract

Noninferiority trials in oncology assess novel therapies with the potential for slightly worse recurrence or death outcomes (ie, the margin of noninferiority) than standard therapies. This poses a dilemma because, in the absence of potential health outcome advantages, these trials may not provide the treatment equipoise required for an ethical study. Any new treatment with the potential for slightly worse recurrence or death outcomes should have countervailing health outcome advantages, but these are rarely taken into account in the design of noninferiority trials. This article presents the argument that not only the potentially worse health outcomes but also the potential benefits of the novel therapy should be considered when designing, analyzing, and reporting noninferiority trials. Some approaches to study design and analysis that consider both primary and secondary end points are discussed, and reporting the joint distributions of end points for the novel and standard treatments is recommended.

Noninferiority trials are widely used to assess novel therapies in oncology. Notable recent clinical trials for breast cancer, ovarian cancer, colorectal cancer, bladder cancer, central nervous system lymphoma, oropharyngeal cancer, and other malignant conditions have used the noninferiority design.^1–10 Noninferiority trials assess whether novel therapies have (at worst) a tolerable magnitude of inferiority, called the margin of noninferiority, when compared with standard therapies.^11,12 Thus, noninferiority trials in oncology may assess novel therapies with the potential for slightly worse recurrence or death outcomes than standard therapies. However, any modestly inferior new treatment should have countervailing advantages in health outcomes when compared with the standard treatment, such as reduced toxic effects or improved quality of life.^1–5,7,9,10 Also, the new treatment may have nonhealth advantages, such as a lower monetary cost.⁷ In the absence of potential outcome advantages, it is difficult to justify the treatment equipoise required for an ethical noninferiority study.¹³

Yet the results of the new treatment on potentially beneficial secondary health outcomes are rarely considered when designing noninferiority trials.¹⁴ In particular, sample sizes are calculated on the basis of a specified margin of inferiority for the main end point and do not take other health outcomes into account.¹² In this article, we review design principles for noninferiority trials, discuss approaches to analysis that take primary and secondary end points into account, and make suggestions to improve the design and reporting of noninferiority trials.

Superiority vs Noninferiority Trials

For either a superiority or noninferiority trial, there is a health outcome (or end point) of primary interest, and the purpose of the trial is to compare the mean outcome on the new treatment vs the mean outcome on the standard treatment. If δ is the mean beneficial primary health outcome of the new treatment minus the mean beneficial primary health outcome of the current standard treatment, then a δ greater than 0 implies that the new treatment is superior.

The superiority trial tests the null hypothesis that δ is less than or equal to 0.¹⁵ A positive trial result rejects this null hypothesis in favor of the alternative hypothesis (a δ greater than 0), which indicates that the new treatment is superior. When asking a patient to participate in such a trial, a statement of equipoise might be, “We do not know if the new treatment is better than the standard treatment, but we hope it is, and this trial is designed to see if it is better”

A noninferiority trial defines a noninferiority margin (δ₀) less than 0, which is the amount by which the new treatment can be inferior to the standard treatment but still good enough to be acceptable.^11,16 A noninferiority trial tests the null hypothesis that δ is less than or equal to δ₀, and a successful trial rejects this null hypothesis in favor of the alternative hypothesis that δ is greater than δ₀ (ie, that the new treatment is not more than δ₀ units worse than standard treatment).¹⁵ When asking a patient to participate in such a trial, it would be true but awkward to say, “We do not know if the new treatment is better or worse than the standard treatment, but we want to establish that if it is worse, it is only a little worse than the standard treatment.”

Given that a new treatment may have a slightly worse outcome than the standard, why might patients participate in a noninferiority trial? There are several possible reasons.¹⁷ First, some patients may want to contribute to scientific knowledge, and having a new treatment that is at most δ₀ worse than the standard may provide another therapeutic option for patients who cannot take standard treatment. For example, the standard treatment may have unacceptable adverse effects for a given patient, while other patients may not be able to take the standard treatment because of other constraints, such as cost. Second, contrary to expectations, the novel therapy might have better efficacy for the main end point overall or in some subsets of patients. This could happen if the new treatment is better tolerated than the standard treatment, thereby permitting better adherence to prescribed dosage schedules. Finally, patients may feel that they could benefit from the trial personally if the new treatment, although slightly worse than standard treatment with respect to the primary end point, is better than the standard treatment with respect to other health outcomes, such as toxicities.

An excellent example of a noninferiority trial was the National Surgical Adjuvant Breast and Bowel Project Protocol B-06,¹⁸ which was undertaken to determine whether lumpectomy with node resection alone or lumpectomy with node resection and radiation yielded as good outcomes as mastectomy for patients with primary breast cancer. This trial was not formulated statistically as a noninferiority trial, but the main end points, such as recurrence-free survival, were examined to see if the lumpectomy treatments were inferior. The clear benefits to patients from the lumpectomy treatments were less-extensive and less-disfiguring surgery, which might outweigh a small decrease in efficacy against the primary end point.

Method for Combined Assessment of Multiple End Points

There would be fewer ethical concerns if noninferiority trials assessed both the risks and benefits of the new treatment and patients entered these trials with the possibility of receiving a better overall therapeutic result with the new treatment. Thus, it is possible to argue that the design and reporting of noninferiority trials should consider outcomes on the several health end points affected by treatment and not only on the main end point. The need to describe treatment effects (defined as the difference in mean outcomes between the new treatment and control treatment) on multiple aspects of health applies to other trial designs as well.

In fact, the assessment of treatment effects on multiple aspects of health is an important feature of good trial design and reporting, dating back to the earliest trials.¹⁹ Yet even well-reported trials usually assess each health outcome individually and only present marginal estimates of treatment effects for each health outcome. Rarely are such treatment effects considered jointly. One exception is a phase 1/2 dosage-finding chemotherapy trial design that accepts a novel drug at a given dosage only if, with high probability, both the efficacy exceeds a minimum level and the toxic effects are less than a maximum level.²⁰ For such a trial, we would need to know not only the marginal probabilities of efficacy and toxic effects but also their joint probabilities. As another example, the data safety monitoring committee for the Women’s Health Initiative considered several approaches to monitoring its hormone trial,²¹ because estrogen and progestin were expected to have several health outcomes. One approach was to use a so-called global end point, such as total mortality. Another was to examine individual health outcomes, such as incidence of coronary heart disease, hip fractures, breast cancer, and endometrial cancer and deaths from other causes. At the time, hormone therapy was thought to be beneficial for coronary heart disease and hip fracture prevention but have adverse effects for the other outcomes. Therefore, the monitoring committee also considered a weighted linear combination (called the weighted combined index) of the treatment effects on end points. Evans and Follmann²² reviewed methods for multiple end points and stressed the need for a combined assessment of the several outcomes affected by treatments to make a pragmatic selection of the better treatment. Among the methods considered for combined assessment were the use of weights (w_ij), which we call losses, in the examples that we provide in this article.

If the various health outcomes change over time, then the health status of the patient can be summarized by a quality-of-life index at each point, and a mean quality-of-life index over the life of the patient (ie, quality-adjusted life-years) can be used to summarize and compare the benefits and harms of various treatments.²³ If there are a small number of mutually exclusive time-varying health states, such as time free of toxic effects and cancer recurrence (also known as quality-adjusted time without symptoms and toxicity [Q-TWiST]), time with toxic effects before recurrence, and time alive with recurrence, then the time spent in each state can be studied, and if losses are assigned to these states, the mean losses overtime can be computed and compared across treatments.²⁴

In examples, we illustrate a method of assessing the risks and benefits of the novel treatment vs those of the standard treatment. Suppose there are 2 end points, such as cancer recurrence and severe treatment toxic effects. The outcome of Y₁ equal to 1 indicates cancer recurrence, and Y₁ equal to 0 indicates no cancer recurrence. Likewise, Y₂ indicates the presence or absence of severe toxic effects. In those given the new treatment, the probabilities of the joint outcomes (Y₁, Y₂) are P(Y₁ = i,Y₂ = j|new treatment) = P_ij,_new for i and j equal to 0 or 1. Similar probabilities are defined for the standard treatment, $P_{\text{i}{\text{j}}^{\prime }\text{standard}}$.

Suppose we define losses w_ij associated with the joint outcomes (Y = i₁,Y₂ = j); these losses represent the medical harms of the joint events. For example, a w₀₀ equal to 0 might be the harm from having neither event and a w₁₁ equal to 6 might be the harm from having both events. Using these definitions, we can calculate the net benefit of the new treatment, δ_NB, as the reduction in expected losses from using the new treatment instead of the standard treatment (Table).

Table.

Calculation of the Net Benefit (or Reduction in Expected Losses) of Using the New Treatment in Place of the Standard Treatment

Joint Outcome (Y1Y2)	(0,0)	(0,1)	(1,0)	(1,1)
Loss	W00	W01	W10	W11
Probabilities
New treatment	P00,new	P01,new	P10,new	P11,new
Standard	P00,standard	P01,standard	P10,standard	P11,standard
Expected loss
New treatment	${\displaystyle \sum _{i,j}{w}_{ij}}{p}_{ij,\text{new}}$
Standard	${\displaystyle \sum _{i,j}{w}_{ij}}{p}_{ij,\text{standard}}$
Net benefit of new treatment	${\delta }_{NB}={\displaystyle \sum _{ij}{w}_{ij}}\left(p_{ij,\text{standard}}-p_{ij,\text{new}}\right)$

Some researchers assign the word utilities to health outcomes instead of losses. These approaches are equivalent in the sense that the net benefit from a new treatment is the reduction in the expected loss or the increase in expected utility.²⁵

Application of these ideas is context dependent. This is because the expected losses will depend on the nature of the health outcomes and the distributions of joint outcomes (ie, the combinations of primary and secondary outcomes) under the new and standard treatments.

We now provide 2 examples to illustrate estimation of the net benefit of a new treatment for a noninferiority trial. For the first example, consider breast cancer treatment, with Y₁ corresponding to recurrence and Y₂ to severe treatment toxic effects. Both these events can occur in the same patient. Assume w₀₀ is equal to 0, w₀₁ is equal to 1, w₁₀ is equal to 5, and w₁₁ is equal to 6. There are no medical losses in the absence of recurrence and severe toxic effects, the losses from recurrence are 5 times those from severe toxic effects, and the losses from recurrence and severe toxic effects are additive $\left(w_{11}=w_{00}+\left(w_{01}-w_{00}\right)+\left(w_{10}-w_{00}\right)=w_{01}+w_{10}=1+5=6\right)$. Assume values of $P_{\text{ij'estandard}}$ .for(Y = i₁ Y₂ = j) equal to (0,0), (0,1), (1,0), and (1,1) are 0.72, 0.18, 0.08, and 0.02, respectively. Then, for the standard treatment, the marginal probability of recurrence (ie, probability of recurrence considered alone) is P(Y₁ = 1|standard treatment), which equals 0.08 plus 0.02, or 0.10, and the marginal probability of severe toxic effects (ie, probability of toxic effects considered alone) is P(Y₂ = 1|standardtreatment), which equals 0.18 + 0.02, or 0.20. Suppose the new treatment has the corresponding $P_{ij\text{'}new}$ values 0.890, 0.005, 0.090, and 0.015. These probabilities imply a slightly worse marginal probability of recurrence (P[Y₁ = 1|new treatment] = 0.105), but a considerably smaller marginal probability of severe toxic effects (P[Y₂ = 1|new treatment] = 0.02). The inferiority of the new treatment with respect to the recurrence outcome alone is δ, or 0.100 minus 0.105, which is-0.005. However, the superiority of the new treatment with respect to severe toxic effects is 0.200 minus 0.020, or 0.180. Using the previously noted losses, we can compute the expected loss (1 × 0.18 + 5 × 0.08 + 6 × 0.2 = 0.70) for the standard treatment and the loss for the new treatment (1 × 0.005 + 5 × 0.090 + 6 × 0.015 = 0.545). The net benefit for the new treatment is therefore a δ_NB of 0.70 minus 0.545, or 0.155, and the new treatment is superior with respect to expected losses. This would be a good inducement for participation in a noninferiority trial with respect to recurrence, with a noninferiority margin of δ₀ of −0.005.

As a second example, let us suppose that the primary outcome Y₁ is death from any cause, and the secondary outcome Y₂ is severe toxic effects. Reasonable losses might be a w₀₀ equal to 0, w₀₁ equal to 1, w₁₀ equal to 10, and w₁₁ equal to 10. In this case, it is 10 times worse to die than have severe toxic effects. If one dies and has severe toxic effects, the (nonadditive) loss is no worse than dying without severe toxic effects. For the same probability distributions as in the previous example, the expected loss is 1.180 (1 × 0.18 + 10 × 0.08 + 10 × 0.02) for the standard treatment and 1.055(1 × 0.005 + 10 × 0.090 + 10 × 0.015) for the new treatment. Even though the inferiority of the new treatment with respect to death is a δ of −0.005, as before, the net benefit δ_NB is 1.18o minus 1 .055, or 0.125, which is again positive for the new treatment.

Implications for Reporting

Even well-written reports of clinical trials usually present intervention effects separately for the several health outcomes. To fully describe and summarize the intervention effects, however, one needs to present the joint distributions of health outcomes on the several end points, not just their marginal distributions, separately for the new and standard treatments. Such joint information is needed to compute an expected loss for each treatment, as in the previous examples. An exception arises if the losses are additive across health outcomes; then only are marginal outcome distributions needed to estimate net benefit. However, the joint distributions are still needed to compute confidence intervals with the net benefit, unless the outcomes are rare. In the absence of data on the joint distribution of outcomes, one is forced to make the untestable and dubious assumption that the outcomes are statistically independent. In the era of automated data processing, it is possible to store and convey these joint distributions, and, indeed, it is possible in some circumstances that even the deidentified outcomes measured on each patient could be shared.

Clinical Relevance and Design Considerations

To decide whether to use the new treatment, some formal or informal way must be found to combine the intervention effects on the several health outcomes. One approach is by computing expected losses, as described in the Table. One of the most problematic aspects of computing expected losses is defining the losses or weights. In a publication, authors can posit a reasonable set of weights (or perhaps a few such sets of weights) and present the net benefit of the new treatment separately for each set of weights. However, any user can supply weights and compute an expected loss with those weights, provided the joint outcome distributions are available for the new and standard treatments. Furthermore, when counseling a patient, a clinician might ask whether the published weights are acceptable, and if so, they may make clinical decisions based on those weights and the corresponding net-benefit calculations. If the patient wants to use his or her own weights, it should be possible to compute net benefit for those weights using automated calculators, provided the joint outcome distributions are available.

There are important limitations to methodology based on expected losses, apart from specifying the losses. Clinical decision-making is complex, and medical interventions invariably affect more than 2 outcomes. Moreover, some of these outcomes, such as those pertaining to quality of life, might be difficult to measure. Also, there should be consensus as to which outcomes merit assessment, and both clinical investigators and patients should have input in the decision-making process. Adaptations would be necessary for measuring specific types of outcomes, such as level of a continuous biomarker or survival time. One could argue that it is impossible to assign losses to so-called incommensurate outcomes, such as death and severe toxic effects, and one should consider each outcome separately. Yet even this informal perspective would be informed by the joint outcome distributions. It is important to emphasize that methods based on expected losses can allow for patient input in the clinical decision-making process. However, eliciting a patient’s preferred weights may be time-consuming or confusing.

There are other, less formal ways to take multiple outcomes into account in designing and reporting noninferiority trials. The choice of the noninferiority margin δ₀ for the main end point is often based on ad hoc criteria, such as taking δ₀ to correspond to half of the benefit of the standard treatment.¹² If there is external evidence that the new treatment is much less toxic than standard treatment, one might be willing to lower the inferiority margin δ₀, leading to a smaller sample-size requirement. Although this sample-size calculation is still driven by the main end point, the trial should present data on toxic effects to assess whether, in fact, the new treatment is much less toxic, as hypothesized.

If researchers were willing to base the choice of treatment on an expected loss and specify losses and hypothesized joint distributions of the outcomes under the standard and new treatments, they could design the trial with a net benefit as the main outcome. If researchers anticipate that the new treatment will have a positive net benefit, they could compute the sample size needed to have good power to demonstrate the anticipated positive net benefit, thereby turning a noninferiority trial into a superiority trial. In some settings, the primary end point might be so crucial that it is imperative to have high precision on the estimate of the treatment effect on this end point. If the sample-size calculations for net benefit do not yield the precision needed for the main end point, then the sample size needed for a standard noninferiority trial might be recommended. Nonetheless, the reporting could also include estimates of net benefit.

Conclusions

We think that investigators need to pay more attention to how to measure secondary health outcomes and how to include them in analyses of both risks and benefits in noninferiority trials. Consideration of benefits from the new treatment on secondary outcomes also has implications for design. Informally, such benefits could justify a larger margin of noninferiority, reducing required sample sizes. Formally, if the aim of the study was changed to show that the new treatment had smaller expected losses and therefore a positive net benefit, a different sample size might be needed from the number computed for a study to have sufficient power to reject a specified margin of noninferiority.

Noninferiority trials that measure and summarize information on the risks and benefits of novel therapies on several health outcomes may have greater appeal to potential study participants and lead to more informed clinical decision-making. Although further clinical and methodological work is needed to make these ideas clear and practical in specific applications, it is time to make clinical trial data on joint outcomes available in electronic format.