Non-Comparative Comparisons: The Paradox of Randomized Non-Comparative Trials

In the phase II HCC 15-132 trial, investigators asked whether adding concurrent or sequential pembrolizumab to standard-of-care chemoradiation for locally advanced head and neck squamous cell carcinoma should be examined in future, definitive phase III trials.¹ To test this, 80 patients were randomized 1:1 to each experimental arm. However, although randomized trials traditionally compare the outcomes of each arm, HCC 15-132 was designed as a randomized, non-comparative trial (RNCT), meaning that formal statistical comparisons between arms were precluded in a pre-specified fashion.² Consequently, the primary analysis of the trial evaluated a trivariate composite endpoint separately within each arm. If both arms met the trivariate composite endpoint, which turned out to be the case, then the protocol specified that the mean estimate of the 1-year progression-free survival (PFS) probability would be numerically compared between arms to “pick the winner.”

RNCTs are a relatively novel concept in clinical trial design, though our recent meta-epidemiological analysis found that RNCTs are being increasingly used, especially in oncology.³ By precluding statistical comparisons between randomized arms, RNCTs promise reduced sample-size requirements and thus provide early-phase trials an expedited, resource-conserving means of selecting therapeutic signals for more definitive late-phase testing. Accordingly, the HCC 15-132 trial protocol noted that “an early-phase trial cannot provide the statistical power required for a formal statistical comparison between arms.”

However, RNCTs face a number of challenges. First, RNCTs are themselves a methodological paradox.² The purpose of randomization is to facilitate comparisons between arms; thus, randomizing patients without comparison negates the primary benefit offered by random treatment allocation.^4,5 Indeed, there is a strong natural inclination among researchers to compare outcomes between randomized arms in RNCTs exactly because randomization is meant to provide statistical robustness for such comparisons. However, formally precluding such comparisons creates a stark dilemma between presenting interesting (but post-hoc) comparative analyses versus remaining aligned to the protocol and RNCT design. In HCC 15-132, post-hoc Cox regression comparisons are reported for multiple endpoints, and this is commonplace in RNCTs.³ Caution is always warranted with post-hoc analyses, as the lack of pre-specification can increase the risk of potential bias.

As emphasized, in RNCTs, the outcomes of randomized arms are separately compared to historical controls or point thresholds. In HCC 15-132, three outcomes within each arm were numerically compared to point thresholds. However, as both arms met the initial primary endpoint criteria, HCC 15-132 took the additional pre-specified step of numerically comparing a mean estimate of 1-year PFS probability between arms, which is unusual for RNCTs. This analysis is furthermore difficult to interpret for multiple other reasons. First, the milestone survival estimates of each arm are themselves challenging to estimate reliably without random sampling techniques or rigorous modeling accounting for selection biases.⁶ Second, this comparison between arms, even though pre-specified, contradicts the foundational methodology of an RNCT. Third, numerical milestone survival estimates disregard all other data on the survival curve, including the change in survival probability over time prior to the milestone timepoint and all subsequent survival information. As such, milestone estimates such as 1-year PFS probability do not represent the survival function and are typically presented only descriptively.^7,8

HCC 15-132 represents a notable effort to address an important question, but its findings remain difficult to interpret due to the RNCT design and post hoc contradictions to RNCT design principles.³ The foundational statistical issues inherent to RNCTs merit greater attention by oncological researchers and methodologists, and the increasing use of RNCTs should be revisited.² Recognizing that financial support for trials is often limited, how then should early-phase signal-finding trials be designed? If the research seeks to estimate the outcomes of a new experimental arm, a well-conducted single-arm trial is entirely appropriate. Alternatively, if the research seeks to “pick the winner”, randomized comparative trials are the gold standard. Multiple opportunities exist to reduce sample sizes and costs of early-phase randomized trials, while still preserving the comparative essence of randomization. Surrogate endpoints with high expected event rates are commonly used for this reason. Regressing for strongly prognostic covariates adds power at no additional cost.⁹ For signal-finding trials, α can be increased. Beyond α and P values, the probability of superiority in early-phase randomized trials can be readily estimated using Bayesian designs that can reliably compare two treatment arms with sample sizes as low as 90 patients total, even when each arm is divided across three biomarker subgroups.¹⁰ Thus, ample, robust statistical mechanisms are available to facilitate signal identification in resourced-constrained early-phase trials without falling into the avoidable interpretative traps of RNCTs.