Strategies to Boost Statistical Efficiency in Randomized Oncology Trials with Primary Time-to-Event Endpoints

Abstract

Oncology clinical trials are increasingly expensive, necessitating efforts to streamline phase II and III trials to reduce costs and expedite treatment delivery. Randomization is often impractical in oncology trials due to small sample sizes and limited statistical power, leading to biased inferences. The FDA has recently published guidance documents encouraging the use of prognostic baseline measures to improve the precision of inferences around treatment effects. To address this, we propose an extension of Rosenbaum’s exact testing method incorporating a variant of martingale residuals for right censored data. This method can dramatically improve the statistical power of the test comparing treatment arms given time-to-event endpoints as compared to the standard log-rank test. Additionally, the modification of the martingale residual provides a straightforward metric for summarizing treatment effect by quantifying the expected events per treatment arm at each time-point. This approach is illustrated using a phase II clinical trial in small cell lung cancer.

1. Introduction

The cost of conducting an oncology clinical trial per patient was approximately $59,500 as reported in 2013 by Batelle [1]. However, recent studies focusing on cellular therapies indicate significantly higher costs, reaching as much as $500,000 or more per cycle [2]–[4]. Consequently, there is a pressing need to streamline phase II and phase III randomized trials, aiming to reduce both their duration and sample sizes. This imperative not only stems from a desire to cut costs but also from the urgency to expedite the delivery of successful treatments to the broader cancer patient population.

In rare disease and personalized medicine oncology trials, randomization is uncommon due to small sample sizes and limited statistical power, leading to reliance on descriptive analyses and potential bias. Employing statistical methods that support adequately powered randomized studies could improve current practices significantly. A common approach is to use real-world data (RWD) as control arms, combined with matching strategies[5] to reduce confounding. However, regulatory guidance often overlooks time-to-event data, a crucial endpoint in oncology trials. Standard methods like Cox regression and log-rank tests struggle with the issue of the non-informative censoring assumption in RWD scenarios, where informative censoring can bias results and weaken study validity[6]. Developing statistical efficient methods based on randomization strategies are critical for accurate assessment of treatment efficacy in these trials.

In general, the most common strategy to control for prognostic variables measured at baseline and to boost power and induce balance among treatment arms is stratified randomization [8] and the subsequent use of such factors in the statistical modeling. However, there are certain logistical constraints relative to stratification in terms of the number of stratification variables relative to the overall sample size. In addition, stratification often requires the dichotomization or categorization of continuous prognostic covariates, which decreases their utility in terms of boosting the statistical efficiency about the main test of treatment differences[7].

When the number of prognostic variables is large, an alternative to stratification is the minimization approach [8], in which treatment assignment for the $i$th subject depends on the vector of baseline covariate values for that subject, along with the assignments and covariate values of all $i-1$ previously assigned subjects. Relevant to this note, recent theoretical advances have addressed proper inference under minimization in the context of a time-to-event endpoint [9]–[11].

Under broad additivity, defined as fixed subject-level and treatment-level effects combined with a random error term [13]—the assumptions traditionally associated with permutation tests—a simulation study [12] has demonstrated that asymptotic type I error control is maintained when permuting both treatment assignment and covariates, and testing the regression coefficient associated with the treatment indicator variable. Although the authors in [12] refer to these results as a “randomization test,” it is, in fact, a permutation test. This distinction is supported by theoretical and simulation results demonstrating asymptotic type I error control for permutation tests concerning correlation and regression coefficients [14].

An alternative approach involves adjusting for covariates within analysis-of-covariance or regression models under parametric assumptions. However, as Simon and Pocock [15] observe, “Though covariate analysis could be profitably used in many cases, it is a widespread practice, even in large clinical trials, to rely solely on the ‘comparability’ of treatment groups as a basis for inference.”

Despite the FDA’s endorsement of covariate adjustments in inferential procedures, their use remains infrequent in phase III oncology trials. The FDA [19] asserts that covariate adjustment enhances efficiency, particularly when the covariates are prognostic for the trial’s outcome. As noted in the guidance document [19], “this can be done with minimal impact on bias or the Type I error rate.” Accordingly, the agency advises sponsors to incorporate adjustments for covariates expected to show the strongest association with the outcome of interest.

An alternative method, the randomization test approach under strict additivity assumptions [13], proposed by Rosenbaum [20], provides a unified global test of treatment differences while incorporating covariate adjustments. Strict additivity assumes that the only stochastic component in the procedure is the randomization mechanism, with observations expressed as the sum of a fixed treatment effect and a fixed subject effect, both known solely through the randomization process. In contrast, broad additivity assumes that observations are the sum of random treatment effect with a known expectation and a fixed subject effect, where the observed treatment effect is assumed to vary across possible permutations of the treatment assignments. This distinction differentiates a randomization test from a permutation test.

Practically, randomization and permutation tests yield identical inferential procedures but are based on slightly different theoretical underpinnings. Rosenbaum’s approach addresses long-standing concerns regarding the interpretation of results when covariates are used to enhance statistical efficiency—an issue that has been discussed extensively over several decades [15]. Notably, this method is sometimes referred to as a permutation test under broad additivity assumptions [13].

A Cox regression model can also adjust for prognostic variables, potentially enhancing the efficiency of the primary test for treatment effects, provided the proportional hazards assumption holds. While highly powerful under these assumptions, the Cox regression test relies on asymptotic approximations. In contrast, the Rosenbaum test offers an exact, nonparametric randomization test. Importantly, in randomized designs, the Rosenbaum test precisely controls the Type I error rate, ensuring it remains below the desired significance level.

Numerous publications have examined the inflated Type I error rates associated with the Cox model [16]–[18]. As Shao et al. note, the Cox model, based on asymptotic approximations, often performs poorly in settings with few events or substantial censoring in one or both treatment groups. While Shao et al. developed an exact test for comparing two groups based on the Cox model, their method does not accommodate prognostic variables, limiting its applicability in this context. The implications of inflated Type I errors in oncology clinical trials are substantial, potentially advancing treatments to market without the strongest evidence.

The Rosenbaum approach is considered a viable method by the FDA, which explicitly states, “Sponsors can conduct randomization/permutation tests with covariate adjustment (Rosenbaum 2002)” [19]. When the outcome variable of interest is not censored, the Rosenbaum randomization test with covariate adjustment is straightforward and serves as a global test for mean treatment effects. This approach relies on the assumption of strict additivity, as defined above.

In Section 2, we define a novel extension of Rosenbaum’s method [20] tailored to handle right censored data by integrating a modified version of a martingale residual proposed by Therneau [25] into our testing approach. We implement the Rosenbaum [20] randomization test while accounting for right censored data. We then describe various possibilities of this approach, ranging from a straightforward parallel design to the inclusion of prognostic covariates and minimization techniques for treatment assignment. In Section 3, we provide a comprehensive simulation study comparing several scenarios defined in Section 2. In Section 4, we illustrate our new approach using data from a randomized phase II clinical trial in small cell lung cancer. Finally, in Section 5, we provide some closing remarks.

2. Randomization Test Given Right Censored Data

In this section, we present a novel implementation of a randomization test tailored to handle right-censored data by integrating a modified version of the martingale residual proposed by Therneau [25] into our testing framework.

In the straight parallel comparison the randomization test proceeds as follows: For any outcome variable—whether discrete, ordinal, or continuous—we can apply an exact randomization test. Here, the outcome variable of interest is the UMR. To this end, let the total number of subjects be denoted as $N=n+m$, where $n$ represents the number of subjects randomly assigned to treatment $A$, and $m$ represents those assigned to treatment $B$. Consequently, there are $\left(\genfrac{}{}{0pt}{}{N}{n}\right)$ possible treatment assignments, each occurring with equal probability $\frac{1}{(\genfrac{}{}{0pt}{}{N}{n})}$.

Let $I_i$ denote the randomization indicator for the $i$th subject ($i=1,2,\dots ,N$), where $I_i~\mathit{\text{bernoulli}}(\pi )$, and $P(I=1)=\pi \in (0,1)$. Following Rosenbaum’s guidelines [20], the observed treatment outcome for the $i$th subject, measured by the UMR, is represented as

$$R_i=I_i{\mu }_{A_i}+\left(1-I_i\right){\mu }_{B_i},$$

where $I_i=1$ if subject $i$ receives treatment $A$, and $I_i=0$ if subject $i$ receives treatment $B$. Each subject $i$ is also associated with a vector of baseline prognostic covariates, denoted as $x_i$. The potential responses and covariates, (${\mu }_{A_i},{\mu }_{B_i},x_i$), are treated as fixed. Under the assumption of strict additivity [13], the only stochastic elements are the $I_i$’s, forming the foundation for a randomization test.

The terms ${\mu }_{A_i}$ and ${\mu }_{B_i}$ are defined as the sum of fixed subject and treatment effects, specifically ${\mu }_{A_i}=s_i+a_i$ and ${\mu }_{B_i}=s_i+b_i$, where $s_i$ denotes the fixed subject effect, and $a_i$ and $b_i$ represent the fixed treatment effects for $i=1,2,\dots ,N$. Hypothetically, if both treatments were applied to subject $i$, the difference ${\mu }_{A_i}-{\mu }_{B_i}=a_i-b_i$ would remain constant across subjects. Thus, experimental conclusions are drawn based on randomization. We can apply the extension of a randomization test developed by Rosenbaum [20] to boost efficiency by accounting for prognostic covariates as shown in Section 2.2.

Our primary objective is to investigate differences in excess mortality between treatment arms as a measure of treatment efficacy. For illustration, consider a straightforward parallel comparison between treatment $A$ and treatment $B$, where the outcome of interest is a time-to-event variable subject to potential right-censoring.

To this end, let $t_1,t_2,\dots ,t_N$ represent independent and identically distributed failure times, and let $c_1,c_2,\dots ,c_N$ denote the corresponding non-informative right censoring times, independently distributed and indexed by $i=1,2,\dots ,N$, pooled across treatments $A$ and $B$. Due to right censoring, we only observe $g\le N$ of the $t$’s, where $y_i=\text{min}\left(t_i,c_i\right)$ and ${\delta }_i=I_{\left(t_i>c_i\right)}$.

It is crucial to emphasize that within the framework of strict additivity [13], which governs our approach, the theory of randomization implies that all observations are fixed and thus may be treated as discrete. The only stochastic feature is the actual randomization mechanism. Hence, we represent the observations using lower-case letters.

Now, let $0<y_{(1)}<y_{(2)}<\cdots y_{(g)}$ be the distinct ordered observed failure times. The classic maximum likelihood-based derivation of the product-limit estimator starts by assuming the underlying distribution is discrete with probabilities ${\pi }_j=P\left(Y=y_{(j)}\right),j=1,2,\dots ,g$ for $g\le N$. Given a discrete hazard of $h_j=P\left(Y=y_{(j)}\mid Y\ge y_{(j)}\right)$ for $0\le h_j\le 1$, we have that ${\pi }_1=h_1,{\pi }_2=\left(1-h_1\right)h_2,\dots ,{\pi }_g=\left(1-h_1\right)\left(1-h_2\right)\cdots \left(1-h_{g-1}\right)h_g$. Then the estimator of $S(y)=P(Y>y)=\prod _{y_{(j)\le y}}\left(1-h_j\right)$ in the discrete case is given as

$$\hat{S}(y)=\prod _{y_{(j)\le y}}\left(1-{\hat{h}}_j\right),$$ (2.1)

where the estimates of the discrete hazard parameters follow from the log-likelihood

$$\text{log}L=\sum _{j=1}^g{d}_j\text{log}{h}_j+\left(r_j-d_j\right)\text{log}\left(1-h_j\right),$$ (2.2)

where $d_j$ denotes the number of events and $r_j$ denotes the number at risk at time $y_{(j)},j=1,2,\dots ,g$ [27]. Maximizing (2.2) with respect to the parameters $h_j,j=1,2,\dots ,g$ yields the estimates ${\hat{h}}_j=d_j/r_j$. The corresponding Nelson-Aalen estimator of the cumulative hazard function is given as $\hat{H}(y)=\sum _{y_{(j)\le y}}{\hat{h}}_j$. The product-limit based estimator of the cumulative hazard function is given as $\hat{H}(y)=-\text{log}(\hat{S}(y))$, where $\hat{S}(y)$ is given at (2.1). Asymptotically the two estimators are equivalent. For practical programming purposes using built-in R functionality we will utilize the product-limit based estimator of $H(y)$.

Now, let’s delve into the notion of the univariate martingale residual (UMR), serving as our principal endpoint measure, expressed as:

$${\mu }_i={\delta }_i-\hat{H}\left(t_i\right),i=1,2,\dots ,N,$$

where $\hat{H}(t)$ is defined above. In the univariate setting the UMR’s exhibit the property $\sum _{i=1}^N{\mu }_i=0$[25]. The UMR, as defined, can be interpreted at each observed $t_i$ as the expected number of excess deaths over $[0,t]$. We would expect a poorly performing treatment to have a large proportion of observed ${\mu }_i$’s greater than zero and a efficacious treatment to have a large proportion of observed ${\mu }_i$’s less than zero given the pooled sample. The values of ${\mu }_i$ can be readily integrated into the randomization test framework proposed by Rosenbaum [20] to create exact and efficient tests. Thus the distinction between a Cox regression model adjusting for covariates versus utilizing martingale residuals and the covariates of interest within the context of Rosenbaum’s test.

Comment on UMR’s. An added advantage of using UMRs is that, unlike situations where a substantial portion of observations are censored in one or both groups, rendering the median time-to-failure treatment effect inestimable by Kaplan-Meier curves, mean UMR estimates, along with their standard deviations and large-sample confidence intervals, are straightforward to calculate and interpret.

2.1. Straight Parallel Comparison

When there are no prognostic covariates of interest, corresponding to a straightforward two-group comparison, the null hypothesis under strict additivity concerning the UMRs can be expressed as:

$$H_0:F_A(s)=F_B(s),\forall s,$$

where the cumulative distribution functions of the UMRs for treatment $A$ and treatment $B$ are defined as:

$$F_A(s)=\frac{1}{m}\sum _{i=1}^N{I}_i\times I_{\left(r_i\le s\right)}$$

and

$$F_B(s)=\frac{1}{n-m}\sum _{i=1}^N\left(1-I_i\right)\times I_{\left(r_i\le s\right)},$$

where the $r_i$’s are the observed UMR’s. Subsequently, a suitable test statistic, typically one that captures location differences, distinguishing $F_A(s)$ from $F_B(s)$ may be employed, such as the exact Wilcoxon rank-sum test or randomization $t$-test. The p-value is determined by comparing the observed test statistic against all re-randomized values of the test statistic and computing the proportion of re-randomized values that are greater than or equal to the observed test statistic, assuming an alternative hypothesis of the form:

$$H_1:F_A(s)>F_B(s),\forall s.$$

A similar calculation pertains the alternative $H_1:F_A(s)>F_B(s)$, $\forall s$. For large sample sizes, the p-value is typically approximated using Monte Carlo approaches. Two-sided tests necessitate an additional assumption of symmetry for the re-randomization null distribution. We will utilize the convention for two-sided tests of comparing the proportion of re-randomized absolute values that are more extreme than the absolute value of the observed test statistic. The test of

$$H_0:F_A(s)=F_B(s),\forall s,$$

is exact under exchangeability assumptions implicit in randomized studies[26].

2.2. Inclusion of Prognostic Variables

To implement the Rosenbaum [20] randomization test while accounting for prognostic covariates, one can hypothesize either a linear or nonlinear relationship between the UMRs, denoted by the $N\times 1$ vector $\mathbf{R}$, and the covariates represented by $\mathbf{X}\in {\mathcal{R}}^p$. Here, $\mathbf{X}$ is an $N\times (p+1)$ matrix consisting of $p$ covariate values and an intercept column of 1’s. By fitting a model that regresses $\mathbf{R}$ on $\mathbf{X}$ (excluding a treatment indicator variable) separately for treatment groups $A$ and $B$, one can derive the classic least-squares residuals. These residuals, which capture the deviation of observed responses from the model’s predictions, provide the basis for treatment comparisons within the randomization test framework. Various statistical tests, such as the Wilcoxon rank-sum test or the randomization $t$-test, can then be applied to the treatment-specific residuals. Notably, the linear model acts as a data-driven approximation and does not impose strict adherence to stochastic modeling assumptions, thereby allowing the execution of an exact randomization test regardless of whether the model is correctly specified [20].

For clarity, we illustrate the approach using a simple linear regression model. However, it’s important to note that more complex nonlinear models may also be employed. For demonstration purposes, we employ classic least-squares regression. Let $\mathbf{e}$ represent the $N\times 1$ vector of residuals from the linear model, regression $\mathbf{R}$ on $\mathbf{X}$ (excluding an indicator variable for treatment), given by:

$$\mathbf{e}=(\mathbf{I}-\mathbf{H})\mathbf{R},$$

where

$$\mathbf{H}=\mathbf{X}{\left({\mathbf{X}}^{\prime }\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\prime },$$

$\mathbf{I}$ denotes the $N\times N$ identity matrix.

Now, we partition $\mathbf{e}$ into the residuals corresponding to treatment $A$ and treatment $B$, denoted by the $n\times 1$ vector ${\mathbf{e}}_A$ and the $m\times 1$ vector ${\mathbf{e}}_B$. The null hypothesis used to compare treatments now takes the form:

$$H_0:F_{{\mathbf{e}}_A}(s)=F_{{\mathbf{e}}_B}(s),\forall s,$$

where the cumulative distribution functions of the treatment-specific residuals are given as:

$$F_{{\mathbf{e}}_A}(s)=\frac{1}{m}\sum _{i=1}^N{I}_i\times I_{\left(e_i\le s\right)}$$

and

$$F_{{\mathbf{e}}_B}(s)=\frac{1}{n-m}\sum _{i=1}^N\left(1-I_i\right)\times I_{\left(e_i\le s\right)}.$$

The calculation of p-values follows a similar procedure to the case without prognostic covariates by utilizing $\mathbf{e}$ instead of $\mu$. In the linear models framework, it holds that ${\bar{e}}_A\approx {\bar{\mu }}_A$ and ${\bar{e}}_B\approx {\bar{\mu }}_B$, meaning that the average estimated treatment effects remain approximately unchanged with the addition of covariates. However, the inclusion of covariates leads to a reduction in the variance of the estimators. Consequently, there is an increase in statistical power afforded by the incorporation of prognostic covariates in the randomization test.

Type I Error Control. The probability of observing a specific configuration of the vector of residuals $\mathbf{e}$ is $1/\left(\genfrac{}{}{0pt}{}{N}{n}\right)$. Let $T^{*}$ denote the test statistic calculated for a given re-randomization used to test $H_0:F_A(s)=F_B(s)$, $\forall s$, versus $H_0:F_A(s)>F_B(s)$, $\forall s$ or with covariates $H_0:F_{{\mathbf{e}}_A}(s)=F_{{\mathbf{e}}_B}(s)$, $\forall s$, versus $H_0:F_{{\mathbf{e}}_A}(s)>F_{{\mathbf{e}}_B}(s)$, $\forall s$. We aim to test $H_0$ at a desired type I error rate $\alpha$. A critical value $C$ for testing $H_0$ can be determined such that

$$P_{H_0}\left(T^{*}\ge C\right)\le \alpha .$$

Choose $C$ so that the desired significance level is as close to $\alpha$ as possible. Assuming no ties among the $T^{*}$’s, the significance level is given by $\left[\alpha \left(\genfrac{}{}{0pt}{}{N}{n}\right)\right]/\left(\genfrac{}{}{0pt}{}{N}{n}\right)$, where $[\cdot ]$ denotes the integer part. In the case of ties between $T^{*}$’s near $C$, the significance level may be slightly lower than $\left[\alpha \left(\genfrac{}{}{0pt}{}{N}{n}\right)\right]/\left(\genfrac{}{}{0pt}{}{N}{n}\right)$.

Comment on Blocking. If a blocked randomization plan was utilized the only modification to the randomization test is to re-randomize the treatment assignments within block.

2.3. Adjusting for Prognostic Variables: An Alternative or Enhancement to Stratified Randomization

We argue that in the time-to-event framework, the randomization test, which controls for prognostic factors, both continuous and discrete, should replace classic stratification approaches or be used in combination with stratification. The advantages of including prognostic covariates in the context of a randomization test over classic stratification approaches include: 1) Increased statistical efficiency, achieved by using a continuous prognostic factor rather than discretizing it in the stratified approach; 2) Flexibility in the number of prognostic variables that can be included in the randomization test approach compared to stratification, where the number is limited by the regression framework, applicable to both small and large sample sizes; 3) Absence of issues with correlated prognostic variables in the randomization test approach, as they only estimate nuisance parameters in the model, whereas in the stratified randomization framework, correlated prognostic variables increase the likelihood of sparse or empty strata; 4) Simplified logistics, where a simple blocked randomization scheme suffices for implementing the randomization test approach; and 5) Even if stratified randomization is implemented, the randomization approach provides valid and efficient inference using the stratification variables in the regression model of the UMR’s. The advantage of this approach will be examined in our simulation study in Section 3.

2.4. Minimization Approach to Treatment Assignments

In randomized clinical trials, minimizing imbalances between treatment groups across baseline values can be achieved through minimization, an alternative to stratification. Despite its academic interest, minimization is not commonly employed, as studies have indicated (Taves et al., [29]; Sella et al., [30]). Furthermore, the straightforward hypothesis test about treatment effect is typically conservative when using minimization (Li et al., [31]). However, the randomization test approach can be effectively utilized within the minimization framework to approximately control Type I errors at the desired rate and increase the statistical efficiency of the test $H_0:F_{{\mathbf{e}}_A}(s)=F_{{\mathbf{e}}_B}(s)$, $\forall s$. Similar to stratified randomization, the two approaches—randomization test and minimization—can be combined should practical or logistical considerations arise. The minimization approach will be examined in our simulation study in Section 3.

3. Simulation Study

For our simulation study, we will utilize three families of accelerated life models: Log-Normal, Log-Logistic, and the Weibull distribution. Notably, the Weibull distribution also falls within the proportional hazards distribution.

In general, we define $F_0(\cdot )$ as an arbitrary absolutely continuous standardized distribution function from a location-scale family with location parameter $\theta =0$ and scale parameter $\sigma =1$. The distribution functions for survival time $T$ and $X$, where $X=log(T)$ are defined in terms of a common $F_0$ as follows:

$$F_X\left(x\right)=F_0\left(\frac{x-\theta }{\sigma }\right),$$ (3.1)

$$F_T\left(t\right)=F_0\left(\frac{\text{log}\left(t\right)-\theta }{\sigma }\right),$$ (3.2)

Consequently, the probability density functions for $X$ and $T$ are:

$$f_X(x)=\frac{1}{\sigma }{f}_0\left(\frac{x-\theta }{\sigma }\right),$$ (3.3)

$$f_T(t)=\frac{1}{\sigma t}{f}_0\left(\frac{\text{log}(t)-\theta }{\sigma }\right),$$ (3.4)

The quantile functions for $X$ and $T$ are:

$$Q_X\left(u\right)=\theta +\sigma Q_0\left(u\right),$$ (3.5)

$$Q_T\left(u\right)=\text{exp}\left(\theta +\sigma Q_0\left(u\right)\right),$$ (3.6)

respectively.

Additionally, we denote the $p\times 1$ vector ${\mathbf{z}}^{\prime }={\left(1,z_1,z_2,\cdots ,z_p\right)}^{\prime }$ to represent a set of possible $p-1$ covariates or prognostic factors. The usual extension of the accelerated life model allows $\theta$ to vary as a function of $\mathbf{z}$, where the location parameter $\mu ={\beta }^{\prime }\mathbf{z}$ with regression coefficients ${\beta }^{\prime }={\left({\beta }_0,{\beta }_1,{\beta }_2,\cdots ,{\beta }_p\right)}^{\prime }$.

This yields another quantity of interest known as the acceleration factor, given by $\text{exp}\left({\beta }^{\prime }\mathbf{z}\right)$. The acceleration factor can be interpreted as a shift in the time scale as a function of the prognostic factors. In our simulation study, the treatment indicator $z_1$ will be 0 for treatment A and 1 for treatment B. Given a randomized experiment, the acceleration factor will differ between treatments A and B only if there is a true treatment effect, i.e., ${\beta }_1\ne 0$.

The most common choices for $F_0$, as included in standard statistical software packages, correspond to the standard normal, standard logistic, and extreme value distributions. The forms for $F_0,f_0$, and $Q_0$ are provided in Table 1. It is well-known that these transformed values correspond to the log-normal, log-logistic, and Weibull distributions, respectively.

Table 1:

Common Location-Scale Models used in Life Testing

Distribution	$F_0\left(x\right)$	$f_0\left(x\right)$	$Q_0\left(u\right)$
Normal	$\Phi \left(x\right)$	$\text{exp}\left(-x^2/2\right)/\sqrt{2\pi }$	${\Phi }^{-1}\left(u\right)$
Logisitic	$1/\left(1+\text{exp}\left(-x\right)\right)$	$\text{exp}\left(x\right)/{\left(1+\text{exp}\left(x\right)\right)}^2$	$\text{log}\left(u/\left(1-u\right)\right)$
Extreme Value	$1-\text{exp}\left(-\text{exp}\left(x\right)\right)$	$\text{exp}\left(-\text{exp}\left(x\right)+x\right)$	$\text{log}\left(-\text{log}\left(1-u\right)\right)$

There are countless simulation scenarios that demonstrate the utility of the UMR approach in conjunction with the Rosenbaum randomization test [20], consistently leading to similar conclusions. For this study, we employ an exponential censoring distribution, $\theta \text{exp}(-\theta c)$, to simulate the censoring mechanism. Regarding inference, we fit a linear regression model of the UMRs on the prognostic factors $z_2,z_3,\cdots ,z_p$ as part of the Rosenbaum component of the test, generating residuals for the Wilcoxon rank-sum test and randomization $t$-test. The prognostic factors are simulated from independent standard normal distributions. For all simulations, we set ${\beta }_0=0.5$ and $\sigma =1$, with total sample sizes ranging from $N=40$ to $N=80$, equally divided between the two treatment arms. P-values are obtained using the exact Wilcoxon rank-sum and $t$ tests, approximated through Monte Carlo simulations with 500 resamples. Additional simulation scenarios based on alternative censoring and failure time distributions yield power results consistent with those reported here.

We will compare our proposed test with the log-rank test, a non-parametric statistical method commonly used in oncology studies to compare the survival distributions of two independent groups when the data are right-censored. Additionally, the Cox regression model will be included in all simulation results. The log-rank test evaluates whether there is a statistically significant difference in time-to-event outcomes (e.g., survival time) between groups by comparing observed and expected event counts at each event time point [24]. It is particularly effective for analyzing time-to-event data when survival curves do not cross and the proportional hazards assumption holds—that is, the ratio of event rates between groups remains constant over time.

Depending on the scenario, we compare our approach with the standard log-rank test, the log-rank test adjusted for stratification variables, and the Cox regression model, which includes an indicator variable for treatment effect as well as additional covariates based on the simulation scenario. Each simulation is conducted with 1,000 Monte Carlo resamples. Throughout, we label the log-rank tests as LR and our proposed Rosenbaum test based on residuals as UMR.

3.1. One prognostic covariate

In this simulation setting, the censoring distribution is set as $C~\text{exponential}(5)$, and we incorporate one prognostic factor, $z_2$, for log-normal, log-logistic, and Weibull failure time distributions. The regression coefficient corresponding to the treatment effect ranges from ${\beta }_1=0$ to 3 in increments of 1, while the regression coefficient for the covariate is set to ${\beta }_2=0,2,4$. The resulting power curves are displayed in Figures 1–3.

Figure 1:

Power curves for the log-normal distribution given a single covariate adjustment.

Figure 3:

Power curves for the weibull distribution given a single covariate adjustment.

A key observation is that the results remain consistent across the different failure time distributions. As expected, the Rosenbaum test demonstrates increasing power relative to the log-rank test as the association between $z_2$ and failure time strengthens, with no significant difference between the two tests when ${\beta }_2=0$. When ${\beta }_2=2$, the power gains for the Rosenbaum test range from approximately 0.15 to 0.2 when its power is near 0.80. These gains become more pronounced when ${\beta }_2=4$, ranging from approximately 0.35 to 0.4, which is substantial.

An interesting yet undesirable property of the Cox proportional hazards model emerges at $n=40$, where the power curves exhibit non-monotonic behavior. This issue arises when the numerical methods converge but lead to a dramatic overestimation of the standard error of ${\beta }_1$, the treatment effect parameter. While Figures 1–3 may suggest that the Cox model has a power advantage, it is crucial to note that its Type I error rate can be substantially inflated beyond the nominal $\alpha =0.05$ level. For instance, in certain scenarios, the Type I error exceeds 0.095. Since there is no way to determine a priori whether the standard error estimate for ${\hat{\beta }}_1$ is biased or whether the Type I error will be highly inflated, inference based on the Cox model in these cases becomes unreliable.

3.2. Multiple prognostic covariates added

In this setting, we compared the log-rank test, the Rosenbaum test (Wilcoxon and $t$), and the Cox proportional hazards model, incorporating up to three prognostic factors ($z_2,z_3,z_4$) for values of ${\beta }_2,{\beta }_3,{\beta }_4=0,2$ under a Weibull model with $C~\text{exp}(5)$. Table 2 presents the power estimates, ranging from the scenario with no covariates (${\beta }_4=0,{\beta }_3=0,{\beta }_2=0,{\beta }_1=0$) to a fully adjusted model (${\beta }_4=2,{\beta }_3=2,{\beta }_2=2,{\beta }_1=2$), with various intermediate combinations also included. The sample size is set to $N=80$, equally divided between the two treatment arms.

Table 2:

Power Estimates and Type I error control for the log-rank test versu, randomization test (Wilcoxon and $t$), Cox proportional hazards model with varying number of prognostic factors.

Power
${\beta }_4$	${\beta }_3$	${\beta }_2$	${\beta }_1$	LR	UMR-W	UMR-T	Cox PH
0	0	0	0	0.051	0.039	0.040	0.042
0	0	2	0	0.054	0.056	0.058	0.065
2	0	0	0	0.051	0.055	0.051	0.052
0	2	0	0	0.053	0.059	0.063	0.062
2	0	2	0	0.048	0.035	0.033	0.046
2	2	0	0	0.057	0.048	0.049	0.061
0	2	2	0	0.047	0.069	0.072	0.046
2	2	2	0	0.041	0.043	0.043	0.053
0	0	0	2	1.000	1.000	1.000	1.000
0	0	2	2	0.869	0.974	0.978	1.000
2	0	0	2	0.865	0.981	0.984	0.852
0	2	0	2	0.864	0.984	0.930	0.857
2	0	2	2	0.662	0.941	0.986	0.843
2	2	0	2	0.645	0.921	0.944	0.833
0	2	2	2	0.650	0.931	0.934	0.653
2	2	2	2	0.536	0.868	0.876	0.630

Incorporating prognostic factors in the simulation model while holding $\sigma$ constant increases the variability of the time-to-event endpoint, leading to the lowest power for all tests in the setting where ${\beta }_4=2,{\beta }_3=2,{\beta }_2=2,{\beta }_1=2$. However, as shown in Table 2, substantial power gains can be achieved using the Rosenbaum randomization test compared to the log-rank test and the Cox proportional hazards model, particularly when multiple prognostic factors that are moderately correlated with the outcome variable are included in the model.

3.3. Stratification scenarios

In this setting, we compared the log-rank test, the Rosenbaum test (Wilcoxon and $t$), and the Cox proportional hazards model. The variable $z_2$ was dichotomized at 0 to form the stratification variable, while it was treated as a continuous prognostic variable in the Rosenbaum test for ${\beta }_2=0,2,4$ under a Weibull model with $C~\text{exponential}(5)$. The regression coefficient corresponding to the treatment effect ranged from ${\beta }_1=0\text{to}3$ in increments of 1. Figure 4 presents the power estimates for each test.

Figure 4:

Power curves for one stratification factor for LR test and Cox proportional hazards model and one prognostic factor for the Rosenbaum test (Wilcoxon and $t$) given a Weibull distribution.

As shown in the figure, the power curves for the log-rank and Rosenbaum tests closely align, except in regions where the Cox proportional hazards model exhibits poor performance for small sample sizes. Once again, we observe the non-monotonic behavior of the Cox model, which stems from biased estimates of the standard error for ${\hat{\beta }}_1$, leading to unreliable inference in these cases.

In the next scenario, we compared the same three tests—log-rank, Rosenbaum (Wilcoxon and $t$), and Cox proportional hazards—while incorporating one stratification variable and an additional prognostic variable. Here, $z_2$ was again dichotomized at 0 to form the stratification variable, while $z_3$ was used as a continuous prognostic variable in the Rosenbaum test for ${\beta }_2=2,{\beta }_3=0,2,4$ under a Weibull model with $C~\text{exp}(5)$. Notably, for the Rosenbaum test, re-randomization was performed separately within each stratum, preserving the original randomization scheme.

Figures 5–7 illustrate the power curves, demonstrating proper Type I error control and improved power performance for the Rosenbaum test (Wilcoxon and $t$) compared to the log-rank test and the Cox proportional hazards model. Furthermore, additional power improvements could be achieved by incorporating more prognostic variables into the model, as evidenced by the non-stratified comparison in Table 2.

Figure 5:

Power curves for one stratification factor for LR test and Cox proportional hazards model and additional prognostic factors for the Rosenbaum test (Wilcoxon and $t$) given a Weibull distribution.

Figure 7:

Power curves for one stratification factor for LR test and Cox proportional hazards model and additional prognostic factors for the Rosenbaum test (Wilcoxon and $t$) given a Weibull distribution.

3.4. Minimization type I error

Given that minimization destroys the properties of statistical inference we can no longer guarantee an exact test based on the Rosenbaum method. However, we show that we can approach the exact Type I error levels when incorporating the prognostic variables utilized in the minimization process. We utilized the method of Pocock and Simon[33] and set the probability $p=1$ such that the allocation was deterministic after the 11th subject. In this setting, we set the censoring distribution as $C~\text{exponential}(5)$ and incorporate two prognostic factors $z_2$ and $z_3$ for Weibull failure time distributions. The covariates $z_2$ and $z_3$ are independent and follow standard normal distributions. The sample size ranged from $n=40$ to $n=80$. The regression coefficients were set to values ${\beta }_1=0,1,2,3,{\beta }_2=2$ and ${\beta }_3=0,2,4$. All other settings were identical as the simulations above. Figure 8 shows the standard LR test is overly conservative. The minimization was based on both $z_2$ and $z_3$ after being dichotomized at 0. When ${\beta }_3=2$, the type I error rates are even below 0.01. On the other hand, the UMR achieves satisfactory type I error control across all settings because it relies on a regression model which adjusts for the covariates. Correpsondingly, the difference in statistical power between LR and UMR tests is even larger. For example, when ${\beta }_1=2,{\beta }_3=4$, the UMR even shows 50% higher power than the LR test.

Figure 8:

Type I error rate of LR and UMR with Weibull failure time and exponential censoring distributions under minimization based on two covariates.

4. Example

For our example data, we utilized publicly available information from Project Data Sphere (https://data.projectdatasphere.org/). Specifically, we examined a randomized phase II clinical trial [32] that evaluated the efficacy and safety of LY2510924 (LY) added to first-line standard of care (SOC) chemotherapy for extensive-disease small cell lung cancer (ED-SCLC). The primary efficacy endpoint was progression-free survival (PFS), while a key secondary endpoint was overall survival (OS). In our examples, we performed 100,000 Monte Carlo resamples to estimate the values for both the exact Wilcoxon rank-sum test and the exact randomization $t$-test, ensuring that the approximate p-values closely matched the true p-values.

Median PFS (95% confidence interval [CI]) was 5.88 (4.83, 6.24) months for LY + SOC=Arm A versus 5.85 (4.63, 5.51) months for SOC=Arm B[32]. Median OS (95% CI) was 9.72 (6.64, 11.70) months for Arm A versus 11.14 (8.25, 13.44) months for Arm B[32]. The sample sizes for each arm were $n_A=47$ and $n_B=42$, respectively.

Given the minimal difference in the PFS endpoint, we are focusing on the OS endpoint to illustrate our new approach. For comparison, the mean UMR per arm and the corresponding standard deviation for the PFS endpoint were ${\bar{\mu }}_A=-0.007,s_{{\mu }_A}=0.81$ and ${\bar{\mu }}_B=0.008,s_{{\mu }_B}=0.91$, which is consistent with the median PFS times.

One limitation of this example, as compared to the original work, is that the publicly available data set only contained information for one of the stratification variables, namely ECOG status (0, 1 versus 2), and not the other stratification factor based on serum lactate dehydrogenase levels. This example also illustrates the pitfalls of stratification, where only five subjects were in the ECOG=2 strata. Hence, it is unclear how an additional stratification factor could be included in the final analysis.

Regarding overall survival (OS), the stratified log-rank test based on the ECOG status strata yielded a two-sided p-value of $p=0.085$. The mean UMR per arm and the corresponding standard deviation for the OS endpoint were ${\bar{\mu }}_A=0.130,s_{{\mu }_A}=0.69$ and ${\bar{\mu }}_B=-0.145,s_{{\mu }_B}=0.97$, which aligns with the median OS times. This suggests that Arm A has an average expected number of excess deaths at each observed time point of 0.130, while the control arm has an average expected number of excess deaths at each observed time point of −0.145. This corresponds to the median survival times and the Kaplan-Meier curves displayed in Figure 10. The Monte Carlo approximation to the exact Wilcoxon rank-sum test based on the UMRs, without covariates but accounting for the ECOG stratification factor, yielded a two-sided p-value of $p=0.348$. The Monte Carlo approximation to randomization $t$-test yielded a p-value of $p=0.124$. Although there is a significant difference between the p-values from the log-rank test and the Wilcoxon rank-sum test based on the UMRs, on average they have the same power as indicated in Section 3. Furthermore, it should be noted that the one ECOG strata had small numbers, which is the likely reason for a more discrepant set of p-values for this particular example.

Figure 10:

Kaplan-Meier survival curves for example data with ARM A= LY + SOC=ARM and ARM B= SOC.

Next, we conducted the Rosenbaum test using baseline prognostic factors of age, serum glucose, alkaline phosphatase, serum sodium, and calcium. This resulted in a mean UMR per arm and the corresponding standard deviation for the OS endpoint of ${\bar{e}}_A=0.181,s_{e_A}=0.66$ and ${\bar{e}}_B=-0.204,s_{e_B}=0.85$. The interpretation of the UMR-based residuals indicates an excessive number of expected deaths in Arm A and a decreased number of deaths in Arm B at any given point in time. The Monte Carlo approximation to the exact Wilcoxon rank-sum test based on the $e$’s, incorporating these prognostic factors and accounting for the ECOG stratification factor, yielded a two-sided p-value of $p=0.044$. The Monte Carlo approximation to the exact randomization $t$-test yielded a p-value of $p=0.021$. The test for treatment effect from the Cox regression model adjusting for the prognostic factors yielded $p=0.028$.

One cautionary note when using the Rosenbaum approach is that, on average, adding more prognostic covariates will always produce the greatest power gains and will reduce the variance of the grouped residuals produced from the regression model fit for the purpose of carrying out the Rosenbaum test. However, for a given data realization, additional covariates may attenuate the effect size of the treatment difference proportionally smaller to the reduction in variance relative to a reduced model.

For example, suppose we add another prognostic factor to our model, such as total protein. Our results now become ${\bar{e}}_A=0.156,s_{e_A}=0.68$ and ${\bar{e}}_B=-0.176,s_{e_B}=0.79$, which yields a p-value based on the Monte Carlo approximation to the exact Wilcoxon rank-sum test of $p=0.075$. The Monte Carlo approximation to the exact randomization $t$-test yielded a p-value of $p=0.041$. The test for treatment effect from the Cox regression model adjusting for the prognostic factors yielded $p=0.052$.

We observed that the adjustments for prognostic covariates reduced the variance of the UMRs. In this instance, the choice of prognostic factors, except for age, was arbitrary and used to illustrate the new methodology. This phenomenon is a general issue with the Rosenbaum approach and is not specific to our use of UMRs. In practice, the set of prognostic factors needs to be specified in the clinical trial protocol. A global test statistic is being developed to protect against the possibility that a subset of prognostic variables produces a significant result at the desired $\alpha$ level compared to using the maximum number of prognostic variables.

5. Summary

In this work, we introduce a new metric, termed the univariate martingale residual, which enables the application of Rosenbaum’s exact testing method to improve the efficiency of statistical tests for treatment effects by incorporating prognostic baseline covariates, in particular in conjunction with the exact randomization $t$-test. The univariate martingale residual is straightforward to interpret and aligns well with other measures such as the median time-to-event. Utilizing this Rosenbaum-martingale residual testing approach in oncology trials can reduce costs, shorten trial durations, and potentially make randomization feasible in rare disease clinical trials. Additionally, we demonstrated that the univariate martingale residual can be used in inference procedures based on minimization, with type I error approximately controlled through re-randomization of treatment assignments according to the minimization criteria.

A key takeaway is that stratification introduces additional logistical challenges for generating the randomization scheme compared to using a continuous prognostic adjustment with the Rosenbaum test. Moreover, multiple prognostic factors can be incorporated in small sample sizes with the Rosenbaum approach, whereas the stratification method is constrained by sample size limitations. There is also a non-zero probability that multi-strata studies may include strata with no enrolled subjects. However, stratification can still be applied within the framework of the Rosenbaum test, which may be particularly advantageous in practical scenarios, such as using clinical site as a stratification factor.

The Rosenbaum method, endorsed by the FDA guidance document [19], advocates for using prognostic variables to enhance clinical trial testing efficiency. An emerging strategy in the academic literature is the integration of supplementary prognostic information to improve clinical trial efficiency. While using supplementary data to refine mean estimates in finite sampling has been long established in survey sampling literature [21], its application in clinical trials is recent [10, 22, 23]. The supplementary variable approach can be used with stratification and minimization treatment assignment methods. Unlike the Rosenbaum approach, it accommodates interactions between treatment and prognostic covariates. However, this approach relies on large sample approximations, is currently not a component of the FDA guidelines and is less efficient in two-group balanced designs [10], the most common design in cancer clinical trials.

Future work will involve exploring more complex nonlinear models to enhance statistical efficiency further [34], and investigating testing approaches that account for both the model and all subset models. This addresses the issue that, in the Rosenbaum approach, a subset of prognostic variables may yield a significant test while the full model may not, despite the full model being typically the most statistically efficient. A comparative study of these methods would also be worthwhile.

In conclusion, we emphasize the need for robust, exact, and powerful statistical methods to ensure unbiased treatment efficacy assessments in oncology trials.

Figure 2:

Power curves for the log-logistic distribution given a single covariate adjustment.

Figure 6:

Power curves for one stratification factor for LR test and Cox proportional hazards model and additional prognostic factors for the Rosenbaum test (Wilcoxon and $t$) given a Weibull distribution.

Figure 9:

Statistical power of LR and UMR with Weibull failure time and exponential censoring distributions under minimization based on two covariates.