Two-Sample Survival Probability Curves: A Graphical Approach for the Analysis of Time to Event Data in Clinical Trials

Abstract

With the aim to improve the communication of trial results, we introduce a novel graphical approach that complements the analysis of time to event outcomes in two-arm randomized trials. We define the so-called two-sample survival probability curve and propose a nonparametric estimator of the curve based on a random walk using Kaplan-Meier survival estimates for the two arms. We then use the estimated curve to visualize treatment effect as well as potential effect modification of factors of interest. We also propose to estimate two-sample survival probability curves within the framework of the Cox model to graphically assess model fit. The proposed two-sample survival probability plot puts trials in a standardized [0,1] × [0,1] space, allowing for a simple visualization of the main effect, effect modification, and the adequacy of a model fit.

1. Introduction

Time from randomization until the occurrence of a clinically relevant event is the primary interest in many clinical trials, but communication of the associated results remains a challenge [1, 2]. Standard analyses of time to event outcomes in two-arm trials include using Kaplan-Meier curves to visually check for differences in survival probabilities between arms followed by the log-rank test and effect estimation, such as hazard ratio estimation [1, 3]. In addition, one may use regression models such as Cox regression [4] for additional predetermined analyses [1, 3]. Basic survival analysis training may be needed for users to understand statistical concepts such as hazard ratios. In a study of the interpretation of time to event trial results, Weir et al. (2019) found misinterpretation of the hazard ratio when reading abstracts common among medical professionals despite their level of epidemiology/biostatistics training (misinterpretation occurred among 49% of professionals without this training, 54% of those with a non-degree course completed, and 40% of those with a degree in either field) [5]. Misinterpretation of time to event trial results could affect treatment selection, communication with patients, and policy or funding decisions. These issues can be addressed by providing multiple measures [2, 5, 6] and depictions of the treatment effect. With this aim in mind, we develop a graphical approach that synthesizes two Kaplan-Meier curves into one curve, the so-called two-sample survival probability curve for the comparison of treatment effect between arms.

We show that our proposed curve and the receiver operating characteristic (ROC) curve have similar structures, but their axes differ in meaning. This new curve supplements existing survival analysis tools by allowing the visualization of the treatment effect in a single curve that has a standardized [0,1] × [0,1] space, and the area under the curve (AUC) is equivalent to Efron’s estimate of the concordance between two samples [7, 8]. Several concordance estimators have been developed for censored data [7, 9–12], but this is the first graphical depiction for such a measure.

Graphical methods are frequently used as diagnostic tools in survival analysis. In the case of the Cox model, several tests and graphical methods, e.g., plots of residuals or log cumulative hazards, are utilized for checking the proportional hazards (PH) assumption [13–16]. More recently, Cox and Czanner (2015) proposed a graphical method to assess this assumption based on the divergence between two survival distributions [17]. However, survival analysis training is again needed for understanding these plots, including their shape and axes, and communicating these plots’ clinical meaning can be challenging.

In the following sections, we present how the proposed curves have axes that are easy to understand and how they can be used to depict the treatment effect, asses potential effect modification, and diagnose the model fit for regression models used in either predetermined or post-hoc analyses. We use well-known survival textbook data and data from more recent clinical trials to illustrate the proposed method. This graphical approach is not meant to replace standard survival analysis methods, rather it is to assist with the communication of the results from these methods to non- statisticians.

2. Methods

2.1. The two-sample survival probability curve and its nonparametric estimator

We first review literature that places ROC curves in a survival context and then apply this concept in a randomized trial setting. The ROC curve has been used to determine the optimal cutoff for a biomarker to differentiate diseased (D = 1) from non-diseased (D = 0) groups [18]. For a continuous biomarker Y used to test for the disease D, the ROC curve maps false positive rates (FPR) to true positive rates (TPR), with both the FPR and the TPR having values between 0 and 1 [18,19]. Pepe shows that in this setting a receiver operating characteristic curve is also a map between the survival functions of the diseased and non-diseased groups [19, chap. 4]. Mathematically, this is formulated as

$$\mathrm{ROC}\left[S_0(y)\right]=S_1(y)$$ (1)

where y ∈ [0, ∞) corresponds to values of the biomarker Y, and subjects with Y > y test positive [19, chap. 4]. The FPR used for the horizontal coordinate of the ROC is mathematically equivalent to the survival probability of the marker for the non-diseased population, S₀(y), while the TPR, which provides the corresponding vertical coordinate, is equal to the survival probability for the diseased population S₁(y) [19, chap. 4].

We consider this function in a randomized trial setting where the time to an event is the primary endpoint and is subject to right censoring. Suppose we have N independent and identically distributed (iid) observations, {(X_i, δ_i, Trt_i), i = 1, … , N}, where X_i = min(T_i, C_i) is the observed time, T_i and C_i are subject i’s underlying time to the study endpoint and the right censoring time, respectively, and δ_i = I(T_i ≤ C_i) is the event indicator. For subject i, Trt_i is an indicator such that Trt_i = 1 if the individual was randomized to the treatment arm and zero for the control. We assume that the occurrence of the endpoint is independent of the time of the censoring event, in the sense that f_T(t) = f_T|C(t|C), where f(·) is a probability density function. Considering the survival functions for the two arms, S_k(t) = P(T ≥ t|Trt = k), k = 0,1, one can map the survival function of the control arm S₀(t) into that of the treatment arm S₁(t) using Equation (1) and define a two-sample survival probability curve (R) for the trial as R[S₀(t)] = S₁(t), t ∈ [0, ∞) or equivalently $R(u)=S_1\left[S_0^{-1}(u)\right]$ for u ∈ [0,1], where $S_0^{-1}(\cdot )$ is the inverse function of S₀(·) [19]. A natural nonparametric estimator for this curve, $\widehat{R}(u)$, arises by replacing the survival functions with their Kaplan Meier estimates, ${\widehat{S}}_0(t)$ and ${\widehat{S}}_1(t)$ [7, 8, 12].

Several authors [18, 20–22] have used a random walk approach to construct ROC curves for biomarker samples from diseased and non-diseased groups. We use this technique to draw the two-sample survival probability curve for right-censored survival data using Kaplan-Meier estimates. To simplify the discussion, we let the event be death. Suppose we have a trial with L ordered, unique death times t₍₁₎ < ⋯ < t_(L) being observed from the N participants in the combined treatment and control sample. The proposed curve starts at point (1,1) and then continues in steps either to the left, downward or diagonally depending on whether the death(s) observed at each time t_(l) corresponds to participant(s) from the treatment arm or from the control arm. Specifically, the curve moves to the left if the death(s) at the time of interest time t_(l) occurred only among the control arm participants, downward if only among the treatment arm participants, and diagonal if deaths were observed at this time point in both arms (i.e., ties). The magnitude of each step is determined by the vertical drop in the Kaplan-Meier curve(s) for the corresponding arm(s).

Now, we use a toy example to illustrate the plotting procedure of the proposed method. Suppose that we have a randomized trial with 10 participants in each arm, the treatment arm’s times to death (or censoring) are: 2, 2.5, 3, 4.5, 4.5+, 7, 7+, 7+, 7+, 7+, and the control arm: 1.5, 1.5, 1.5, 2.5, 2.5+, 3, 3, 4.5, 4.5, 4.5, where + denotes censoring times. Then, we have L = 6 unique death times observed from the overall sample: t₍₁₎ = 1.5, t₍₂₎ = 2, t₍₃₎ = 2.5, t₍₄₎ = 3, t₍₅₎ = 4.5, t₍₆₎ = 7. Since three deaths are observed in the control arm at time t₍₁₎, the first move, as shown in Figure 1, is a horizontal step from (1,1) to (0.7,1), where the size of the horizontal step (from 1 to 0.7) is based on the Kaplan-Meier estimate for the control arm. Then, at time t₍₂₎ one participant in the treatment arm dies, so the curve drops vertically to (0.7,0.9), where the size of the vertical step (from 1 to 0.9) is based on the Kaplan-Meier estimate for the treatment arm. Then, at time t₍₃₎ we observe deaths in both arms, so the next step is a diagonal move from (0.7,0.9) to (0.6,0.8), where the step size in the horizontal direction (from 0.7 to 0.6) and that in the vertical direction (from 0.9 to 0.8) are determined by the Kaplan-Meier estimate for the control arm and the treatment arm, respectively. Continue this procedure until all 6 observed unique event times have been plotted.

Figure 1:

Illustration of the plotting procedure of the proposed two-sample survival probability curve with a toy example. The left panel shows the data (top) and the working table of the Kaplan-Meier estimate (number at risk, number of events, number of censored, and survival probability at each unique, observed event time) for the control arm (middle) and the treatment arm (bottom); the right panel is the two-sample survival probability curve with the solid line and arrows showing the direction of the random walk and the dashed lines and arrows showing the direction of the treatment effect.

We know that when using ROC curves to study classifiers, the closer to the point (0,1) the ROC curve is, the better the classifier predicts each class [19, chap. 4]. This property is similar in the current setting. In the case when the estimated two-sample survival probability curve meets the (0,1) point, all deaths in the control arm occurred before any death occurred in the treatment arm and the treatment is hence unequivocally better. As displayed in Figure 1, the closer the curve is to the top left corner, the more benefit the treatment provides compared to the control. Curves that remain close to the diagonal line show no benefit and those below the diagonal line show the treatment under study is not as effective as the control. The curves are depicted in the [0,1] × [0,1] space across different trials, so a common visual reference for the null effect (the diagonal line) can be used despite possible differences in follow-up time and censoring rate.

2.2. Illustrations

We present the proposed curves for three trials in Figure 2. Panel (a) displays the Kaplan Meier curves and a two-sample survival probability curve for a well known 52-week trial of 42 leukemia patients in remission after chemotherapy who were randomized to receive either placebo or 6-mercaptopurine (6-MP) [4, 23]. One can see that 6-MP is superior to the control as its two-sample survival probability curve is far from the diagonal line.

Figure 2:

Kaplan-Meier curves with corresponding two-sample survival probability curves for a trial com-paring 6-MP with placebo (panel a), the peripheral-blood stem cells versus bone marrow transplantation from unrelated donors trial (panel b), and a radiation and 5-Fluorouracil (RoRx+5-Fu) versus control therapy trial (panel c). The two-sample survival probability curves at the bottom of each panel contain a diagonal line that represents a null effect.

The bottom of panel (b) shows the two-sample survival probability curve for a trial where participants were randomized to either peripheral-blood stem-cells or bone marrow transplantation from unrelated donors [24]. The curve shows no difference between the two arms as it remains close to the diagonal.

Finally, the bottom of panel (c) shows the estimated two-sample survival probability curve for a trial comparing the use of radiation and 5-Fluorouracil (RoRx+5-Fu) and a control therapy for participants with bile duct cancer [25]. Visual inspection shows that the combined treatment had varying effects on survival as compared to the control therapy. When the survival probability for the control arm and the treatment arm was 36% or higher (around 250 days post-treatment), the combined radiation and RoRx+5-Fu therapy showed benefit compared to the control therapy. Beyond that time point, the two arms’ survival probabilities cross over, and hence the two-sample survival probability curve crosses the diagonal line, indicating that the control therapy started to show benefit over the combined therapy.

2.3. The two-sample survival curve and concordance probability

We note that the numerical integration of the nonparametric estimate of the two-sample survival curve leads to Efron’s concordance estimate, $\text{AUC}=\widehat{P}\left(T_1>T_0\right)$, i.e., Efron’s estimate of the probability that the time to event for a randomly chosen observation among those in the treatment arm is longer than that for the counterpart in the control arm [7, 8, 12, 19]. Various authors [7, 8,12] have discussed the instability of Efron’s concordance estimator in the presence of censoring. Brentnall and Cuzick (2018) describe how Efron’s concordance becomes highly unstable if at the maximum study time the survival functions for both arms are larger than zero [12]. The current graphical approach allows for a visualization of this problem.

For the 6-MP trial we estimate that $\widehat{P}\left(T_1>T_0\right)=0.84$, indicating a treatment benefit as expected from the shape and position of the curve. However, for the trial in panel (b), the estimate is $\widehat{P}\left(T_1>T_0\right)=0.389$, wrongly pointing to a benefit from bone marrow over peripheral-blood stem cell transplantation despite all graphical evidence indicating that the two treatments had similar effects and P(T₁ > T₀) should be close to 0.5. The reason is that the curve for the trial in (b) is estimable only in the $[\mathfrak{s},1]\times [R(\mathfrak{s}),1]$ space; $0<\mathfrak{s}<1$, rather than the full space of [0,1] × [0,1] ($\mathfrak{s}$ is the minimum value of the survival function depicted on the horizontal axis of the plot, and for this trial, $\mathfrak{s}=0.494$ and $R(\mathfrak{s})=0.529$). Thus, we obtain a gross underestimate if we ignore the censoring problem. The naive AUC estimate reflects the estimator’s instability in the presence of censoring as illustrated in this example. Finally, the RoRx+5-Fu trial has a curve estimable in $[\mathfrak{s},1]$ with $\widehat{P}\left(T_1>T_0\right)=0.599$. The estimate of 0.599 reflects the averaging of the effect during periods of treatment benefit and lack thereof, but is still above 0.5 as the curve mostly remains above the diagonal. We are less concerned with bias in this case as $\mathfrak{s}\text{=}0\text{.}04$ and $R(\mathfrak{s})=0$.

For trials where $\mathfrak{s}$ and $R(\mathfrak{s})$ are far from 0, such as the one in panel (b), this graphical depiction indicates a need to consider an alternate concordance estimator. Some nonparametric options include those proposed by Harrel et al. [9, 10] and Uno et al. [11], or Brentnall and Cuzick’s modification [12] for Efron’s concordance.

2.4. Effect Modification

The proposed nonparametric two-sample survival probability curves can also be used in graphical evaluations of potential effect modifiers. As an example, we use data from the peripheral-blood stem cells versus bone marrow transplantation from unrelated donors trial for a post-hoc study of the possibility of effect modification by sex. A standard graphical approach for this purpose would begin by laying out four Kaplan-Meier curves for each possible sex and treatment combination. However, this approach becomes increasingly difficult for modifiers with more levels, so instead, we synthesize the subgroup Kaplan-Meier curves into two-sample survival probability curves as shown in Figure 3. This reduces the dimension of the problem by half. For instance, with the above data example, one can more easily see that the estimated two-sample survival probability curves for males and females largely overlap, and thus we expect that sex leads to only minimal differences in the treatment effect.

Figure 3:

Investigation of sex as an effect modifier for the peripheral-blood stem cells versus bone marrow transplantation from unrelated donors trial.

3. Cox Model Based Two-Sample Survival Probability Curves and a Diagnostic Plot for Model Fit

Let λ be the hazard ratio between treatment and control such that λ < 1 indicates treatment benefit. A Cox model based estimator for the two-sample survival probability curve is formulated as $\tilde{R}(u;\lambda )=u^{\lambda }$ where u ∈ [0,1] so that the area under the curve is $\tilde{P}\left(T_1>T_0\right)=\frac{1/\lambda }{1+1/\lambda }$ [12, 26]. In the literature, it is also shown how 1/λ is the odds of concordance [27], and hence, after exponentiation, the regression coefficient of an interaction term between the treatment and a binary covariate in a Cox model can be interpreted as an odds ratio. Note that we may again have an incomplete curve if all the u and $\tilde{R}(u)$ values used to depict the curve are larger than zero. For instance, in the trial in Figure 2 panel (b), estimates of u at two years using Breslow or Kaplan-Meier methods are higher than zero, as are the corresponding values of $\tilde{R}(u)$, making estimates from numerical integration biased unless we allow for extrapolated values beyond two years. That is unnecessary, and instead we recommend the use of Gönen and Heller’s [26] concordance probability estimator, which is robust in the presence of censoring under the PH assumption [12, 26].

One criticism of the hazard ratio is that it becomes less relevant when proportionality is not met [2, 27]. This problem remains under the AUC framework described above and makes interpretations of the conventional hazard ratio as an odds (or the exponentiated interaction term as an odds ratio) invalid. In response, we propose the use of two-sample probability curves as an additional graphical diagnostic tool to check the PH assumption. We first estimate a Cox-based two-sample survival probability curve using the estimated hazard ratio $(\widehat{\lambda })$ and then superimpose it onto the nonparametric estimate of the curve, $\widehat{R}(u)$. If the Cox model fits the data well then the two curves $\tilde{R}(u;\widehat{\lambda })$ and $\widehat{R}(u)$ would largely overlap. Figure 4 shows the superimposed curves for the three previously discussed trials. Figures 4a and 4b show that the Cox model based curves fit well compared to the nonparametric curves for the 6-MP vs. placebo and the peripheral-blood stem cells vs. bone marrow transplantation from unrelated donors trials. However, for the radiation and 5-Fluorouracil versus control trial, in Figure 4c, the Cox model based curve and the nonparametric curve are largely deviated from each other. This reflects a violation of the PH assumption and indicates that other models or modifications of Cox regression should be considered. These visualizations are consistent with the results of a commonly used test developed by [15] using weighted residuals, based on which we failed to reject the PH assumption for the first two trials at 0.05 significance level, but rejected the PH assumption for the 5-Fluorouracil versus control trial (p-values = 0.800, 0.954 and 0.024, respectively).

Figure 4:

Two-sample survival probability curves based on Kaplan-Meier estimates and the Cox model for a trial comparing 6-MP with placebo (panel a), the peripheral-blood stem cells versus bone marrow transplantation from unrelated donors trial (panel b), and the radiation and 5-Fluorouracil versus controltrial (panel c).

Additionally, we compared our proposed method with the Schoenfield residuals plot [28] to graphically check the PH assumption for the radiation and 5-Fluorouracil vs. control trial. The top panel of Figure 5 displays Schoenfield residual plots for the simple Cox model, h(t|Trt) = h₀(t)exp(βTrt) in panel (a) and for a change-point Cox model with the hazard function, h(t|Trt) = h₀(t)exp(β₁Trt) for t ≤ 250, and h₀(t)exp{(β₁ + β₂)Trt} for t > 250 in panel (b). We note that the scales for the two residual plots differ, which makes a direct comparison between the two residual plots less straightforward. Nevertheless, we find the residuals and the lowess fit for the residuals show very similar pattern between the two Cox models. However, when we compare the two-sample survival probability curve for the two Cox models (Fig. 4 c vs. Fig. 5 c), we can see that the two-sample survival probability curve for the change-point model has considerably more overlap with the nonparametric curve than the simple Cox model, indicating that the change-point model has a better fit than the simple model. This finding is consistent with the weighted residual test result (p-value = 0.024 vs. 0.316 for the simple vs. the change-point Cox model). This indicates that, though useful, direct comparisons of Schoenfield residual plots across models can be less sensitive than our proposed method. However, in practice, Schoenfield residuals continue to be one of the commonly used visualizations for this assumption so we included this comparison. More options for graphical assessment of the PH assumption can be found in [13–17].

Figure 5:

Schoenfield residual plots for the simple Cox model (panel a) and the change-point Cox model (panel b) used for analyzing the data from the 5-Fluorouracil vs. control trial and the two-sample survival probability curve for the change-point Cox model (panel c).

4. Discussion

We have introduced a graphical method to complement the analysis of time to event outcomes in two-arm clinical trials. The proposed two-sample survival probability plot puts trials in a standardized [0,1] × [0,1] space so that one simple graph can display the main effect, effect modification, and the adequacy of a model fit, allowing statisticians to simplify visualizations and use this tool for communication with broader audiences. The proposed curves retain various desirable properties from ROC curves, such as the comparability to the diagonal line. In addition, the relationship between the distance of the two-sample survival probability curve and the diagonal with the Kolmogorov-Smirnov test as outlined by Pepe (2003) remains [19, 25]. Furthermore, this distance is also related to a newer test for comparing two survival distributions proposed by Lin and Xu in 2009 [29].

In this paper, we focus on the introduction of this graphical framework while acknowledging that the AUC estimates based on numerical integration of either the nonparametric or semiparametric curves are biased when there is heavy or administrative censoring. In such circumstances, if concordance estimates are desired we recommend using other procedures for estimation such as those proposed Harrel et al. [9], Uno et al. [11], and Brentnall and Cuzick [12], or, when assuming proportional hazards, Gönen and Harrel’s [26] concordance.

Overall, two-sample survival probability curves show promises as a complement to current practice in survival analysis. These also open up multiple directions for future research. For instance, there are a number of alternative models and model adjustments that can be used when the Cox model does not fit the data [13]. The graphical diagnosis for the fit that we discussed for the Cox model can be extended to other regression models if the proposed curves can be constructed, for example, when the survival function of the two arms can be estimated individually or their functional relationship can be derived based on an assumed regression model. Thus, these plots could be used as a supplemental graphical tool for model selection or diagnosis. One may also accompany the diagnostic plots with measures of the closeness between the nonparametric curve and model-based curve. Two ad-hoc measures may be proposed: the sum of squared residuals $(\text{SSR}={\{\widehat{R}(u)-\tilde{R}(u;\widehat{\lambda })\}}^2)$ and the area between the two curves (ABTC). If a model, such as Cox model led to a perfect fit, these measures would be zero. Further study of the SSR and ABTC, especially in the presence of heavy or administrative censoring is warranted.

Another possible future research direction could be creating two-sample survival probability curves depicting the results of different trials for the same therapy by using published Kaplan-Meier curves and digitizing software without requiring individual trial data and synthesizing individual study curves to one overall curve in the context of a meta-analysis. Additional research directions include how to construct these curves under other censoring mechanisms, such as interval or double censoring, or for competing risks data. Finally, the current approach could be extended to observational studies by using causal inference techniques such as inverse probability weighting (IPW) and the related IPW or Adjusted Kaplan-Meier estimator [30–32] and IPW Cox model [33].