The Kaplan-Meier Estimator as an Inverse-Probability-of-Censoring Weighted Average

Abstract

The Kaplan-Meier (product-limit) estimator of the survival function of randomly-censored time-to-event data is a central quantity in survival analysis. It is usually introduced as a nonparametric maximum likelihood estimator, or else as the output of an imputation scheme for censored observations such as redistribute-to-the-right or self-consistency. Following recent work by Robins and Rotnitzky, we show that the Kaplan-Meier estimator can also be represented as a weighted average of identically distributed terms, where the weights are related to the survival function of censoring times. We give two demonstrations of this representation; the first assumes a Kaplan-Meier form for the censoring time survival function, the second estimates the survival functions of failure and censoring times simultaneously and can be developed without prior introduction to the Kaplan-Meier estimator.

1. Introduction

The Kaplan-Meier (product-limit) estimator for the survival function of randomly-censored time-to-event data (Kaplan and Meier, 1958) is often introduced as the maximizer of a nonparametric maximum likelihood (Kalbfleisch and Prentice, 1978). Because data are subject to censoring, estimating the survival function can be thought of as a missing data problem. There are two general approaches to missing data problems: imputation, and weighting. Alternate presentations of the Kaplan-Meier estimator, including the redistribution-to-the-right algorithm of Efron (1967), the self-consistency property (Efron, 1967), or the E-M algorithm approach (Turnbull, 1976) are all examples of the imputation approach. In a series of papers, Robins and coworkers have shown that the weighting approach to missing data problems has a number of advantages over the imputation approach (Robins and Rotnitzky, 1992; Robins 1993; and Robins and Finkelstein 2000 relate directly to survival analysis). An outcome of their approach applied to survival analysis is an inverse-probability-of-censoring representation of the Kaplan-Meier estimator. The purpose of this paper is to provide a straightforward demonstration of this representation. We give two simple demonstrations of this representation. The first, found in Section 3, is more straightforward but uses as weights the Kaplan-Meier estimator for censoring times, and hence does not stand alone. For this reason, we give a second approach in Section 4 that simultaneously estimates the cumulative distribution functions of survival and censoring times using coupled inverse-probability-weighted sums. The weighted average form given in this paper is convenient for asymptotic theory and it leads to an interesting variance decomposition for the Kaplan-Meier estimator (not shown here; see Satten et al. 2001 or Robins and Finkelstein 2000 for examples of this type of result).

2. Notation and Preliminary Results

For i = 1, …, N let $T_i^{\ast }$ be the random variable denoting the (possibly unobserved) failure time and C_i be the random variable denoting the (possibly unobserved) censoring time for the ith person. We adopt the usual convention that realizations of random variables are denoted by lower-case letters. Let T_i = min( $T_i^{\ast }$,C_i) and let Δ_i = I[ $T_i^{\ast }$ ≤ C_i.]. The observed data consist of i.i.d. replicates of (T_i, Δ_i). We assume “random censoring,” i.e. that $T_i^{\ast }$ and C_i are independent. The goal is to estimate the survival function S(t) = Pr[ $T_i^{\ast }$>t] or, equivalently, the cumulative distribution function F(t) = 1−S(t).

Let the ordered failure or censoring times be τ_j, j = 1, …, J and let n_j be the number of persons who fail at time τ_j and m_j be the number of persons censored at time τ_j. We assume that no person can have a failure time equal to their censoring time (i.e. such persons are taken to be uncensored with failure time τ_j). Then, the risk set (number of persons at risk for failure at time t) can be written as

$$Y(t)={\displaystyle \sum _{j=1}^J(n_j+m_j)I[{\tau }_j\ge t]}.$$ (1)

The Kaplan-Meier estimator is ${\widehat{S}}_{km}\left(t\right)$ of S(t) is

$${\widehat{S}}_{km}\left(t\right)={\displaystyle \prod _{\left\{j|{\tau }_j\le t\right\}}(1-\frac{n_j}{Y({\tau }_j)})}$$ (2)

We can also estimate the survival function for censoring times, K(t) = Pr[C_i>t] using the Kaplan-Meier approach but considering failure events as “censored” observations and censored observations as “failures.” The Kaplan-Meier estimator of K(t) is thus

$$\widehat{K}(t)={\displaystyle \prod _{\left\{j|{\tau }_j\le t\right\}}(1-\frac{m_j}{Y({\tau }_j)})}.$$ (3)

If there were no censoring, we could estimate F(t) by the empirical cumulative distribution function

$$F^{\ast }(t)=\frac{1}{N}{\displaystyle \sum _{i=1}^{N}I}[t_i^{\ast }\le t],$$ (4)

which, considered as a random variable for each t, is an average of iid terms. The inverse-probability-of-censoring estimator analogous to F*(t) is also an average of iid terms $I[t_i^{\ast }\le t]$, each multiplied by ${\delta }_i=[t_i^{\ast }\le c_i]$ and weighted inversely by the probability that the failure time is observed, i.e. by $Pr[C_i\ge t_i^{\ast }]\equiv K(t_i^{\ast }-)$. Of course we do not know K(t) so we must use an estimate; we use the Kaplan-Meier estimator of K(t) given in (3). Because this estimator was first proposed by Robins and Rotnitzky (1992) we denote the resulting estimator by ${\widehat{F}}_{rr}(t)$; it is given by

$${\widehat{F}}_{rr}(t)=\frac{1}{N}{\displaystyle \sum _{i=1}^N\frac{I[t_i\le t]{\delta }_i}{\widehat{K}(t_i-)}}.$$ (5)

Note that we have used I[t_i≤ t] rather than $I[t_i^{\ast }\le t]$ in (4) to emphasize that ${\widehat{F}}_{rr}(t)$ can be calculated using the observed data; this replacement is possible as $I\left[t_i\le t\right]{\delta }_i=I\left[t_i^{\ast }\le t\right]{\delta }_i$.

3. Equivalence of ${\widehat{F}}_{rr}(t)$ and ${\widehat{F}}_{km}(t)$

Note that both ${\widehat{F}}_{rr}(t)$ and ${\widehat{F}}_{km}(t):=1-{\widehat{S}}_{km}(t)$ are right-continuous step functions with possible jumps at times τ_j. Thus, ${\widehat{F}}_{rr}$ and ${\widehat{F}}_{km}$ are the same if the magnitudes of the jumps in the two functions are equal. The jump in ${\widehat{F}}_{km}$ at time τ_j is given by

$${\widehat{S}}_{km}({\tau }_j-)-{\widehat{S}}_{km}({\tau }_j)={\widehat{S}}_{km}({\tau }_j-)\frac{n_j}{Y({\tau }_j)}$$ (6)

while the jump in ${\widehat{F}}_{rr}({\tau }_j)$ is given by

$${\widehat{F}}_{rr}({\tau }_j)-{\widehat{F}}_{rr}({\tau }_j-)=\frac{1}{N}\frac{n_j}{\widehat{K}({\tau }_j-)}$$

The jumps are equal provided

$$\frac{1}{N}\frac{1}{\widehat{K}({\tau }_j-)}=\frac{{\widehat{S}}_{km}({\tau }_j-)}{Y({\tau }_j)}$$

or

$${\widehat{S}}_{km}({\tau }_j-)\widehat{K}({\tau }_j-)=\frac{1}{N}Y({\tau }_j).$$ (7)

As long as there is no time τ_j for which n_jm_j > 0 (i.e., no ties between deaths and censored values), then

$${\widehat{S}}_{km}({\tau }_j-)\widehat{K}({\tau }_j-)={\displaystyle \prod _{j^{\prime }<j}\left(1-\frac{n_{j^{\prime }}}{Y({\tau }_{j^{\prime }})}\right)}{\displaystyle \prod _{j^{\prime }<j}\left(1-\frac{m_{j^{\prime }}}{Y({\tau }_{j^{\prime }})}\right)}={\displaystyle \prod _{j^{\prime }<j}\left(1-\frac{n_{j^{\prime }}+m_{j^{\prime }}}{Y({\tau }_{j^{\prime }})}\right)};$$

but

$${\displaystyle \prod _{j^{\prime }<j}\left(1-\frac{n_{j^{\prime }}+m_{j^{\prime }}}{Y({\tau }_{j^{\prime }})}\right)}=\left(1-\frac{n_1+m_1}{n_1+m_1+\cdots +n_J+m_J}\right)\left(1-\frac{n_2+m_2}{n_2+m_2+\cdots +n_J+m_J}\right)\cdots \left(1-\frac{n_{j-1}+m_{j-1}}{n_{j-1}+m_{j-1}+\cdots +n_J+m_J}\right)=\left(\frac{n_2+m_2+\cdots +n_J+m_J}{n_1+m_1+\cdots +n_J+m_J}\right)\left(\frac{n_3+m_3+\cdots +n_J+m_J}{n_2+m_2+\cdots +n_J+m_J}\right)\cdots \left(\frac{n_j+m_j+\cdots +n_J+m_J}{n_{j-1}+m_{j-1}+\cdots +n_J+m_J}\right)=\left(\frac{n_j+m_j+\cdots +n_J+m_J}{n_1+m_1+\cdots +n_J+m_J}\right)=\frac{Y({\tau }_j)}{N}$$

since n₁+m₁ ⋯+n_J+m_J = N, so that equation (7) holds.

For the case where n_jm_j>0 for some j, the argument above breaks down because

$$\left(1-\frac{n_j}{Y({\tau }_j)}\right)\left(1-\frac{m_j}{Y({\tau }_j)}\right)\ne \left(1-\frac{n_j+m_j}{Y({\tau }_j)}\right)$$

We can ask, what function K′(t) of the form $K\prime \left(t\right)={\displaystyle \prod _{\left\{j|{\tau }_j\le t\right\}}(1-d_j)}$ would make ${\widehat{F}}_{rr}(t)$ equal to ${\widehat{F}}_{km}(t)$ even in the presence of ties. The appropriate choice of d_j solves

$$\left(1-\frac{n_j}{Y({\tau }_j)}\right)(1-d_j)=\left(1-\frac{n_j+m_j}{Y({\tau }_j)}\right)$$

for each j, from which we obtain d_j = m_j/{Y(τ_j)−n_j}and hence

$$K^{\prime }(t)={\displaystyle \prod _{\left\{j|{\tau }_j\le t\right\}}\left(1-\frac{m_j}{Y({\tau }_j)-n_j}\right)}.$$

Note that K′ is the Kaplan-Meier estimator of censoring times we would obtain if we broke the ties between failures and censored observations by assuming that the failures had occurred just before the censored observations. This coincides with the usual convention when calculating the Kaplan-Meier estimator of failure times with data where there are ties between failure and censoring times (Kaplan and Meier 1958, p. 461).

4. Coupled Estimation of the Distribution of Failure and Censoring Times

The results in Section 3 are somewhat unsatisfactory in that the definition of ${\widehat{F}}_{rr}({\tau }_j)$ uses a Kaplan-Meier estimator (for the censoring times, $\widehat{K}$). Hence, these results would be unsuitable for an a priori development of the Kaplan-Meier estimator. In this section, we introduce a “new” inverse-probability-of-censoring weighted estimator of F(t) that makes no reference to the Kaplan-Meier estimator of the censoring times. We then show that this “new” estimator is identical to the Kaplan-Meier estimator. Our approach is to simultaneously estimate $F(t)=Pr[T_i^{\ast }\le t]$ and $G(t)=1-K(t)=Pr[C_i\le t]$ using coupled inverse-probability-of-censoring weighted estimators. Let $\widehat{F}(t)$ and $\widehat{G}(t)$ be given by

$$\widehat{F}(t)=\frac{1}{n}{\displaystyle \sum _{i=1}^n\frac{I[t_i\le t]{\delta }_i}{1-\widehat{G}(t_i-)}}$$

and

$$\widehat{G}(t)=\frac{1}{n}{\displaystyle \sum _{i=1}^n\frac{I[t_i\le t]{\overline{\delta }}_i}{1-\widehat{F}(t_i)}}$$

where ${\overline{\delta }}_i=I[c_i<t_i^{\ast }]$. Then $\widehat{F}(t)$ is a step function with jumps at times τ_j for which n_j > 0 and $\widehat{G}(t)$ is a step function with jumps at times τ_j for which m_j.> 0. The asymmetry in definitions of $\widehat{F}(t)$ and $\widehat{G}(t)$ reflects the choice that when failure and censoring times are tied, the censored observations are considered to have been lost to follow-up after the failures had occurred. Denoting the jumps in $\widehat{F}({\tau }_j)$ and $\widehat{G}({\tau }_j)$ by f_j and g_j we have

$$f_j=\frac{1}{N}\frac{n_j}{\left(1-{\displaystyle \sum _{j^{\prime }<j}{g}_{j^{\prime }}}\right)}$$ (8)

and

$$g_j=\frac{1}{N}\frac{m_j}{\left(1-{\displaystyle \sum _{j^{\prime }\le j}{f}_{j^{\prime }}}\right)},$$ (9)

where the sum ${\displaystyle \sum _{j<j}{g}_j}$ is to be interpreted as 0. Note that these equations are easily uncoupled to yield

$$f_j=\frac{n_j}{N-{\displaystyle \sum _{j^{\prime }<j}\frac{m_{j^{\prime }}}{\left(1-{\displaystyle \sum _{j^{″}\le j^{\prime }}{f}_{j^{″}}}\right)}}}$$ (10)

and

$$g_j=\frac{m_j}{N-{\displaystyle \sum _{j^{\prime }\le j}\frac{n_{j^{\prime }}}{\left(1-{\displaystyle \sum _{j^{″}<j^{\prime }}{g}_{j^{″}}}\right)}}}.$$ (11)

Equations (10)–(11) for the f_j and g_j are triangular, i.e. the right hand side of the equation (10) expresses f_j in terms of $f_{j^{\prime }},j^{\prime }<j$. Hence, the f_j and hence $\widehat{F}(t)$ can be calculated recursively using (10). Similarly, $\widehat{G}(t)$ can be calculated using (11), if desired.

Although it is not immediately obvious, the fact is that $\widehat{F}(t)={\widehat{F}}_{km}(t)$. To see this recall that the masses in the Kaplan-Meier estimator is the maximizer of the likelihood

$$L={\displaystyle \prod _{j=1}^J{f}_j^{n_j}{\left({\displaystyle \sum _{j^{\prime }>j}{f}_{j^{\prime }}}\right)}^{m_j}}$$ (11)

subject to ${\displaystyle \sum _{j=1}^{J+1}{f}_j=1}$. Following Turnbull (1976), note that $\{f_j,1\le j\le J+1\}$ solves this maximization problem if

$$D_j:=\frac{\partial lnL}{\partial f_j}-{\displaystyle \sum _{j=1}^{J+1}{f}_j}\frac{\partial lnL}{\partial f_j}=0$$

and ${\displaystyle \sum _{j=1}^{J+1}{f}_j=1}$. Some algebra shows that the condition D_j = 0 can be rewritten as

$$\frac{n_j}{f_j}+{\displaystyle \sum _{j^{\prime }<j}\frac{m_{j^{\prime }}}{\left({\displaystyle \sum _{j^{″}<j^{\prime }}{f}_{j^{″}}}\right)}}-N=0;$$ (12)

solving (12) for f_j yields Equation (10), establishing the equivalence of $\widehat{F}(t)$ and ${\widehat{F}}_{km}(t)$

Discussion

The Kaplan-Meier estimator is a fundamental tool in survival analysis. It is usually introduced as a nonparametric maximum likelihood estimator. The likelihood-based approach is useful, leading to useful generalizations when data are subject to interval censoring (Turnbull, 1976), truncation (Woodroofe 1985, Wang, Jewell and Tsai 1986) or both (Frydman 1994). We have shown that the Kaplan-Meier estimator can also be expressed as an inverse-probability-of-censoring weighted estimator.

The weighted average form given in this paper with the true K is an average of i.i.d. terms under the random censoring model. Even under the model when censoring times are regarded fixed (Meier, 1975), it is an average of independent (but not necessarily identically distributed) terms and is therefore subject to appropriate laws of large numbers and central limit theorems. Thus, the inverse-probability-of-censoring weighted estimator is also convenient for asymptotic theory.

Since the inverse-probability-of-censoring approach in survival analysis was introduced by Robins and Rotnitzky (1992) it has also led to useful generalizations, primarily to more general censoring models where the censoring hazard may depend on an observable covariate history (see e.g. Robins and Finkelstein, 2000, Satten and Datta 2002 and Satten et al., 2001, for recent discussions). We have given two demonstrations of the equivalence of the inverse-probability-of censoring weighted sum and product-limit representations of the Kaplan-Meier estimator. The first, given in Section 3, is designed to achieve the result quickly, but requires the availability of the Kaplan-Meier estimator of censoring times. The second, given in Section 4, is less direct, but constructs the weighted estimator without making any reference to the Kaplan-Meier estimator.