Nonparametric estimation of current status data with dependent censoring

Abstract

This paper discusses nonparametric estimation of a survival function when one observes only current status data (McKeown and Jewell, Lifetime Data Anal 16:215-230, 2010; Sun, The statistical analysis of interval-censored failure time data, 2006; Sun and Sun, Can J Stat 33:85-96, 2005). In this case, each subject is observed only once and the failure time of interest is observed to be either smaller or larger than the observation or censoring time. If the failure time and the observation time can be assumed to be independent, several methods have been developed for the problem. Here we will focus on the situation where the independent assumption does not hold and propose two simple estimation procedures under the copula model framework. The proposed estimates allow one to perform sensitivity analysis or identify the shape of a survival function among other uses. A simulation study performed indicates that the two methods work well and they are applied to a motivating example from a tumorigenicity study.

1 Introduction

This paper discusses nonparametric estimation of a survival function when one observes only current status data (McKeown and Jewell 2010; Sun 2006; Zhang et al. 2005). By current status data, we mean that each subject is observed only once and the failure time of interest is observed to be either smaller or larger than the observation time. This study was motivated by the analysis of tumorigenicity experiments, in which current status data routinely occur and the estimation of tumor prevalence function is often required (Keiding 1991). This is because that in this situation, the failure time of interest is usually the time to tumor onset, which is commonly not observed. Instead only the death or sacrifice time of an animal, serving as the observation time here, is known (Hoel and Walburg 1972; Lagakos and Louis 1988). If the tumor is nonlethal, then it is usually reasonable to assume that the time to tumor onset and the death time are independent. Of course, if the tumor is lethal, the estimation is straightforward as we would have right-censored data on the time to tumor onset. On the other hand, it is well-known that most types of tumors are between lethal and nonlethal, meaning that the time to tumor onset and the death time are correlated.

Nonparametric estimation of a survival function is often the first task in performing the failure time data analysis and many procedures have been developed for the problem (Kalbfleisch and Prentice 2002; Hu and Lawless 1996; Hu et al. 1998; Klein and Moeschberger 2003). However, most of the existing procedures are for right-censored failure time data. Several procedures have been developed for current status data if the failure time of interest (e.g. tumor onset time) and the observation time (e.g., death time) can be assumed to be independent. Lagakos and Louis (1988) discussed several examples in which the two times are related and pointed out that for the problem, the sensitivity type analysis has to be performed as the correlation between the two time variables, often referred to as lethality, cannot be estimated. In this paper, we will consider the situation where the independent assumption does not hold, for which there does not seem to exist a nonparametric estimation procedure, and present two estimation procedures. The proposed estimates will make the sensitivity analysis possible among other uses.

As mentioned above, several procedures have been proposed for nonparametric estimation of the survival or cumulative distribution function (CDF) based on current status data when the failure time of interest and the observation time can be assumed to be independent (Sun 2006). In the following, such data will be referred to as independent current status data and otherwise as dependent current status data. For the former, the maximum likelihood estimate of a survival function can be easily derived by using the pool-adjacent-violator algorithm for the isotonic regression. Also many authors have considered other analysis issues on them such as treatment comparison and regression analysis (Sun 2006). For dependent current status data, on the other hand, there exists only limited literature in general except in the field of tumorigenicity experiments. Among others, Zhang et al. (2005) considered regression analysis of general dependent current status data and proposed an estimating equation-based inference procedure. A large literature exists for tumorigenicity experiments and in this case, a common approach is to apply the three-state model to describe the tumor process. However, with respect to estimation of tumor growth and prevalence function, most of the existingmethods are parametric procedures or assume that tumors are lethal or nonlethal. It is obvious that a nonparametric estimate would be very useful for identifying the shape of the prevalence function or model checking among other uses.

Several approaches are commonly used to deal with correlated failure times or the failure time data with dependent censoring (Hougaard 1986). One is the frailty approach that models the correlation by using some latent variables and another is the copula model approach. Let T and X denote two possibly related random variables, F and G their marginal distributions, respectively, and H their joint distribution. Then it has been shown (Nelsen 2006, Theorem 2.3.3) that there exists a copula function C(u, v), defined on I² = [0, 1] × [0, 1] with C(u, 0) = C(0, v) = 0, C(u, 1) = u and C(1, v) = v, such that

$$H(t,x)=C\left(F\right(t),G(x\left)\right),t\ge 0,x\ge 0.$$ (1)

If F and G are continuous, C is unique. Furthermore, for any given copula function C and marginal distribution functions F and G, the function H defined in Eq. 1 is the corresponding joint distribution function. Among others, Zheng and Klein (1995) employed the copula model approach for estimation of a survival function in the case of dependent right-censored data. In the following, we will adopt the same approach. However, it should be noted that the two situations are quite different as in the latter, the data structure is much more complex and the observed relevant information is extremely less.

In the following, we will assume that T and X represent the failure time of interest and the observation time, respectively. The main goal of this paper is to discuss non-parametric estimation of F under model (1). In Sect. 2, we will show that if the copula function C is known, the survival function F of interest can be uniquely identified. In Sect. 3, we will present two simple consistent estimates of F and both can be easily obtained. The first one is developed for the Archimedean copula functions and has a closed form, while the second one is for general copula functions. Section 4 gives some numerical results and in Sect. 5, we apply the methods to a set of current status data arising from a tumorigenicity experiment. Section 6 concludes with some discussion and remarks. In the following, we will assume that both T and X are continuous variables.

2 Estimation of the marginal distribution function F

Consider a survival study that involves n independent subjects and gives the observed data {X_i, δ_i = I (X_i ≥ T_i); i = 1, …, n}, the i.i.d. replications of {X, δ = I (X ≥ T )}. Let F, G and H be defined as before and suppose that P(T_i = X_i) = 0. It is easy to see that given the observed current status data, one can directly estimate the following two functions

$$G\left(x\right)=P(X\le x)$$

and

$$p_1\left(x\right)=P(X>x,T<X),(0\le x\le \infty )$$

by, say, their empirical estimates $\widehat{G}\left(x\right)=n^{-1}{\sum }_{i=1}^{n}I(X_i\le x)$ and ${\widehat{p}}_1\left(x\right)=n^{-1}{\sum }_{i=1}^{n}I(X_i<x,{\delta }_i=1)$, respectively. In the following, we will show that the marginal distribution function F of T is uniquely determined by these two functions.

Let C be the copula function defined in (1). Then we have

$$C(u,v)=H\left(F^{-1}\right(u),G^{-1}(v\left)\right),u,v\in [0,1].$$

Let μ_C denote the probability measure corresponding to C. Then a simple calculation gives

$$p_1\left(x\right)={\mu }_C\left(A_x\right(F,G\left)\right),$$

where A_x(F, G) = {(u, v): 0 ≤ u ≤ FG⁻¹(v), G(x) ≤ v ≤ 1}. The following theorem establishes the identifiability of F for the situation considered here.

Theorem 1 Suppose the marginal distribution functions F and G are continuous and strictly increasing over (0, ∞). Also suppose μ_C(E) > 0 for any open set E in I². Then given the copula function C, the marginal distribution function F is uniquely determined by G(x) and p₁(x).

Proof To show the theorem, it is sufficient to show that if the distribution function pairs (F₁, G) and (F₂, G) yield the same p₁(x) under the copula function C, then F₁ = F₂. Suppose that there exists a point x₀ > 0 such that F₁(x₀) < F₂(x₀). Denote v₀ = G(x₀). Then we have F₁G⁻¹(v₀) < F₂G⁻¹ (v₀) and it follows from the assumption that

$$p_1\left(x_0\right)={\mu }_C\left(A_{x_0}\right(F_1,G\left)\right)={\mu }_C\left(A_{x_0}\right(F_2,G\left)\right).$$

Note that the equation above suggests that μ_C(A_x0 (F₂, G)\A_x0 (F₁, G)) = 0 and v₀ is strictly less than 1 since G(·) is strictly increasing. Then the set E₀ = {(u, v) : 0 ≤ u₁₀ < u < u₂₀, v₀ < v < 1} with u₁₀ = F₁G⁻¹(v₀) < u₂₀ = F₂G⁻¹(v₀) is open, not empty and contained in A_x0 (F₂, G)\A_x0 (F₁, G). Hence μ_C(E₀) = 0, which contradicts with the assumption that μ_C(E) > 0 for any open set E in I². This shows that F₁ = F₂ and completes the proof.

The theorem above suggests that one can estimate F by estimating G(x) and p₁(x). Actually one can similarly show that F is uniquely determined by G(x) and

$$p_2\left(x\right)=P(X\le x,T<x),(0\le x\le \infty )$$

too. Zheng and Klein (1995) gave a similar result for right-censored data under the framework of competing risks. As pointed out before, the situation considered here is more complex and quite different. In the next section, we will present two estimation procedures for F.

3 Two estimation procedures

In this section, we will discuss two situations about the copula function C and present two estimates of the marginal distribution function F. First we will consider the Archimedean copula function, one of the most commonly used type of copula functions, and in this case, we will show that F can be explicitly expressed by other known or easily estimated functions under model (1). This naturally yields a closed form estimate. We will then consider general copula functions and in this situation, the proposed estimate involves solving some simple equations.

3.1 Estimation with archimedean copula functions

In this section, we will assume that C is an Archimedean copula function given by

$$C_{\varphi }(u,v)={\varphi }^{-1}\left[\varphi \right(u)+\varphi (v\left)\right],\varphi \in \Phi ,$$ (2)

where Φ denotes the class of functions ϕ : [0, 1] → [0, ∞] with continuous first and second derivatives and satisfying

$$\varphi \left(1\right)=0,{\varphi }^{\prime }<0,{\varphi }^{\prime \prime }>0,0<t<1.$$

Then we can show the following theorem.

Theorem 2 Suppose that the conditions given in Theorem 1 hold and also suppose that ϕ(t) → ∞ and |ϕ′(t)| → ∞ when t → 0. Then F can be expressed as

$$F\left(t\right)={\varphi }^{-1}\left\{\varphi \left({\left({\varphi }^{\prime }\right)}^{-1}\left(\frac{g\left(t\right){\varphi }^{\prime }\left(G\right(t\left)\right)}{D_1\left(t\right)}\right)\right)-\varphi \left(G\right(t\left)\right)\right\},$$ (3)

where D₁(t) = dp₂(t)/dt and g(t) = dG(t)/dt.

Proof Let c(u, v) denote the density function of C(u, v), U = F(T), V = G(X). First note that

$$\begin{array}{cc}\hfill P(T<X,X\le x)& =P\left(F^{-1}\right(U)<G^{-1}(V),G^{-1}(V)\le x)\hfill \\ \hfill & =P(U<F{G}^{-1}(V),V\le G(x\left)\right)\hfill \\ \hfill & ={\int }_0^{G\left(x\right)}dv{\int }_0^{F{G}^{-1}\left(v\right)}c(u,v)du\hfill \\ \hfill & ={\int }_0^{x}g\left(v\right)dv\left\{{\int }_0^{F\left(v\right)}c(u,G(v\left)\right)du\right\}.\hfill \end{array}$$

It then follows that

$$D_1\left(t\right)=g\left(t\right){\int }_0^{F\left(t\right)}c(u,G(t\left)\right)du=g\left(t\right)C_v\left(F\right(t),G(t\left)\right)$$

with

$$C_v(u,v)=\frac{\partial }{\partial v}{\varphi }^{-1}\left[\varphi \right(u)+\varphi (u\left)\right]=\frac{{\varphi }^{\prime }\left(v\right)}{{\varphi }^{\prime }\left({\varphi }^{-1}\right[\varphi \left(u\right)+\varphi \left(u\right)\left]\right)}$$

based on Theorem 4.3.8 of Nelsen (2006). Thus we have

$$D_1\left(t\right)=g\left(t\right)\frac{{\varphi }^{\prime }\left(G\right(t\left)\right)}{{\varphi }^{\prime }\left({\varphi }^{-1}\right(\varphi \left(F\right(t\left)\right)+\varphi \left(G\right(t\left)\right)\left)\right)},$$

$${\varphi }^{\prime }\left({\varphi }^{-1}\right(\varphi \left(F\right(t\left)\right)+\varphi \left(G\right(t\left)\right))=\frac{g\left(t\right){\varphi }^{\prime }\left(G\right(t\left)\right)}{D_1\left(t\right)},$$

$$\varphi \left(F\right(t\left)\right)+\varphi \left(G\right(t\left)\right))=\varphi \left\{{\left({\varphi }^{\prime }\right)}^{-1}\left(\frac{g\left(t\right){\varphi }^{\prime }\left(G\right(t\left)\right)}{D_1\left(t\right)}\right)\right],$$

and

$$\varphi \left(F\right(t\left)\right)=\varphi \left\{{\left({\varphi }^{\prime }\right)}^{-1}\left(\frac{g\left(t\right){\varphi }^{\prime }\left(G\right(t\left)\right)}{D_1\left(t\right)}\right)\right\}-\varphi \left(G\right(t\left)\right).$$

All these together give Eq. 3 and complete the proof.

Let Ĝ(t) and p̂₁ denote the empirical estimates of G and p₁ as given in the previous section and define ${\widehat{p}}_2=n^{-1}{\sum }_{i=1}^{n}I(X_i\le t,{\delta }_i=1)$, the empirical estimate of p₂. Also define

$$\widehat{g}\left(t\right)=\frac{1}{n}\sum _{i=1}^{n}I(X_i=t)$$

and

$${\widehat{D}}_1\left(t\right)=\frac{1}{n}\sum _{i=1}^{n}I(X_i=t,{\delta }_i=1).$$

More comments on these estimates will be given below. Then the theorem above suggests that one can estimate F by

$${\widehat{F}}_1\left(t\right)={\varphi }^{-1}\left\{\varphi \left({\left({\varphi }^{\prime }\right)}^{-1}\left(\frac{\widehat{g}\left(t\right){\varphi }^{\prime }\left(\widehat{G}\right(t\left)\right)}{{\widehat{D}}^1\left(t\right)}\right)\right)-\varphi \left(\widehat{G}\right(t\left)\right)\right\}$$ (4)

at t where D̂₁(t) > 0.

3.2 Estimation with general copula functions

In this section, we consider estimation of the marginal distribution function F for general copula functions. Let Ĝ p̂₂ and ĝ be defined as before. By Theorem 1, F can be uniquely determined by G and p₂ given C. So it is natural to develop an estimate of F by using Ĝ and p̂₂.

Let 0 < x₁ < x₂ < … < x_k denote a sequence of fixed time points. Their selection will be discussed below. To derive the estimate, note that

$$\begin{array}{cc}\hfill P(T<X\mid X=y)& =\frac{P(T<y,X=y)}{P(X=y)}\hfill \\ \hfill & =\underset{\Delta y\to 0}{\lim }\frac{P(T<y,0\le X<y)}{P(0\le X<y)}\hfill \\ \hfill & =\underset{\Delta y\to 0}{\lim }\frac{H(y,y+\Delta y)-H(y,y)}{G(y+\Delta y)-G\left(y\right)}\hfill \\ \hfill & =\underset{\Delta y\to 0}{\lim }\frac{C\left(F\right(y),G(y+\Delta y\left)\right)-C\left(F\right(y),G(y\left)\right)}{G(y+\Delta y)-G\left(y\right)}\hfill \\ \hfill & =\underset{\Delta G\left(y\right)\to 0}{\lim }\frac{C\left(F\right(y),G(y+\Delta y\left)\right)-C\left(F\right(y),G(y\left)\right)}{\Delta G\left(y\right)}\hfill \\ \hfill & =\frac{\partial C\left(F\right(y),G(y\left)\right)}{\partial G\left(y\right)}.\hfill \end{array}$$

This gives

$$p_2\left(x\right)={\int }_0^{x}P(T<X\mid X=y)dG\left(y\right)={\int }_0^x{C}_v\left(F\right(y),G(y\left)\right)dG\left(y\right),$$ (5)

where C_v = ∂C(u, v)/∂v. It follows that a natural estimate of F can be derived by replacing G and p₂ in Eq. 5 with their empirical estimates given above. More specifically, let F̂₂ denote the resulted estimate of F. Then at x_j, F̂₂(x_j) can be determined by solving the equation

$$\sum _{l=1}^j{C}_v\left\{F\right(x_l),\widehat{G}(x_l\left)\right\}\widehat{g}\left(x_l\right)={\widehat{p}}_2\left(x_j\right)$$ (6)

with replacing F(x_l) by F̂₂(x_l) for l < j. If taking C to be the Frank copula function, given in Eq. 7 below, then the equation above becomes

$$\sum _{l=1}^j\widehat{g}\left(x_l\right)\frac{(e^{-\theta \cdot F\left(x_l\right)}-1)e^{-\theta \widehat{G}\left(x_l\right)}}{e^{-\theta }-1+(e^{-\theta F\left(x_l\right)}-1)\left(\right(e^{-\theta \widehat{G}\left(x_l\right)}-1\left)\right)}={\widehat{p}}_2\left(x_j\right).$$

It can be easily shown that the two estimates proposed above are consistent and nondecreasing functions for large n. Also it can be seen that both estimates F̂₁ and F̂₂ are defined only at the observation time points X_i with δ_i = 1. For their values between these time points, it is natural to define them as linear functions. In practice, to apply the two estimation procedures, it is better to choose the x_j’s as a subset of the observation times with δ_i = 1 and tied observations, which is often the case in medical follow-up times. Sometimes for small n, one may want to group the data before choosing the x_j’s and applying the procedures to get accurate estimates. Also for small n, it should be noted that the resulting estimates may not be nondecreasing functions sometimes and in this case, a simple modification can be carried out. For example, one can define F̃₁(x_j) = max{F̂(x_l); l = 1, …, j} and do the same for F̂₂.

4 A numerical study

A simulation study was conducted to assess the performance of the two simple estimates proposed in the previous section. In the study, we assumed that both F and G were the exponential distributions with the hazards λ₁ and λ₂, respectively, and considered two different copula functions. One is the Archimedean copula function defined in Eq. 2 with

$$\varphi \left(t\right)=-\ln \left\{\frac{e^{-\alpha t}-1}{e^{-\alpha }-1}\right\},$$

which gives

$$C(u,v)=-\frac{1}{\alpha }ln\left\{1+\frac{(e^{-\alpha u}-1)(e^{-\alpha v}-1)}{e^{-\alpha }-1}\right\},$$ (7)

and the other is the Farlie–Gumbel–Morgenstern (FGM) copula function

$$C(u,v)=uv+\theta uv(1-u)(1-v).$$ (8)

The first one is usually referred to as the Frank copula function and the second one is a non-Archimedean copula function. For given the copula function above and the parameter values, we first generated the data on the X_i’s and T_i’s by using the random number generation function in R. The current status data were then defined by keeping the X_i’s and defining δ_i = I (X_i ≥ T_i).

Figure 1a presents the means of the two estimates F̂₁ and F̂₂ based on 1,000 simulated data sets generated under model (7) with n = 200, λ₁ = 1.0961, λ₂ 0.4 and α = 1. For comparison, the true function F is also included in the figure. Figure 1b displays the mean of the estimate F̂₂ based on the simulated data generated under model (8) with n = 200, λ₁ = 0.8537, λ₂ = 0.4 and θ = 0.5 also along with the true F. Note that here the values of λ₁ and λ₂ were chosen to give 25% right-censored data. In both figures, we use estimate 1 to denote that given by formula (4) and estimate 2 for the other proposed estimate. These results indicate that both proposed estimates seem to perform well for the situations considered here and the two estimation procedures give similar results under the Archimedean copula model. We also considered other cases and obtained similar results.

Fig. 1.

Estimation of the CDF by two proposed procedures a Estimation using Frank copula model. b Estimation using FGM copula model

5 An application

In this section, we apply the estimation procedures proposed in the previous sections to a set of lung tumor current status data that motivated this study. It arose from a tumorigenicity experiment on 146 mice and was first discussed in Hoel and Walburg (1972). The experiment consists of two treatments: conventional environment (CE, 96 mice) and germfree environment (GE, 48 mice). For each animal, the data gave the death time, serving as the observation time, and the status of the presence or absence of the lung tumor at the death. That is, we only have current status data on lung tumor onset time, the response variable of interest.

Following Hoel and Walburg (1972), many authors analyzed the data set and most of them assumed that the tumor is nonlethal, meaning that the tumor time and the death time are not related. One exception was given by Lagakos and Louis (1988), who discussed the comparison of the lung tumor occurrence rates between the two treatment groups conditional on the correlation between the occurrence of lung tumor and the death or the lethality of the tumor. They showed that the test result heavily depended on the assumed correlation or lethality. Our goal here is to give some graphical view and comparison of the prevalence functions between the two groups.

Figure 2 displays the estimated CDFs of the lung tumor onset time for the animals in the two treatment groups given by Eq. 7 by assuming the correlation equal to 0.1631, 0.4567, or 0.8329, respectively. It can be easily seen that both the estimates and the difference between the two estimates depend on the correlation. In other words, both the estimates and the treatment difference depend on the lethality. If we believe that the lethality is low, then it seems that there was no significant treatment difference and otherwise, the animals in the two groups seem to have significantly different tumor growth rates. More specifically, if assuming that the lethality is high, Fig. 2 would indicate that the animals in the GE group survived much longer than those in the CE group.

Fig. 2.

Estimated CDF of the lung tumor onset time given by formula (4). a Estimation with correlation = 0.8329. b Estimation with correlation = 0.4567. c Estimation with correlation = 0.1631

We also tried several other copula functions including the FGM model, for which the estimate given by the Eq. 6 was obtained, and they all gave estimates and conclusions similar to those given in Fig. 2. Note that the main goal here is to conduct a sensitivity analysis as mentioned above and for this, a general and simple procedure is to try different copula functions unless there exists more or prior information about the problem. The analysis indicates that the level of lethality plays an important role here and more information on it is needed to have more conclusive results. Also one needs to be careful to interpret the results as the sample size is relatively small.

6 Discussion and concluding remarks

This paper discussed the analysis of dependent current status data, which are sometimes also phased as current status data with dependent censoring. As it is well-known and discussed before, informative censoring is a difficult but important problem in general and in this case, most of the developed approaches are not applicable. Among others, one major issue is how to characterize the relationship between the censoring variable and the variable of interest. For the nonparametric estimation considered here, we took the copula model approach and developed two estimation procedures for estimation of the marginal distribution function of the failure time of interest. The proposed estimates allow one to perform sensitivity analysis of dependent current status data or the data from tumorigenicity experiments on tumors that are between lethal and nonlethal. As commented above, in these situations, it is not possible to conduct a deterministic analysis.

In both of the proposed estimation procedures, the empirical estimates of G and p₂ along with the raw estimates of g and D₁ were used. It should be noted that instead of these, any consistent estimates of them such as some smooth estimates could be used. One situation in which one may want to employ smooth or other estimates is that all observed time points X_i’s are different or there do not exist many tied observations although this does not happen often in clinical trials or medical follow-up studies. In this case, as mentioned before, an alternative to the use of other estimates is to apply some grouping techniques to the data.

The methodology has some limitations or more work remains to be done. At this moment, no method is available for the variance estimation of the proposed estimates and also the asymptotic distribution of the estimates is unknown. Both seem to be very difficult and are beyond the scope of this study. Of course, for variance estimation, one could employ some bootstrap procedures. Another limitation is that we have assumed that the copula function C is completely known as otherwise it is not possible to identify the marginal distribution of interest. As shown in the example, one way to deal with this in practice is to assume that C belongs to some copula family and to fit the data using different values of the association parameter. For this, it would be useful to develop some procedures for checking the model fit.

The focus of this paper has been on current status data, a special case of interval-censored data (Fang et al. 2002; Li and Pu 1999, 2003; Sun 2005). One of the key differences between the two types of data is the data structure and the censoring mechanism. For current status data, two variables are involved with one being the failure time variable of interest and the other being the censoring or observation time variable. In contrast, for the analysis of interval-censored data, one has to deal with three variables with two being related censoring variables. It is easy to see that the problem considered here can easily occur in the case of interval-censored data but it is not straightforward to generalize the estimation procedures presented above to general situations. In other words, it would be very useful to develop some appropriate estimation procedures for interval-censored data with dependent censoring.