Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2025 Apr 25.
Published in final edited form as: Stat Sin. 2023 Oct;33(4):2763–2786. doi: 10.5705/ss.202021.0213

NONPARAMETRIC ESTIMATION AND TESTING FOR PANEL COUNT DATA WITH INFORMATIVE TERMINAL EVENT

Xiangbin Hu 1, Li Liu 1, Ying Zhang 1, Xingqiu Zhao 1
PMCID: PMC12030113  NIHMSID: NIHMS2018426  PMID: 40290451

Abstract

Informative terminal events often occur in the long term recurrent event follow-up studies. To reflect their effects on recurrent event processes explicitly, we propose a reversed nonparametric mean model for panel count data with a terminal event subject to right censoring. This model enjoys meaningful interpretation for applications and robustness for statistical inference. Treating the distribution of the right-censored terminal event time as a nuisance functional parameter, we develop a two-stage estimation procedure by combining the Kaplan-Meier estimator and nonparametric sieve estimation techniques. The consistency, convergence rate and asymptotic normality of the proposed nonparametric estimator are established. Then we construct a class of new statistics for two-sample test. The asymptotic properties of the new tests are established and evaluated by extensive simulation studies. Panel count data from Chinese Longitudinal Healthy Longevity study are analyzed using the proposed method for illustration.

Key words and phrases: Panel count data, Two-stage estimation, Nonparametric test, Terminal event, Monotone polynomial spline

1. Introduction

In longitudinal follow-up studies, recurrent events are the same type events that individuals may experience repeatedly. However, exact recurrent event times may not be observed in many practical applications. Instead, individuals can be examined at distinct time points, and the number of recurrent events between two observation time points are collected. Such data are socalled panel count data which frequently occur in medical follow-up studies and clinical trials. Many methods have been developed for the analysis of panel count data. For example, Sun and Kalbfeisch (1995), Wellner and Zhang (2000), Hu, Lagakos, and Lockhart (2009), and Lu, Zhang, and Huang (2007) studied nonparametric estimation of mean function of a underlying counting process with panel count data; Thall and Lachin (1988), Sun and Fang (2003), Zhang (2006), Balakrishnan and Zhao (2009), and Zhao and Zhang (2017) developed some nonparametric tests for comparison of mean functions of counting processes based on panel count data. Among others, Cheng and Wei (2000), Sun and Wei (2000), Hu, Sun, and Wei (2003), Sun, Tong, and He (2007), Wellner and Zhang (2007), Lu, Zhang, and Huang (2009), He et al. (2017), and Jiang, Su, and Zhao (2020) studied semiparametric regression models with panel count data, while Zhao et al. (2019) investigated a nonparametric regression model for panel count data analysis.

The recurrent events are often truncated by a terminal event such as death. A commonly used approach is a joint modeling approach through a shared frailty variable to deal with the correlation between the terminal event and recurrent event processes; see for example, Liu, Wolfe, and Huang (2004), Zhao, Zhou, and Sun (2011), Sun et al. (2012), Zhao, Li and Sun (2013a,b), and Zhou et al. (2017). However, these approaches do not exhibit an explicit effect of the terminal event on recurrent events, which is often the study of interest in scientific investigations. For example, Chan et al. (1995) found that the occurrences of AIDS-defending events were accumulated sharply before the death of the HIV-infected individuals; Lunney et al. (2003) claimed that the functional decline changed tremendously in the last year of the HIV-infected individuals. Directly modeling the effect of terminal event on recurrent events is a challenging task. Recently, Chan and Wang (2010) developed a backward model to study the behavior of recurrent events prior to a terminal event. Chan and Wang (2017) considered effects of covariates and extended the backward model to a semiparametric model. Kong et al. (2018) proposed a conditional model for longitudinal data by treating the terminal event time as a covariate. Our goal is to propose a reversed mean model for panel count data with an informative terminal event, and then to develop nonparametric inference procedures for comparing the mean functions of recurrent event processes that are likely truncated by a terminal event.

The main contributions of this work are fourfold. First, we propose a reversed nonparametric mean model for panel count data with a right-censored terminal event, where the nonparametric mean function is increasingly dependent on the terminal event time. Thus, the proposed model provides an intuitive interpretation of the effect of terminal event on recurrent event processes. Second, we develop a two-stage sieve-based nonparametric estimation procedure by treating the distribution function of the terminal event time as a nuisance functional parameter. Third, we establish the asymptotic properties of the proposed estimator by overcoming the challenges from both a nuisance functional parameter and a nonparametric estimator with a convergence rate slower than the standard rate n-1/2. Fourth, we develop a class of nonparametric tests for comparison of mean functions of recurrent event processes with panel count data in the presence of informative terminal event.

The remainder of this paper is organized as follows. In Section 2, we present a reversed mean model anchoring at a terminal event and propose a two-stage nonparametric sieve-based estimation procedure. In Section 3, we establish the asymptotic properties of the proposed estimator. In Section 4, we propose a class of new test statistics for two-sample comparison and establish their asymptotic normality. In Section 5, we conduct simulation studies to demonstrate the finite-sample performance of the proposed methods. In Section 6, we apply the proposed methods to panel count data from Chinese Longitudinal Healthy Longevity study. Some concluding remarks are made in Section 7. All technical proofs are provided in the supplementary materials.

2. Model Setting and Estimation Procedure

Suppose that a counting process {N(t):0≤t≤τ} denotes the number of recurrent events occurring up to time t, where τ is a fixed time point. Let T_=T1,T2,⋯,TK be the observation times of N(t), where K represents the total number of observation times. Then the observed counting process is N_=N1,N2,⋯,NK=NT1,NT2,⋯,NTK. Let U and C be the terminal event time and the censoring time, respectively. The time for the last observation is denoted by Y=U∧C and whether the time is the terminal even time is denoted by Δ=1{U≤C}. The observed data for subject i consists of Xi=Yi,Δi,Ki,Ti_,Ni_,i=1,⋯,n, where Ti_=Ti1,Ti2,⋯,TiKi and Ni_=NTi1,NTi2,⋯,NTiKi. Let 𝒳=Xi,i=1,…,n denote the sample of such panel count data from n subjects.

To investigate the effect of terminal event on the recurrent event process, motivated by Chan and Wang (2010, 2017) and Kong et al. (2018), we consider a counting process N˜(t;U) denoting the event counts from time t to the terminal event U, and propose a reversed nonparametric mean model anchoring at the terminal event:

EN˜t;UU=u=Λu-t, (2.1)

where Λ(⋅) is an unknown nondecreasing function with Λ(0)=0 to ensure the identifiability of this model. In this paper, we propose to model the recurrent events anchoring at the terminal event time through (2.1). The advantage of this approach is to straightforwardly describe the behavior of the stochastic process from the terminal event while all the event counts can still be used for learning the model of the entire process. This model implies that EN˜t1;U∣U=u-EN˜t2;U∣U=u=Λu-t1-Λu-t2, where 0≤t1≤t2≤u. Noting that Nt2-Nt1=N˜t1;U-N˜t2;U and N(0)=0, we obtain E(N(t)∣U=u)=Λ(u)-Λ(u-t).

Let F(u) denote the underlying distribution function of U. To make a valid inference for our proposed model, we consider the following data scenarios: (i) U and C are independent; (ii) The censoring event time C is noninformative to Λ. (iii) Given (Y,Δ), the distribution of (K,T_) is non-informative to Λ. Define △Nj=NTj-NTj-1 and △Λj(u)=Λu-Tj-1-Λu-Tj for j=1,⋯,K with T0=0. A straightforward calculation yields that

E∑j=1K△Nj-ΔΛj(U)2∣Y,Δ,K,T_,N_=∑j=1KΔΔNj-ΔΛj(Y)2+∑j=1K1-Δ∫Y∞△Nj-ΔΛju2dFu1-FY,

hence we propose a least-squares based loss function

ℓn(Λ,F;𝒳)=1n∑i=1n∑j=1KiΔiΔNi,j-ΔΛi,jYi2+1-Δi∫Yi∞ΔNi,j-ΔΛi,j(u)2dF(u)1-FYi, (2.2)

where △Ni,j=NiTij-NiTi(j-1) and △Λi,j(u)=Λu-Ti(j-1)-Λ(u-Tij. A natural idea is to take the minimizer of ln(Λ,F;𝒳) defined in (2.2) as the estimator of the parameter. However, since the loss function involves an unknown distribution function F, it is difficult to estimate Λ and F simultaneously. To tackle the problem, we propose a two-stage approach. Specifically, in Stage 1, we estimate F by using the Kaplan-Meier (KM) estimator Fˆn(u) Kaplan and Meier, 1958). In Stage 2, Λˆn is obtained by minimizing the loss function ℓnΛ,Fˆn;𝒳 with respect to Λ. Since the observed data of (Y,Δ) are used in both Stages 1 and 2, to distinguish them, we use the notation (Y˜,Δ˜) to represent the data when we obtain the KM estimator in Stage 1 without any ambiguity.

We adopt the spline-based sieve estimation method to estimate the nondecreasing function Λ because of its numerical advantage and good statistical properties (Lu, Zhang and Huang, 2007, 2009). Let ti:i=1,⋯,mn+2d be a sequence of knots that partition [0,τ] into mn+1 subintervals, where

0=t1=⋯=td<td+1<⋯<tmn+d<tmn+d+1=⋯=tmn+2d=τ.

Let qn=mn+d and Il(s),l=1,⋯,qn be the I-spline basis functions of order d (Ramsay, 1988). We then define the functional space of the estimator for Λ to be

Φn=∑l=1qnαlIl(s):αl≥0,l=1,⋯,qn.

Since Fˆn is a monotone step function, as shown in Section S1 in the online Supplementary Material, minimizing the loss function lnΛ,Fˆn;𝒳 is a quadratic programming with the constraint that αl≥0 for l=1,⋯,qn. Let I(s)=I1(s),⋯,Iqn(s)T,α=α1,⋯,αqnT, and the solution of quadratic programming be αˆ=αˆ1,⋯,αˆqnT. Then the spline estimator of Λ(s) is Λˆn(s)=I(s)Tαˆ.

3. Asymptotic Properties

To present the asymptotic results, we introduce some notation. Let g(r) be the rth derivative of function g. For r≥1, define

ℋr=g:g(r-1)(s)-g(r-1)(t)≤c0|s-t|forall0≤s,t≤τΦ=Λ∈ℋr:Λisanondecreasingcontinuousfunctionon[0,τ]withΛ(0)=0},ℱ=F:Fisadistributionfunctionon0,∞.

Denote the true value of (Λ,F) to be Λ0,F0∈Φ×ℱ. For

B1,B2∈B0,τ=:B∩0,τ:B∈B,

where B denotes the collection of Borel sets, set

μ1B1×B2=∫∑k=1∞P(K=k∣U=u)×∑j=1kPu-Tj∈B1,u-Tj-1∈B2∣K=k,U=udF0u,μ2B1×B2=∫∑k=1∞P(K=k∣U=u)×Pu-TK∈B1,u∈B2∣K=k,U=udF0(u). (3.3)

Then μ1 and μ2 are two measures on [0,τ]2,B[0,τ]2. For any functions Λ1,Λ2∈Φ, we define the metric as

d1Λ1,Λ22=ΔΛ1s1,s2-ΔΛ2s1,s2L2μ12=E∑j=1KΔΛ1,j(U)-ΔΛ2,j(U)2=E∑j=1KΔΔΛ1,j(Y)-ΔΛ2,j(Y)2+(1-Δ)∫Y∞ΔΛ1,j(u)-ΔΛ2,j(u)2dF0(u)1-F0(Y),

where △Λs1,s2=Λs2-Λs1. For any F1,F2∈ℱ, we define the metric as

d2F1,F2=F1-F2∞,

where ∥⋅∥∞ represents the L∞ norm. Denote the δ-neighbourhood of F0 by ℱδ=F∈ℱ:d2F,F0≤δ for any small δ>0.

To establish the asymptotic properties of the proposed estimator, we assume:

  • (C1) 0<Λ0τ<∞.

  • (C2) 0<F0τ<1. F0 is absolutely continuous with respect to Lebesgue measure. Moreover, the density function f0(s) has a uniform positive lower bound for all s∈M1,τ, where M1 is a constant representing the minimum value of the support of F0.

  • (C3) E∑j=1K△Nj-△Λ0,j(U)2<∞.

  • (C4) The probability of censoring ϱ=P(Y<U) satisfies that 0<ϱ< 1.

  • (C5) mn=Onν for 0<ν<1/2. Moreover, we suppose that
    maxd+1≤i≤mn+d+1ti-ti-1=On-ν,
    and there is a constant M2>0 such that
    maxd+1≤i≤mn+d+1ti-ti-1mind+1≤i≤mn+d+1ti-ti-1≤M2
    uniformly for n.
  • (C6) There is a constant M3>0 such that PK≤M3=1.

  • (C7) PTj-Tj-1≥M4 for all j=1,⋯,K=1 with some constant M4>0.

Remark 1. Condition (C1) is standard in the literature of nonparametric estimation. Condition (C2) generally holds for the cumulative distribution function of a continuous random variable in view of applications. Condition (C3) requires that ∑j=1K△Nj has finite second order central moment. Condition (C4) ensures the censoring rate between 0 and 1, which is commonly assumed in practice of survival data analysis. Condition (C5) is a regularity condition for monotone spline estimation, see Lu, Zhang, and Huang (2007, 2009). Condition (C6) is similar to Condition (C2) in Wellner and Zhang (2007), which indicates that the number of observations is bounded. Condition (C7) requires that the adjacent observation times are separable (Wellner and Zhang, 2007), which is generally the case in applications.

Theorem 1 (Consistency for the Two-Stage Estimator). Suppose that Conditions (C1)-(C7) hold. Then, for every 0≤b1≤b2≤τ satisfying μ20,b1×b2,τ>0, we have

△Λˆns1,s21s1,s2∈b1,b2×b1,b2-Λ0s1,s21s1,s2∈b1,b2×b1,b2L2μ12=op(1)

In particular, if μ2({0}×{τ})>0, then d1Λˆn,Λ0=op(1).

To establish the rate of convergence and the asymptotic normality, we further assume:

  • (C8) μ1 is absolutely continuous with respect to Lebesgue measure with a derivative μ˙1, and μ˙1 has a uniform positive lower bound.

  • (C9) There is a constant 0<λa<∞ such that 1/λa<Λ0′(s)<λa for all s∈τ′,τ with 0<τ′≤τ such that Λ0τ′>0.

  • (C10) P(U≥τ)=ω1>0 and P(C≥τ)=ω2>0.

  • (C11) EecN(t) is uniformly bounded for t∈[0,τ] and some constant c.

Remark 2. Condition (C8) implies that the metric μ1 defined in (3.3) has a strictly positive intensity. Condition (C9) requires the true conditional mean function be absolutely continuous with bounded intensity function, which is reasonable as explained in Wellner and Zhang (2007). (C10) is used as a technical condition in the proof of the uniform weak convergence rate of the KM estimator according to Kong et al. (2018). Condition (C11) holds when N(t) is from a Poisson-type process or is uniformly bounded conditional on terminal event time, which is also generally true in view of applications.

Theorem 2 (Rate of Convergence). Suppose that Conditions (C1)-(C11) hold and μ2({0}×{τ})>0. Taking ν=1/(1+2r), we have d1Λˆn,Λ0=Opn-r/(1+2r).

To present the asymptotic normality, we introduce more notation. Let 𝒫 and Pn denote probability measure and empirical measure with 𝒫f=∫fd𝒫 and Pnf=n-1∑i=1nfXi, respectively. Write ℓn(Λ,F;𝒳)=Pnm(Λ,F;X). Let l∞ℋr be the space of bounded functionals on ℋr under the L∞ norm. For h∈ℋr, define a sequence of maps Qn of a neighborhood of Λ0,F0, denoted by 𝒰, in the parameter space for (Λ,F) into l∞ℋr as the derivative of ℓn(Λ,F;𝒳) with respect to Λ in the direction h:

Qn(Λ,F)[h]=limε→0ℓn(Λ+εh,F;𝒳)-ln(Λ,F;𝒳)ε=Pnlimε→0m(Λ+εh,F;X)-m(Λ,F;X)ε=Pnψ(Λ,F;X)[h]

and define Q(Λ,F)[h]=𝒫ψ(Λ,F;X)[h], where

ψ(Λ,F;X)[h]=∑j=1KΔΔNj-ΔΛj(Y)△hj(Y)+(1-Δ)∫Y∞ΔNj-△Λj(u)△hj(u)dF(u)1-F(Y)

with △hj(u)=hu-Tj-1-hu-Tj,j=1,…,K. Furthermore, define

Q˙Λ0,Fˆn(1)Λˆn-Λ0[h]=limε→0QΛ0+εΛˆn-Λ0,Fˆn[h]-QΛ0,Fˆn[h]ε=-𝒫ςΛˆn,Fˆn;X[h]

with

ς(Λ,F;X)[h]=∑j=1KΔ△Λj(Y)-△Λ0,j(Y)△hj(Y)+(1-Δ)∫Y∞△Λj(u)-△Λ0,j(u)△hj(u)dF(u)1-F(Y),

and

Q˙Λ0,F0(2)Fˆn-F0[h]=limε→0QΛ0,F0+εFˆn-F0[h]-QΛ0,F0[h]ε=𝒫∫Y∞φ‾Λ0,F0(u;X)[h]dFˆn-F0(u),

with

φ‾Λ,F(u;X)[h]=1-Δ1-F(Y)∑j=1KΔNj-ΔΛj(u)⋅Δhj(u)-∫Y∞ΔNj-ΔΛj(s)⋅Δhj(s)dF(s)1-F(Y).

Theorem 3 (Asymptotic Normality). Suppose that Conditions (C1)-(C11) hold and μ2({0}×{τ})>0.

  1. Then for any bounded function h∈ℋr, we have
    -nQ˙Λ0,Fˆn1Λˆn-Λ0h=nQ˙Λ0,F02Fˆn-F0h+nQnΛ0,F0h+op1.
  2. Moreover, n𝒫ςΛˆn,Fˆn;X[h]→dN0,σ02, where
    σ02=E𝒫φΛ0,F0;X;Y˜,Δ˜[h]+ψΛ0,F0;X[h]2,

    and φ(Λ,F;X;Y˜,Δ˜)[h] is defined in the online Supplementary Material.

4. Two-Sample Test

Suppose that n subjects are from two groups with corresponding sample sizes n1 and n2, where n1+n2=n. Denote the observed data of the lth group as Xi(l):i=1,⋯,nl=Yi(l),Δi(l),Ki(l),Ti(l)_,Ni(l)_:i=1,⋯,nl, for l=1,2. Given the terminal event time U(l)=u, the conditional mean function of N˜(l)(t) is Λl(u-t).

4.1. Terminal Events with an Identical Distribution

In this subsection, we assume that terminal event times share the same distribution function F0 for all the subjects so that its estimator Fˆn can be obtained based on the pooled data. We investigate two-sample test with the null hypothesis H0:Λ1=Λ2=Λ0. Denote Λˆl and Λˆn as the estimates of Λl and Λ0 based on the data set of group l and the pooled data, respectively.

Theorem 4 (Two-Sample Test with an Identical Distribution of Terminal Events). In addition to the conditions in Theorem 3, we suppose that hn(⋅) is a bounded weight process, and there is a bounded function h∈ℋr such that

d12hn,h=E∑j=1K△hn,j(U)-△hj(U)2=opn-1/(1+2r).

Assume that n1/n→p as n→∞, where 0<p<1. Then under H0:Λ1=Λ2,Un=nPnςΛˆ1,Fˆn;Xhn-ςΛˆ2,Fˆn;Xhn converges in distribution to N0,(1/p+1/(1-p))σ02, where σ02=Eψ2Λ0,F0;X[h]. Moreover, σ02 can be consistently estimated by σˆn2=Pnψ2Λˆn,Fˆn;Xhn.

Remark 5. Theorem 4 states the asymptotic normality of the new statistic Un and gives a consistent estimator of its asymptotic variance. Then the hn-specific standardized statistic, Tnhn=Unσˆnnn1+nn2-1 can be applied to conduct the two-sample hypothesis test. A natural choice is to take hn=Λˆn. Other possible choices of the weight processes are explored and evaluated in the simulation studies. It is worth noting that the weight process is only required to be bounded, which is more flexible compared to the monotone condition in Zhang (2006) and Balakrishnan and Zhao (2009).

4.2. Terminal Events with Different Distributions

In this subsection, we assume the distribution and density functions of U(l),l=1,2 to be Fl and fl, which may be different for the two groups. Let Fˆl and Λˆl be the estimator of Fl and Λl, respectively. Given a partition 0=t0(l)<t1(l)<⋯<tνnl(l)=τ, we define the histogram-type estimators of fl as fˆl(u)=Fˆltil(l)-Fˆltil-1(l)/ti(l)-ti-1(l) for til-1(l)≤u<til(l) following Földes, Rejtő, and Winter (1981). Set fT(u)=p1f1(u)+p2f2(u),

wlu-t1-wlu-t2=hu-t1-hu-t2fT(u)/fl(u),wn(l)u-t1-wn(l)u-t2=hnu-t1-hnu-t2nln+nrnfˆr(u)fˆl(u),

where pl=limn→∞nl/n for l,r=1,2,l≠r.

Theorem 5 (Two-Sample Test with Different Distributions of Terminal Events). Suppose that the conditions in Theorem 4 hold for each group and fl ‘s are Lipschitz continuous. Then for Λ0∈ℋr,r≥2, under the null hypothesis H0:Λ1=Λ2=Λ0 we have

  1. U˜n=1n∑l=12∑i=1nlςΛˆ1,Fˆl;Xi(l)hn-ςΛˆ2,Fˆl;Xi(l)hn converges in distribution to N0,σ12/p1+σ22/p2, where
    σl2=E𝒫φlΛ0,Fl;X(l);Y˜(l),Δ˜(l)wl+ψΛ0,Fl;X(l)wl2

    with φl(Λ,F;X;Y˜,Δ˜)[w] defined in the online Supplementary Material.

  2. In addition, suppose the knots of partition satisfy
    maxi=1,⋯,νnl(l)ti(l)-ti-1(l)→0andnlogn1/4mini=1,⋯,νnl(l)ti(l)-ti-1(l)→∞
    as n→∞. Then σl2 can be consistently estimated by σˆl2, and the asymptotic variance of U˜n can be consistently estimated by σ˜n2=nσˆ12/n1+σˆ22/n2, where
    σˆl2=PnlPnlφn(l)Λˆl,Fˆl;X(l);Y˜(l),Δ˜(l)wn(l)+ψΛˆl,Fˆl;X(l)wn(l)2

    with φn(l)(Λ,F;X;Y˜,Δ˜)[w] defined in the online Supplementary Material.

According to Theorem 5, we can apply the statistics T˜nhn=U˜n/σ˜n to test the equality of the mean function of counting processes under the scenario of different terminal event time distributions between the two samples.

5. Simulation Studies

5.1. Two-Stage Estimation for the Mean Function

We conducted simulation studies to evaluate the finite-sample performance of the two-stage estimator Λˆn with sample size n=100 and 200. For subject i, the observation Xi=Yi,Δi,Ki,Ti_,Ni_ was generated as below. The latent terminal event time Ui was from 6 + exp(1), and the censoring time Ci was from 6+κexp(3). The study ending time was set at τ=10. The values of κ were chosen to be 1.375 and 0.500 such that the terminal events were censored by 20% and 40%, respectively, in the study. Let Yi=Ui∧Ci and Δi=1Ui≤Ci. The number of observation Ki was generated from the discrete uniform distribution in {1,…,6}. Given the censored terminal event time Yi, observation time Ti_=Ti1,Ti2,…,TiKi was an ordered sample from uniform distribution Unif0,Yi. Let Ni be a Poisson process with ENi(t)∣U=u=Λ0(u)-Λ0(u-t) and Λ0(s)=s, which means that NiTi1 was from Poisson distribution with mean Ti1, and NiTij-NiTi(j-1) was from Poisson distribution with mean Tij-Ti(j-1). For the knots of spline, took d=mn=3 and let td+1,td+2,td+3 be quartiles of Yi-Tij:j=1,…,Ki,i=1,…,n. All simulation studies were based on 1000 replications.

Figure 1 shows the results of two-stage estimators for the conditional mean functions. The solid line represents the true mean function, the dashed line represents the average of the estimated mean function, and the dotted-dash lines represent the 2.5% and 97.5% pointwise percentiles of the functional estimator based on 1000 replications. From (a)-(d) in Figure 1, we can conclude that the sample mean of the estimated mean functions is very close to the true one, the degree of the variability of the functional estimator decreases as the sample increases indicating the asymptotic unbiasedness of the proposed method, and the sampling distribution of the estimator does not appear to be much impacted by the censoring rate.

Figure 1:

Figure 1:

Estimates of the mean function.

5.2. Two-Sample Test with an Identical Distribution of Terminal Event Time

We conducted the two-sample test under the scenario where the two groups have a same distribution of terminal event time. We generated two groups of independent and identically distributed sample Xi(l):i=1,⋯,nl by the same way as in Subsection 5.1 for l=1,2, with sample size n1=n2=100, 150 or 200. Let N(l) be a Poisson process with EN(l)(t)∣U(l)=u=Λl(u)-Λl(u-t) for the lth group. We considered the following two cases:

Case1:Λ1(s)=s,Λ2(s)=βs;
Case2:Λ1s=s,Λ2s=10ss+β.

We took β=1,1.1,1.2,1.3 in Case 1 and β=1,3,5 in Case 2.

Figure 2 plots the true mean functions for the two cases with different values of β. It can be seen that the conditional mean functions of two groups in Case 1 do not overlap but they cross over in Case 2. The weight processes hn(j)(t),j=1,2,3 in Theorem 4 were chosen to be

hn1t=t,hn2t=1n∑i=1n1t≥TiKi,hn3t=Λˆnt.

The results of the simulation study were evaluated with 1000 replications.

Figure 2:

Figure 2:

Discrepancy of the mean functions

Figure 3 presents the quantile plots of the test statistics with the three weight processes against the standard normal distribution for Case 1 with n1=n2=200,β=1 and censoring rate 20%. They demonstrate that the normality of the test statistics is generally satisfactory with sample size of 200. Similar results were obtained for other situations and hence were not presented here.

Figure 3:

Figure 3:

Q-Q plots for n1=n2=200,β=1, and censoring rate = 20% when the distribution functions of the terminal event time are identical for the two groups.

Tables 1 and 2 report the sizes and powers of the proposed test statistics Tnhn(j) with the three weight processes at significance level 0.05 for different values of β for Cases 1 and 2, respectively, where Tnhn(j) represents the test statistic with the jth weight process for j=1,2,3. The simulation results show that the proposed test possesses good properties: (i) the estimated sizes are all around the significance level of 0.05; (ii) for any fixed value of β in both cases, the power increases when sample size increases; (iii) the power increases to 1 when the discrepancy of the mean functions becomes more evident implying the consistency of the proposed test; (iv) the test powers also appear not to be much impacted by the censoring rate.

Table 1:

Simulation results of the two-sample tests with different weights for Case 1 when the distribution functions of the terminal event time are identical for the two groups.

Censoring rate 20% Censoring rate 40%
β Tn(hn(1)) Tn(hn(2)) Tn(hn(3)) Tn(hn(1)) Tn(hn(2)) Tn(hn(3))
n1=n2=100
1 0.051 0.047 0.052 0.052 0.048 0.051
1.1 0.242 0.229 0.243 0.258 0.240 0.255
1.2 0.728 0.703 0.728 0.715 0.690 0.717
1.3 0.967 0.956 0.967 0.962 0.956 0.960
n1=n2=150
1 0.048 0.047 0.048 0.049 0.048 0.048
1.1 0.359 0.344 0.361 0.355 0.338 0.354
1.2 0.873 0.851 0.873 0.877 0.859 0.878
1.3 0.996 0.989 0.996 0.995 0.991 0.995
n1=n2=200
1 0.042 0.044 0.043 0.043 0.051 0.043
1.1 0.438 0.398 0.443 0.467 0.433 0.467
1.2 0.944 0.924 0.943 0.954 0.939 0.954
1.3 1.000 1.000 1.000 0.999 0.998 0.999

Table 2:

Simulation results of the two-sample tests with different weights for Case 2 when the distribution functions of the terminal event time are identical for the two groups.

Censoring rate 20% Censoring rate 40%
β Tn(hn(1)) Tn(hn(2)) Tn(hn(3)) Tn(hn(1)) Tn(hn(2)) Tn(hn(3))
n1=n2=100
1 0.999 1.000 0.752 1.000 1.000 0.868
3 0.982 0.971 0.859 0.987 1.000 0.892
5 0.998 1.000 0.991 0.999 1.000 0.991
n1=n2=150
1 1.000 1.000 0.900 1.000 1.000 0.960
3 0.998 1.000 0.949 0.998 1.000 0.970
5 1.000 1.000 0.998 1.000 1.000 0.999
n1=n2=200
1 1.000 1.000 0.941 1.000 1.000 0.977
3 1.000 1.000 0.986 1.000 1.000 0.989
5 1.000 1.000 1.000 1.000 1.000 1.000

5.3. Two-Sample Test with Different Distributions of Terminal Event Times

We conducted a parallel simulation study to evaluate the proposed two-sample test for two groups with different distributions of terminal event time. The data sets were generated in a similar mechanism as in Subsection 5.2 except that the terminal event times Ui(1) and Ui(2) were from 6 + exp(1) and 6 + exp(2), and the censoring times Ci(1) and Ci(2) were from 6+κ1exp(3) and 6+κ2exp(3), where κ1,κ2=(1.375,5.769) and (0.500, 1.010) to result in 20% and 40% censoring rate, respectively. The three weight processes hn(j)(t),j=1,⋯,3 were similarly chosen as

hn1t=t,hn2t=1n∑i=1n1t≥TiKi,hn3t=Λˆ1t+Λˆ2t2.

To calculate the histogram-type estimators of fl, we evenly divided [6,τ] into 5 subintervals. The results of the simulation study were again evaluated with 1000 replications.

Figure 4 presents the quantile plots of the test statistics with the three weight processes against the standard normal distribution with n1=n2=200,β=1 and censoring rate 20% for Case 1. The normality of the test statistics does appear to be justified when sample size is 200. The plots for other situations are similar and hence not presented here. Tables 3 and 4 summarize the sizes and powers of proposed statistics T˜nhn(j),j=1,2,3 at significance level 0.05 for Cases 1 and 2, respectively, which results in the same conclusions as those in Subsection 5.2.

Figure 4:

Figure 4:

Q-Q plots for n1=n2=200,β=1, and censoring rate = 20% when the distribution functions of the terminal event time are different for the two groups.

Table 3:

Simulation results of the two-sample tests with different weights for Case 1 when the distribution functions of the terminal event time are different for the two groups.

Censoring rate 20% Censoring rate 40%
β T˜n(hn(1)) T˜n(hn(2)) T˜n(hn(3)) T˜n(hn(1)) T˜n(hn(2)) T˜n(hn(3))
n1=n2=100
1 0.056 0.061 0.061 0.068 0.056 0.072
1.1 0.231 0.221 0.231 0.260 0.241 0.260
1.2 0.688 0.664 0.693 0.721 0.687 0.717
1.3 0.960 0.956 0.960 0.956 0.951 0.955
n1=n2=150
1 0.043 0.049 0.041 0.056 0.052 0.054
1.1 0.338 0.311 0.341 0.352 0.330 0.348
1.2 0.863 0.835 0.865 0.872 0.851 0.869
1.3 0.993 0.992 0.994 0.994 0.985 0.992
n1=n2=200
1 0.046 0.048 0.045 0.058 0.057 0.060
1.1 0.447 0.427 0.450 0.469 0.451 0.469
1.2 0.951 0.933 0.953 0.952 0.941 0.954
1.3 1.000 0.998 1.000 0.999 0.999 0.998

Table 4:

Simulation results of the two-sample tests with different weights for Case 2 when the distribution functions of the terminal event time are different for the two groups.

Censoring rate 20% Censoring rate 40%
β T˜n(hn(1)) T˜n(hn(2)) T˜n(hn(3)) T˜n(hn(1)) T˜n(hn(2)) T˜n(hn(3))
n1=n2=100
1 1.000 1.000 0.834 1.000 1.000 0.926
3 0.986 1.000 0.884 0.999 1.000 0.947
5 0.997 1.000 0.987 1.000 1.000 0.996
n1=n2=150
1 1.000 1.000 0.940 1.000 1.000 0.985
3 0.998 1.000 0.976 0.999 1.000 1.000
5 1.000 1.000 0.999 1.000 1.000 0.999
n1=n2=200
1 1.000 1.000 0.972 1.000 1.000 0.990
3 1.000 1.000 0.994 1.000 1.000 0.995
5 1.000 1.000 1.000 1.000 1.000 1.000

6. Real Data Analysis

In this section, we applied our method to analyzing the dataset from Chinese Longitudinal Healthy Longevity Survey (CLHLS) during the period from 1998 to 2014 (Zeng et al., 2017). This survey was conducted by the Center for Healthy Aging and Development Studies (CHADS) of National School of Development at Peking University and Chinese Center for Disease Control and Prevention (CDC) to study the determinants of healthy human longevity and the later-life mortality. The data consist of survey information in the seven waves at 1998, 2000, 2002, 2005, 2008, 2011 and 2014 by interviewing 9093 respondents who were 77 years or older in the 1998 baseline survey. In each wave, the respondents were asked to provide information regarding their health, socioeconomic status, family, lifestyle and demographic profile on a date randomly chosen in the interview year.

For this dataset, we analyzed panel count data of serious illness before death, for which the serious illness is defined as the illness making people bedridden, and compared the incidences of serious illness among elderly people living in different areas during the whole study period. The incidences of serious illness were counted repeatedly if the respondents were confined to bed again after recovering from the previous serious illness. We chose to analyze data for those subjects who had at least one follow-up observation after 1998 to allow for a meaningful analysis of panel count data. Therefore, we included 4362 elders for analysis by removing 3368 individuals, among which 2,005 dies in 2000 or between 1998 and 2000; 894 lost to follow-up in 2000, and 469 had missing or mistakenly documented records. Among the 4362 individuals, 1489 and 1561 lived in the urban and rural areas, respectively, all the time, and 1312 lived in both areas during this period. For the ith individual, we defined Tij as the time in month between the j survey and the baseline survey, Ni(t) as the accumulated number of serious illness that the individual suffered up to time t, and Ki as the total number of follow-up surveys in the study. Death was the terminal event and loss to follow-up was treated as censoring event to death. In sum, the longest follow-up time was τ=200 (months) and the censoring rate was 25.72%.

We studied the mean function of the serious illness process using model (2.1). We estimated the mean function using I-spline with order d=3 and mn=3 to divide [0,τ] into 4 subintervals with the internal knots of splines selected at td+1=τ/4,td+2=τ/2,td+3=3τ/4. Figure 5 plots the estimated mean functions for the combined sample (solid line), urban sample (dash line), rural sample (dotted-dash line), and duo-area sample (longdash line), respectively. It reveals that the urban-area residents tended to experience more serious illnesses than the rural-area and duo-area residents, which may be the outcome that the rural-area and duo-area residents had limited access to advanced medical system compared to their counterparts in the urban area.

Figure 5:

Figure 5:

The two-stage estimates of the mean function for the CLHLS data.

We then used the same three weight functions hn(j),j=1,2,3 as those in Subsection 5.2 to conduct the two-sample test for each pair of two mean functions for the urban sample, rural sample, and duo-area sample. That is the null hypotheses were H0(1):ΛU(s)=ΛR(s),H0(2):ΛU(s)=ΛD(s), and H0(3):ΛR(s)=ΛD(s), where ΛU,ΛR, and ΛD are the corresponding mean functions for the urban sample, rural sample, and duo-area sample, respectively. To conduct the test, we first concluded by the log-rank test that the survival function of death for the urban-area residents was significantly different from the rural-area residents and the duo-area residents with p-values 3×10-6 and 0.006, respectively. Furthermore, the log-rank test revealed that the survival functions of death for the rural-area residents were not significantly different from the duo-area residents with p-value =0.09. Figure 6 displays the KM estimates for the survival functions of death for elders in the three groups. Therefore, we used the test statistic in Theorem 4 to compare the mean functions for the rural-area and duo-area residents. For the tests H0(1):ΛU(s)=ΛR(s) and H0(2):ΛU(s)=ΛD(s), we divided [0,τ] evenly into 8 subintervals to obtain the histogram-type estimators, and then applied the test statistic in Theorem 5. The test results are summarized in Table 5 suggesting a significant difference in later-life serious illness experience at level 0.01 between the urban-area and the rural-area residents, and between the urban-area and the duo-area residents. Though there is no significant difference in the mean function between the duo-area and rural-area residents from a statistical perspective (p-value >0.05), the duoarea residents seemed to have fewer incidences of severe illness compared to rural-area residents. It may be due to the facts that: (i) many rural-area residents were transitioned to urban along with the urbanization movement in China in recent years; (ii) the elder duo-area residents tend to be in a better family socioeconomic status with better health status compared to other rural-area residents. The conclusion was consistent regardless of the choice of the weight processes.

Figure 6:

Figure 6:

The KM estimates of the survival function for the CLHLS data.

Table 5:

Two-sample test results with the three weights for the CLHLS data.

hn(1) hn(2) hn(3)
H0(1):ΛU(s)=ΛR(s)
U˜n 287.904 3.968 4.325
σ˜n 77.849 1.138 1.182
T˜n 3.698 3.488 3.659
p-value < 0.001*** < 0.001*** < 0.001***
H0(2):ΛU(s)=ΛD(s)
U˜n 368.945 4.132 5.225
σ˜n 79.298 1.193 1.205
T˜n 4.653 3.465 4.335
p-value < 0.001*** < 0.001*** < 0.001***
H0(3):ΛR(s)=ΛD(s)
Un 84.872 0.384 0.932
σn 66.322 0.957 0.833
Tn 1.280 0.401 1.119
p-value 0.201 0.688 0.263

Tn(h) (or T˜n(h): the observed value of the test statistic with different weight functions;

***

represents significance level of 0.01

7. Discussion

Considering the potential association between recurrent event and terminal event processes that often occur in life sciences, we have developed a conditional nonparametric mean functional model to study the explicit effects of the terminal events on the recurrent event process with panel count data. This approach is particularly useful in ascertaining the recurrent event behavior prior to the terminal event. However, the approach has challenges in both numerical computation and statistical inference due to the nature of panel count data subject to informative termination. We have designed a two-stage spline-based estimation procedure to ease the computation burden, and studied the asymptotic behavior of the proposed nonparametric estimator using modern empirical process theories. In particular, the asymptotic normality can be readily expanded to address multi-sample hypothesis testing problems. Our extensive simulation studies have demonstrated the good performance of the proposed method in finite-sample settings.

The proposed method can be extended to consider a conditional semiparametric mean function model:

EN˜t;UU=u,Z=expβTZΛu-t,0≤t≤u,

where N˜(t;u)=N(u)-N(u-t) and N(t) is the number of recurrent events up to time t,U is the terminal event time, Z is a covariate vector, β is an unknown parameter, and Λ(⋅) is an unknown nondecreasing baseline function. This model has a broader application that not only captures the effect of the terminal event but also allows to assess external risk factors on recurrent event process, and is currently under our investigation.

Supplementary Material

Suppl

Acknowledgements

The authors are grateful to the Editor, Associate Editor, and two referees for their valuable comments and suggestions that greatly improved the paper. This work is supported in part by the Research Grant Council of Hong Kong (15301218, 15303319) for Xingqiu Zhao and by NIH/NIGMS (2 U54 GM115458-06) for Ying Zhang.

Footnotes

Supplementary Materials

The online Supplementary Materials provide the calculation of the loss function, some preliminary lemmas, the proofs of Theorems 1–5, some additional works on simulation studies and the real data analysis.

References

  1. Balakrishnan N, and Zhao X (2009). New multi-sample nonparametric tests for panel count data. Annals of Statistics 37, 1112–1149. [Google Scholar]
  2. Chan KCG, and Wang M-C (2010). Backward estimation of stochastic processes with failure events as time origins. Annals of Applied Statistics 4, 1602–1620. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Chan KCG, and Wang M-C (2017). Semiparametric modeling and estimation of the terminal behavior of recurrent marker processes before failure events. Journal of the American Statistical Association 112, 351–362. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Chan SF, Neaton JD, Saravolatz LD, Crane LR, and James Osterberger for the Community Programs for Clinical Research on AIDS. (1995). Frequencies of opportunistic diseases prior to death among HIV-infected persons. Aids 9, 1145–1151. [DOI] [PubMed] [Google Scholar]
  5. Cheng SC, and Wei LJ (2000). Inferences for a semiparametric model with panel data. Biometrika 87, 89–97. [Google Scholar]
  6. Földes A, Rejtő L, and Winter BB (1981). Strong consistency properties of nonparametric estimators for randomly censored data, II: Estimation of density and failure rate. Periodica Mathematica Hungarica 12, 15–29. [Google Scholar]
  7. Harrington DP, and Fleming TR (1982). A class of rank test procedures for censored survival data. Biometrika 69, 553–566. [Google Scholar]
  8. He X, Feng X, Tong X, and Zhao X (2017). Semiparametric partially linear varying coefficient models with panel count data. Lifetime Data Analysis 23, 439–466. [DOI] [PubMed] [Google Scholar]
  9. Hu XJ, Lagakos SW, and Lockhart RA (2009). Generalized least squares estimation of the mean function of a counting process based on panel counts. Statistica Sinica 19, 561–580. [PMC free article] [PubMed] [Google Scholar]
  10. Hu XJ, Sun J, and Wei LJ (2003). Regression parameter estimation from panel counts. Scandinavian Journal of Statistics 30, 25–43. [Google Scholar]
  11. Jiang H, Su W, and Zhao X (2020). Robust estimation for panel count data with informative observation times and censoring times. Lifetime Data Analysis 26, 65–84. [DOI] [PubMed] [Google Scholar]
  12. Kaplan EL, and Meier P (1958). Nonparametric estimation from incomplete observations. Journal of the American Statistical Association 53, 457–481. [Google Scholar]
  13. Kong S, Nan B, Kalbfleisch JD, Saran R, and Hirth R (2018). Conditional modeling of longitudinal data with terminal event. Journal of the American Statistical Association 113, 357–368. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Liu L, Wolfe RA, and Huang X (2004). Shared frailty models for recurrent events and a terminal event. Biometrics 60, 747–756. [DOI] [PubMed] [Google Scholar]
  15. Lu M, Zhang Y, and Huang J (2007). Estimation of the mean function with panel count data using monotone polynomial splines. Biometrika 94, 705–718. [Google Scholar]
  16. Lu M, Zhang Y, and Huang J (2009). Semiparametric estimation methods for panel count data using monotone B-splines. Journal of the American Statistical Association 104, 1060–1070. [Google Scholar]
  17. Lunney JR, Lynn J, Foley DJ, Lipson S, and Guralnik JM (2003). Patterns of functional decline at the end of life. Journal of Nonparametric Statistics 289, 2387–2392. [DOI] [PubMed] [Google Scholar]
  18. Ramsay JO (1988). Monotone regression splines in action. Statistical Science 3, 425–441. [Google Scholar]
  19. Sun J, and Fang HB (2003). A nonparametric test for panel count data. Biometrika 90, 199–208. [Google Scholar]
  20. Sun J, and Kalbfleisch JD (1995). Estimation of the mean function of point processes based on panel count data. Statistica Sinica 5, 279–290. [Google Scholar]
  21. Sun L, Song X, Zhou J, and Liu L (2012). Joint analysis of longitudinal data with informative observation times and a dependent terminal event. Journal of the American Statistical Association 107, 688–700. [Google Scholar]
  22. Sun J, Tong X, and He X (2007). Regression analysis of panel count data with dependent observation times. Biometrics 63, 1053–1059. [DOI] [PubMed] [Google Scholar]
  23. Sun J, and Wei LJ (2000). Regression analysis of panel count data with covariate-dependent observation and censoring times. Journal of the Royal Statistical Society: Series B 62, 293–302. [Google Scholar]
  24. Thall PF, and Lachin JM (1988). Analysis of recurrent events: Nonparametric methods for random-interval count data. Journal of the American Statistical Association 83, 339–347. [Google Scholar]
  25. van der Vaart AW, and Wellner JA (1996). Weak convergence and empirical processes. New York: Springer-Verlag. [Google Scholar]
  26. Wellner JA, and Zhang Y (2000). Two estimator of the mean of a counting process with panel count data. Annals of Statistics 28, 779–814. [Google Scholar]
  27. Wellner JA, and Zhang Y (2007). Two likelihood-based semiparametric estimation methods for panel count data with covariates. Annals of Statistics 35, 2106–2142. [Google Scholar]
  28. Zeng Y, Vaupel J, Xiao Z, Liu Y, and Zhang C Chinese Longitudinal Healthy Longevity Survey (CLHLS), 1998–2014. Ann Arbor, MI: Inter-university Consortium for Political and Social Research; [distributor], 2017-04-11. 10.3886/ICPSR36692.v1 [DOI] [Google Scholar]
  29. Zhang Y (2006). Nonparametric k-sample tests for panel count data. Biometrika 93, 777–790. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Zhao H, Li Y, and Sun J (2013a). Analyzing panel count data with a dependent observation process and a terminal event. Canadian Journal of Statistics 41, 174–191. [Google Scholar]
  31. Zhao H, Li Y, and Sun J (2013b). Semiparametric analysis of multivariate panel count data with dependent observation processes and a terminal event. Journal of Nonparametric Statistics 25, 379–394. [Google Scholar]
  32. Zhao H, Zhang Y, Zhao X, and Yu Z (2019). A nonparametric regression model for panel count data analysis. Statistica Sinica 29, 809–826. [Google Scholar]
  33. Zhao X, and Zhang Y (2017). Asymptotic normality of nonparametric M-estimators with applications to hypothesis testing for panel count data. Statistica Sinica 27, 931–950. [Google Scholar]
  34. Zhao X, Zhou J, and Sun L (2011). Semiparametric transformation models with timevarying coefficients for recurrent and terminal events. Biometrics 67, 404–414. [DOI] [PubMed] [Google Scholar]
  35. Zhou J, Zhang H, Sun L, and Sun J (2017). Joint analysis of panel count data with an informative observation process and a dependent terminal event. Lifetime Data Analysis 23, 560–584. [DOI] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Suppl

RESOURCES