Robust Dynamic Risk Prediction with Longitudinal Studies

Abstract

Providing accurate and dynamic age-specific risk prediction is a crucial step in precision medicine. In this manuscript, we introduce an approach for estimating the τ-year age-specific absolute risk directly via a flexible varying coefficient model. The approach facilitates the utilization of predictors varying over an individual's lifetime. By using a nonparametric inverse probability weighted kernel estimating equation, the age-specific effects of risk factors are estimated without requiring the specification of the functional form. The approach allows borrowing information across individuals of similar ages, and therefore provides a practical solution for situations where the longitudinal information is only measured sparsely. We evaluate the performance of the proposed estimation and inference procedures with numerical studies, and make comparisons with existing methods in the literature. We illustrate the performance of our proposed approach by developing a dynamic prediction model using data from the Framingham Study.

1 Introduction

Accurate and individualized risk prediction is a key component of precision medicine. For example in colorectal cancer, risk calculator can help tailoring individual's screening regimen and making decisions on specific ages for screening initialization and surveillance. Factors pertaining to specific ages, such as family history and nutrition intake are important to be incorporated in the outcome prediction. For cardiovascular disease (CVD), the Framingham risk score (FRS) (Wolf and others 1991) has been developed separately for men and women based on risk factors such as total cholesterol, high-density-lipoprotein, systolic blood pressure and smoking status. Patients with 10-year FRS below 10% are considered to be at lower risk for vascular events during the next decade, whereas patients with scores between 10% and 20% are at moderate risk and those larger than 20% are at higher risk. Various intervention strategies can be implemented based on such risk stratification (Mosca and others 2004). In these clinical settings, the analytical goal is to provide patients with an estimate of the likelihood of developing a disease within the next τ-years given the subject is disease free at age a and his/her risk profile updated by age a, i.e., the age-specific absolute τ-year residual life risk.

Currently available prediction models are often limited in predicting such age-specific absolute residual life risk. For example, the FRS model includes age as a standard risk factor with linear effects. However it is well recognized that the relationship between age and CVD risk may change in magnitude through complex interactions with other risk factors (Ridker and others 2007). Naturally, a risk equation should be a more stochastic function of age (Lloyd-Jones 2010). Such simplistic models, failing to capture the complex age varying effects, may lead to poor risk estimates and prediction models with low discriminatory power. To be clinically useful with sufficiently adequate prediction accuracy, an ideal prediction model should take into account an individual's most up to date health information and reflect the fact that risk factors may have differential effects on the τ-year residual life risk over various stages of an individual's life span.

Constructing a model that accurately captures how risks change dynamically over lifetime, while of great importance, is challenging for several reasons. Often important risk factors change over time. However, collecting such time-varying information for a large prospective cohort can be a major undertaking. Most cohort studies only collect age-specific information intermittently, sometimes irregularly. Characterizing changes over time with limited measures at discrete time points requires much deliberation. Furthermore, the importance of risk factors on disease outcome may change during an individual's lifetime. For example, body mass index (BMI) may have substantially different effects on future CVD risks depending on the age. Characterizing age-varying effects with a powerful yet flexible statistical model can be challenging. Similar challenges arise when assessing the prediction performance of an age-specific prediction model since the accuracy may also vary with age.

A popular approach in the risk prediction literature is to decompose the absolute risk into two components: the age-dependent disease risk for a baseline risk profile, which can be estimated from a prospective cohort study or external disease incidence data, and the relative risk of developing disease for a particular risk factor profile compared to the baseline (Gail and others 1989, Liu and others 2014). While such an approach can accommodate differential effects of risk factors at different ages by adding interaction terms of age and the other covariates in the regression model, it does not incorporate time-varying covariates collected over time. To incorporate repeated measurements, joint modeling of both the covariate process Z(·) and a survival outcome T have been developed in recent years (Tsiatis and others 1995, Wang and Taylor 2001, Ye and others 2008). Parameter estimation typically involves specifying the covariate process Z(·) and then linking Z(·) to T via a proportional hazard (PH) model with time-varying covariates. In addition to requiring strong modeling assumptions about Z(·) and T, these joint modeling methods have a limitation of being computationally infeasible when many time-varying covariates are under consideration. Semi-parametric methods have been proposed to directly make prediction of τ-year residual life risk given covariate information at a landmark time t₀ (Zheng and Heagerty 2005, Parast and others 2012). However, no existing methods allow for incorporating age-specific effects non-parametrically with sparsely measured time-varying covariates. Furthermore, procedures for non-parametrically evaluating such age-specific prediction models with longitudinal markers and censored event times are not yet available.

In this paper, we propose to directly model the age-specific absolute risk function via a flexible varying-coefficient model and estimate the covariate effects as functions of age via inverse probability weighted (IPW) kernel estimating equations. The procedure allows the estimation of age-specific risks flexibly with the longitudinally collected risk factor information on the same patient, while borrowing strength across individuals of similar ages at different study time points. It also handles irregularly measured serial covariates and censoring easily. Our proposed model, by allowing the effects of risk factors to change over age and the target residual life span τ, is more realistic and could potentially lead to improved predictive performance. To quantify the performance of the age-specific models of τ-year residual life risk, it would be desirable to consider measures of prediction performance specific to age and prediction time since the prediction accuracy of such models is likely to vary over both dimensions. A wide range of performance measures have been considered to quantify the time-specific prediction accuracy of τ-year absolute risk models constructed with baseline markers (Zheng and others 2006, Uno and others 2007, Gerds and others 2008). In the longitudinal setting, Zheng and Heagerty (2004) considered a model-based approach for estimating the accuracy in the absence of censoring. To guard against potential model mis-specification and incorporate censored outcomes, we propose an IPW kernel estimator to calculate model performance parameters that quantify the accuracy of the proposed prediction models in predicting τ-year residual life at age a. No existing methods provide non-parametric estimates of such prediction performance with longitudinal markers and censored outcomes.

The remainder of this manuscript is organized as follows. In Section 2, we describe the proposed model and estimation framework. Then we present simulation results comparing our approach to other popular methods in Section 3. We apply the proposed method to the Framingham Heart Study in Section 4, assessing the age-specific effects of routinely used cardiovascular risk factors on the 10-year residual CVD risk and quantifying the performances of the prediction models. We conclude with a brief discussion in Section 5.

2 Methods

Let T_i be the time to event onset since a baseline time such as study entry. Due to censoring, one can only observe X_i = min{T_i, C_i} and δ_i = I(T_i ≤ C_i), where C_i is the censoring time and I(·) is the indicator function. To facilitate the calculation of age-specific risks, we also record age at the occurrence of the event and censoring. Let A_i0 be the age at which subject i enters the study and then $T_i^{\mathbb{A}}=T_i+A_{i0}$ is the age at which the event occurs, and $C_i^{\mathbb{A}}=C_i+A_{i0}$ is the age at which $T_i^{\mathbb{A}}$ might be censored. Let $X_i^{\mathbb{A}}=X_i+A_{i0}=min\{T_i^{\mathbb{A}},C_i^{\mathbb{A}}\}$. In addition to the event time information, risk markers are ascertained repeatedly during the follow up. For the ith subject, let Z_i(a) = (Z_i1(a), Z_i2(a), …, Z_ip(a))^⊺ denote a vector of p risk factors measured at age a, let {A_ik, k = 0, …, m_i} be the ages at which these risk factors are collected and Z_ij = Z_i(A_ij), where m_i is the total number of measurement times. We assume that Z_i(a) is potentially observable among those with $T_i^{\mathbb{A}}>a$, the values of Z_i(A_ij) are not dependent on the study measurement time A_ij – A_i0 given age A_ij. In addition, C_i is assumed independent of T_i, Z_i(·), entry age A_i0, and the underlying study measurement times, with support not shorter than that of A_ij – A_i0 + τ. Figure 1 provides a graphical illustration of the data structure.

Figure 1.

Data structure.

2.1 Modeling, Estimation and Inference

We are interested in estimating the risk of experiencing the event in the next τ-years for subjects who are at age a and event-free, based on the risk factor measured at age a, Z(a). Thus, the goal is to estimate the conditional risk function

$${\pi }_{\tau ,a}(\mathbf{z})=Pr\{T^{\mathbb{A}}-a\le \tau |T^{\mathbb{A}}>a,\mathbf{Z}(a)=\mathbf{z},A=a\}.$$

To approximate π_τ,a(z), we propose a flexible varying coefficient model

$${\pi }_{\tau ,a}\{\mathbf{Z}(a)\}=g\{{\mathit{\beta }}_{\tau }{(a)}^{⊺}\mathbf{U}(a)\},$$ (1)

where g(·) is a known smooth probability distribution function, such as g(x) = exp(x)/{1 + exp(x)} (which is used in Section 4), U(a) = ψ{Z(a)} represents transformed risk factors for some known function ψ and U(a) includes 1 as the first component, such as a log transformed risk factor (used in Section 4), β_τ(a) is an unknown smooth function representing the covariate effects on the τ-year residual life risk at age a. Model (1) allows the effects of risk factors Z(a) to vary over both age and the residual life span τ. This flexibility is attractive when the risk factors have different effects on long term versus short term risks and when certain risk factor profiles have more detrimental effects for younger subjects than for older subjects.

To estimate β_τ(a) for any given age a in the presence of censoring, we propose to obtain β̂_τ(a) as the solution to the IPW kernel smoothed estimating equation, Φ̂_a(β) = 0, where

$${\widehat{\mathbf{\Phi }}}_a(\mathit{\beta })=\frac{1}{n}\sum _{i=1}^n\sum _{j=0}^{m_i}{\widehat{w}}_{\tau ij}I(X_i^{\mathbb{A}}\ge A_{ij})K_h(A_{ij}-a){\mathbf{U}}_{ij}\{I(X_i^{\mathbb{A}}<A_{ij}+\tau )-g({\mathit{\beta }}^{⊺}{\mathbf{U}}_{ij})\},$$ (2)

where U_ij = U_i(A_ij), ${\widehat{w}}_{\tau ij}={\delta }_{i}I(X_i^{\mathbb{A}}\le A_{ij}+\tau )/\widehat{G}(X_i)+I(X_i^{\mathbb{A}}>A_{ij}+\tau )/\widehat{G}(A_{ij}-A_{i0}+\tau )$ with Ĝ(·) being the Kaplan-Meier estimator for the survival function of the censoring time G(c) = P(C_i > c) of C_i, and K_h(s) = K(s/h)/h is a symmetric standard kernel function K(·) with a finite support and with h the smoothing parameter. Note that under the independent censoring assumption, $E({\widehat{w}}_{\tau ij}|T_i^{\mathbb{A}},A_{ij},Z_{ij},j=0,,\dots ,m_i)\approx 1$.

Following similar arguments as given in Cai and others (2010) and Parast and others (2012), one may show that β̂_τ(a) converges in probability to a deterministic vector β̄_τ(a) as n → ∞ regardless of whether (1) is correctly specified or not. In addition, one may also show that for h = O_p(n^−v) with v ∈ (1/5, 1/2), ${(nh)}^{\frac{1}{2}}\{{\widehat{\mathit{\beta }}}_{\tau }(a)-{\mathit{\beta }}_{\tau }(a)\}$ converges in distribution to a zero-mean normal random vector for any given a. However, it is difficult to directly estimate the asymptotic variance of ${(nh)}^{\frac{1}{2}}\{{\widehat{\mathit{\beta }}}_{\tau }(a)-{\mathit{\beta }}_{\tau }(a)\}$. To construct confidence interval (CI) for β_τ(a) in practice, we suggest using a perturbation resampling (sometimes referred to as wild bootstrap) method (Wu 1986, Park and Wei 2003, Tian and others 2005) to approximate the distribution of the proposed estimator. Compared to the standard bootstrap, perturbation resampling tends to be more stable especially in the survival setting since all observations take positive weights and contribute to the estimation. Let ${\mathcal{V}}^{(b)}=\{V_1^{(b)},\dots ,V_n^{(b)}\}$, b = 1, …, B, be B sets of independent positive random variables from a known distribution with mean and variance equal to one. Then one may obtain perturbed estimates of β_τ(a), ${\widehat{\mathit{\beta }}}_{\tau }^{(b)}(a)$, as the solution to the equation

$$\frac{1}{n}\sum _{i=1}^n\sum _{j=0}^{m_i}{\widehat{w}}_{\tau ij}^{(b)}I(X_i^{\mathbb{A}}\ge A_{ij})K_h(A_{ij}-a){\mathbf{U}}_{ij}\{I(X_i^{\mathbb{A}}<A_{ij}+\tau )-g({\mathit{\beta }}^{⊺}{\mathbf{U}}_{ij})\}=0,$$

where ${\widehat{w}}_{\tau ij}^{(b)}=V_i^{(b)}\{{\delta }_{i}I(X_i^{\mathbb{A}}\le A_{ij}+\tau )/{\widehat{G}}^{(b)}(X_i)+I(X_i^{\mathbb{A}}>A_{ij}+\tau )/{\widehat{G}}^{(b)}(A_{ij}-A_{i0}+\tau )\}$ and Ĝ^(b)(·) is the weighted Kaplan-Meier estimator of G(·) with each subject's contribution to the estimator weighted by $V_i^{(b)}$. The asymptotic variance of β̂_τ(a), ${\widehat{\sigma }}_{{\mathit{\beta }}_{\tau }(a)}^2$, can be estimated by the empirical variance of { ${\widehat{\mathit{\beta }}}_{\tau }^{(b)}(a)$, b = 1, …, B}. The 100(1 – α)% simultaneous confidence bands for {β_τ(a), a_l < a < a_u} can be obtained as {β̂_τ(a) ±ζ_ασ̂_βτ(a)}, where ζ_α is the 100(1 – α)th percentile of { ${sup}_{a_l<a<a_u}|{\widehat{\mathit{\beta }}}_{\tau }^{(b)}(a)-{\widehat{\mathit{\beta }}}_{\tau }(a)|/{\widehat{\sigma }}_{{\mathit{\beta }}_{\tau }(a)}$, b = 1, …, B}. To justify the resampling method, we may first show that

$${(nh)}^{\frac{1}{2}}\{{\widehat{\mathit{\beta }}}_{\tau }(a)-{\mathit{\beta }}_{\tau }(a)\}={(nh)}^{-1/2}\sum _{i=1}^n\sum _{j=0}^{m_i}{K}_h(A_{ij}-a)\mathcal{U}(A_{ij},{\mathbf{U}}_{ij},X_i^{\mathbb{A}})+O_p(h^{\frac{1}{2}})$$

where 𝒰 is some deterministic function and $E\{\mathcal{U}(A_{ij},{\mathbf{U}}_{ij},X_i^{\mathbb{A}})\}=0$. Thus for any fixed a and h = O_p(n^−v) with v ∈ (1/5, 1/2), ${(nh)}^{\frac{1}{2}}\{{\widehat{\mathit{\beta }}}_{\tau }(a)-{\mathit{\beta }}_{\tau }(a)\}$ is asymptotically normal. Furthermore, following similar arguments as given in Tian and others (2005), we may show that with proper normalization, ${sup}_a|{(nh)}^{\frac{1}{2}}\{{\widehat{\mathit{\beta }}}_{\tau }(a)-{\mathit{\beta }}_{\tau }(a)\}|$ converges to an extreme value distribution. In addition,

$${(nh)}^{\frac{1}{2}}\{{\widehat{\mathit{\beta }}}_{\tau }^{(b)}(a)-{\widehat{\mathit{\beta }}}_{\tau }(a)\}={(nh)}^{-1/2}\sum _{i=1}^n\sum _{j=0}^{m_i}{K}_h(A_{ij}-a)\mathcal{U}(A_{ij},{\mathbf{U}}_{ij},X_i^{\mathbb{A}})(V_i^{(b)}-1)+O_{p\ast }(h^{\frac{1}{2}}),$$

where O_p* is with respect to probability space generated by both the observed data and 𝒱^(b). Then we may show that conditional on the data, ${(nh)}^{-1/2}{\sum }_{i=1}^n{\sum }_{j=0}^{m_i}{K}_h(A_{ij}-a)\mathcal{U}(A_{ij},{\mathbf{U}}_{ij},X_i^{\mathbb{A}})(V_i^{(b)}-1)$ converges in distribution to the same limiting unconditional distribution of ${(nh)}^{-1/2}{\sum }_{i=1}^n{\sum }_{j=0}^{m_i}{K}_h(A_{ij}-a)\mathcal{U}(A_{ij},{\mathbf{U}}_{ij},X_i^{\mathbb{A}})$.

2.2 Accuracy Measure Estimation

With β_τ(a) estimated as β̂_τ(a), one can then use π̂_τ,a{Z(a)} = g{β̂_τ(a)^⊺U(a)} to estimate the τ-year survival probability for event-free subjects with risk factor profile Z(a) at age a. To assess the accuracy of the limiting risk model π̄_τ,a{Z(a)} = g{β̄_τ(a)^⊺U(a)} in predicting the τ-year residual survival status for different age groups, we extend commonly used time-dependent accuracy parameters, such as true positive rate (TPR), false positive rate (FPR) and area under the receiver operating characteristic curve (AUC), to also incorporate the age dimension. Since the proposed model evaluation method is not limited to a specific prediction model, we next describe these accuracy parameters for a genetic τ-year residual life risk function, Π_τ,A(Z), derived based on an age A and the risk factor Z(·) collected up to age A. For the proposed model, Π_τ,A(Z) = g{β_τ(A)^⊺U(A)}.

2.2.1 Time and Age Specific Prediction Accuracy

When a specific age group a is of interest, we summarize the prediction performance of risk model Π_τ,A(Z) using time and age specific TPR and FPR functions, respectively defined as

$$\begin{array}{c}{\text{TPR}}_{\tau ,a}(c)=p\{{\mathrm{\Pi }}_{\tau ,A}(\mathbf{Z})>c|0<T^{\mathbb{A}}-a\le \tau ,A=a\},\\ {\text{FPR}}_{\tau ,a}(c)=P\{{\mathrm{\Pi }}_{\tau ,A}(\mathbf{Z})>c|T^{\mathbb{A}}-a>\tau ,A=a\}.\end{array}$$

We may summarize the overall predictiveness of the model for a given a and τ using

$${\text{AUC}}_{\tau ,a}=\int {\text{TPR}}_{\tau ,a}(c)d{\text{FPR}}_{\tau ,a}(c)=P[{\mathrm{\Pi }}_{\tau ,A_i}({\mathbf{Z}}_i)\ge {\mathrm{\Pi }}_{\tau ,A_{i^{\prime }}}({\mathbf{Z}}_{i^{\prime }})|0<T_i^{\mathbb{A}}-a\le \tau ,T_{i^{\prime }}^{\mathbb{A}}-a\ge \tau ,A_i=A_{i^{\prime }}=a],$$

where i and i′ index two independent individuals. Similar to the estimation of β_τ(a), these parameters can be estimated using an IPW kernel smoothing approach. For example, TPR_τ,a(c) can be estimated as

$${\widehat{\text{TPR}}}_{\tau ,a}(c)=\frac{{\sum }_{i,j:0<X_i^{\mathbb{A}}-a<\tau }{\widehat{w}}_{\tau ij}{K}_h(A_{ij}-a)I\{{\widehat{\mathrm{\Pi }}}_{\tau ,A_{ij}}({\mathbf{Z}}_i)\ge c\}}{{\sum }_{i,j:0<X_i^{\mathbb{A}}-a<\tau }{\widehat{w}}_{\tau ij}{K}_h(A_{ij}-a)}.$$ (3)

and AUC_τ,a can be estimated as

$${\widehat{\text{AUC}}}_{\tau ,a}=\frac{{\sum }_{i,j,i^{\prime },j^{\prime }:0<X_i^{\mathbb{A}}-a<\tau \le X_{i^{\prime }}^{\mathbb{A}}-a}{\widehat{w}}_{\tau ij}{\widehat{w}}_{\tau i^{\prime }{j}^{\prime }}{K}_h(A_{ij}-a)K_h(A_{i^{\prime }{j}^{\prime }}-a)I\{{\widehat{\mathrm{\Pi }}}_{\tau ,A_{ij}}({\mathbf{Z}}_i)\ge {\widehat{\mathrm{\Pi }}}_{\tau ,A_{i^{\prime }{j}^{\prime }}}({\mathbf{Z}}_{i^{\prime }})\}}{{\sum }_{i,j,i^{\prime },j^{\prime }:0<X_i^{\mathbb{A}}-a<\tau \le X_{i^{\prime }}^{\mathbb{A}}-a}{\widehat{w}}_{\tau ij}{\widehat{w}}_{\tau i^{\prime }{j}^{\prime }}{K}_h(A_{ij}-a)K_h(A_{i^{\prime }{j}^{\prime }}-a)}.$$ (4)

where Π̂_τ,Aij(Z_i) is the estimated risk function plugging in estimated model parameters. For the proposed varying coefficient model, Π̂_τ,Aij(Z_i) = g{β̂_τ(A_ij)^⊺U_ij} and using similar arguments as those for the consistency of β̂_τ(a), one may show that ${\widehat{\text{AUC}}}_{\tau ,a}$ is a consistent estimator of AUC_τ,a. Furthermore, ${(nh)}^{\frac{1}{2}}({\widehat{\text{AUC}}}_{\tau ,a}-{\text{AUC}}_{\tau ,a})$ converges in distribution to a normal random variable. The CI for AUC_τ,a can be constructed by perturbed estimates. Specifically, for b = 1, …, B, the bth perturbed estimate of ${\widehat{\text{AUC}}}_{\tau ,a}$ can be obtained as

$${\widehat{\text{AUC}}}_{\tau ,a}^{(b)}=\frac{{\sum }_{i,j,i^{\prime },j^{\prime }:0<X_i^{\mathbb{A}}-a<\tau \le X_{i^{\prime }}^{\mathbb{A}}-a}{\widehat{w}}_{\tau ij}^{(b)}{\widehat{w}}_{\tau i^{\prime }{j}^{\prime }}^{(b)}{K}_h(A_{ij}-a)k_h(A_{i^{\prime }{j}^{\prime }}-a)I\{{\widehat{\mathrm{\Pi }}}_{\tau ,A_{ij}}^{(b)}({\mathbf{Z}}_i)\ge {\widehat{\mathrm{\Pi }}}_{\tau ,A_{i^{\prime }{j}^{\prime }}}^{(b)}({\mathbf{Z}}_{i^{\prime }})\}}{{\sum }_{i,j,i^{\prime },j^{\prime }:0<X_i^{\mathbb{A}}-a<\tau \le X_{i^{\prime }}^{\mathbb{A}}-a}{\widehat{w}}_{\tau ij}^{(b)}{\widehat{w}}_{\tau i^{\prime }{j}^{\prime }}^{(b)}{K}_h(A_{ij}-a)k_h(A_{i^{\prime }{j}^{\prime }}-a)}.$$ (5)

where ${\widehat{\mathrm{\Pi }}}_{\tau ,A_{ij}}^{(b)}({\mathbf{Z}}_i)$ is the perturbed counterpart of Π̂_τ,Aij(Z_i) obtained similar to ${\widehat{\mathit{\beta }}}_{\tau }^{(b)}(a)$ using weights 𝒱^(b).

2.2.2 Time Specific Prediction Accuracy Including Age as a Predictor

When interest lies in evaluating the accuracy of the prediction model treating age as a risk factor, an overall summary that is not conditional on age would be preferred. That is, incorporating age as a predictor, one may seek to assess the accuracy in predicting τ-year residual life of the risk estimate Π_τ,A(Z), constructed using Z(·) information collected up to a random age A among those with T^𝔸 > A. For such settings, one may consider

$$\begin{array}{c}{\text{TPR}}_{\tau }(c)=P[{\mathrm{\Pi }}_{\tau ,A}(\mathbf{Z})>c|0<T^{\mathbb{A}}-A\le \tau ]=\int {\text{TPR}}_{\tau ,a}d\mathcal{F}(a)=E({\text{TPR}}_{\tau ,A}),\\ {\text{FPR}}_{\tau }(c)=P[{\mathrm{\Pi }}_{\tau ,A}(\mathbf{Z})>c|T^{\mathbb{A}}-A>\tau ]=\int {\text{FPR}}_{\tau ,a}d\mathcal{F}(a)=E({\text{FPR}}_{\tau ,A}).\end{array}$$

where ℱ is the distribution of the age at measurement. The overall performance of the risk model Π_τ,a(Z) for predicting τ-year residual life can be summarized by

$${\text{AUC}}_{\tau }=P[{\mathrm{\Pi }}_{\tau ,A_i}({\mathbf{Z}}_i)>{\mathrm{\Pi }}_{\tau ,A_{i^{\prime }}}({\mathbf{Z}}_j)|0\le T_i^{\mathbb{A}}-A_i<\tau ,T_{i^{\prime }}^{\mathbb{A}}-A_{i^{\prime }}\ge \tau ]=\iiint {\text{TPR}}_{\tau ,a}(c)d{\text{FPR}}_{\tau ,a^{\prime }}(c)d\mathcal{F}(a)d\mathcal{F}(a^{\prime })=E\{\int {\text{TPR}}_{\tau ,A_i}(c)d{\text{FPR}}_{\tau ,A_{i^{\prime }}}(c)\},$$

where i and i′ index two independent subjects. Plug-in estimates may be constructed for these parameters with an estimated ℱ(·). For example, in the simple case when only a single measurement is taken at baseline, then AUC_τ may be estimated as

$$\begin{array}{c}{n}^{-2}\sum _{i,i^{\prime }}\int {\widehat{\text{TPR}}}_{\tau ,A_{i0}}(c)d{\widehat{\text{FPR}}}_{\tau ,A_{i^{\prime }0}}(c)\\ \approx \frac{{\sum }_{i,i^{\prime }:0<X_i^{\mathbb{A}}-A_{i0}<\tau \le X_{i^{\prime }}^A-A_{i^{\prime }0}}{\widehat{w}}_{\tau i0}{\widehat{w}}_{\tau i^{\prime }0}I[{\widehat{\mathrm{\Pi }}}_{\tau ,A_{i0}}\{{\mathbf{Z}}_i(A_{i0})\}\ge {\widehat{\mathrm{\Pi }}}_{\tau ,A_{i^{\prime }0}}\{{\mathbf{Z}}_{i^{\prime }}(A_{i^{\prime }0})\}]}{{\sum }_{i,i^{\prime }:0<X_i^{\mathbb{A}}-A_{i0}<\tau \le X_{i^{\prime }}^{\mathbb{A}}-A_{i^{\prime }0}}{\widehat{w}}_{\tau i0}{\widehat{w}}_{\tau i^{\prime }0}}.\end{array}$$ (6)

Given the longitudinal data structure, we may also be interested in estimating these accuracy parameters when the age at measurement A follows the marginal distribution of the observed measurement ages in the study. In which case,

$$\begin{array}{c}{\text{TPR}}_{\tau }(c)=P\{{\mathrm{\Pi }}_{\tau ,A_{ij}}({\mathbf{Z}}_{ij})>c|0<T_i^{\mathbb{A}}-{\mathbb{A}}_{ij}\le \tau \},\\ {\text{FPR}}_{\tau }(c)=P\{{\mathrm{\Pi }}_{\tau ,A_{i^{\prime }{j}^{\prime }}}({\mathbf{Z}}_{i^{\prime }{j}^{\prime }})>c|0<T_{i^{\prime }}^{\mathbb{A}}-A_{i^{\prime }{j}^{\prime }}\le \tau \},\end{array}$$

and ${\text{AUC}}_{\tau }=P\{{\mathrm{\Pi }}_{\tau ,A_{ij}}({\mathbf{Z}}_{ij})>{\mathrm{\Pi }}_{\tau ,A_{i^{\prime }{j}^{\prime }}}({\mathbf{Z}}_{i^{\prime }{j}^{\prime }})|0<T_i^{\mathbb{A}}-{\mathbb{A}}_{ij}\le \tau ,T_{i\prime }^{\mathbb{A}}-A_{i^{\prime }{j}^{\prime }}>\tau \}$.

The accuracy measure AUC_τ in this case can be estimated as

$$\frac{{\sum }_{i,j,i^{\prime },j^{\prime }:0<X_i^{\mathbb{A}}-A_{ij}\le \tau <X_{i^{\prime }}^A-A_{i^{\prime }{j}^{\prime }}}{\widehat{w}}_{\tau ij}{\widehat{w}}_{\tau i^{\prime }{j}^{\prime }}I\{{\widehat{\mathrm{\Pi }}}_{\tau ,A_{ij}}({\mathbf{Z}}_i)\ge {\widehat{\mathrm{\Pi }}}_{\tau ,A_{i^{\prime }{j}^{\prime }}}({\mathbf{Z}}_{i^{\prime }})\}}{{\sum }_{i,j,i^{\prime },j^{\prime }:0<X_i^{\mathbb{A}}-A_{ij}\le \tau <X_{i^{\prime }}^{\mathbb{A}}-A_{i^{\prime }{j}^{\prime }}}{\widehat{w}}_{\tau ij}{\widehat{w}}_{\tau i^{\prime }{j}^{\prime }}}.$$

Standard error (SE) and CIs can be constructed similarly to those given above for ${\widehat{\text{AUC}}}_{\tau ,a}$.

2.3 Selection of Smoothing Parameter

It is known that the choice of the smoothing parameter h is critical as in any nonparametric estimation problem. We employ a K-fold cross validation to select the smoothing parameter. Specifically, the study subjects are divided into K folds of approximately equal sizes. The optimal bandwidth h_opt minimizes the weighted mean squared prediction error

$$\sum _{k=1}^K\sum _{i\in {\mathcal{S}}_k}\sum _{j=0}^{m_i}{\widehat{w}}_{\tau ij}{[I(X_i^{\mathbb{A}}<A_{ij}+\tau )-g\{{\mathbf{U}}_{ij}^{⊺}{\widehat{\mathit{\beta }}}_{\tau }^{(-k)}(A_{ij})\left\}\right]}^2,$$

where S_k is the set of subjects that are in fold k and ${\widehat{\mathit{\beta }}}_{\tau }^{(-k)}(A_{ij})$ is the estimate of β_τ(A_ij) using data excluding those from fold k. To obtain an estimator whose variance dominates bias, we follow the common practice to undersmooth (Neumann and Polzehl 1998, Tian and others 2005, Cai and others 2010) using the final bandwidth $h=h_{\mathit{\text{opt}}}{n}_1^{-0.1}$, where n₁ is the number of observed events by τ years.

3 Simulations

In this section, we report results from simulation studies that examine the finite-sample performance of the proposed methods and compare our methods with existing methods. Although the proposed methods are catered for survival data with longitudinally measured risk factors, it can be applied to traditional survival data with one single measurement of the risk factor at baseline to flexibly capture the age effect. For the single measurement setting, we compare the proposed procedure to the standard PH model which includes age as a covariate. In the longitudinal setting, we will compare the proposed methods to the commonly used joint modeling approach. For both settings, we considered n = 2000 and 5000, let K(·) be the Gaussian kernel, and B = 1000 for perturbations. For each configuration, results are summarized based on 1000 simulated datasets.

3.1 Simulations with a Single Measurement

For this setting, we simulate A_i0 from Uniform (15, 75) and two independent baseline covariates Z_i1(A_i0) and Z_i2(A_i0) from N (0, 4) and N (1, 4) respectively. The survival time from entry, T_i, is generated from log(T_i) = 2.5 + β₁(A_i0)Z_i1(A_i0) + β₂(A_i0)Z_i2(A_i0) + 0.5∊, where β₁(a) = a²/7500, β₂(a) = {(a – 45)² – 100}/2000 and ∊ follows a standard logistic distribution. The censoring time C_i is generated from exp(C̃_i) where C̃_i ∼ N (1.6, 0.36), resulting in about 80% of censoring. For illustration, we choose τ to be 5 years. Throughout, we choose g(·) to be the logistic link function. Using the proposed bandwidth selection procedure with 5-fold cross-validation, h is about 6.4 and 5.4 for n = 2000 and 5000 respectively. We obtain the estimates of β_τ(a) and AUC_τ,a for ages from 20 to 70. Throughout the simulation studies, we let U_ij = Z_ij = Z_i(A_ij).

In Figure 2, we present the average of the point estimates, the average of the SE estimates compared with the empirical SEs and the coverage probabilities (CovPs) of the 95% CIs for β_τ(a) and AUC_τ,a across a range of a. The results suggest that the proposed estimators produce negligible biases, and the estimated SEs are close to the empirical SEs. The empirical CovPs of the 95% CIs are close to their nominal level for β_τ(a) coefficients. For AUC, the CovPs of the CIs are close to the nominal level but slightly below 95% for younger ages when n = 2000, possibly due to the fact the curvature of the AUC function is high in that range leading to a slight bias. The results are much improved when n increases to 5000.

Figure 2.

Average of estimates, average of the standard error estimates (ASE), empirical standard errors (ESE) and empirical coverage probabilities (CovP) of the 95% CI. One measurement at baseline. Each entry is based on 1000 simulated samples. The x-axis in all the plots is age.

For comparison, we obtain an alternative τ-year risk estimate, ${\widehat{\mathrm{\Pi }}}_{\tau ,A_{i0}}^{\text{cox}}({\mathbf{Z}}_{i0})=g_{\mathit{\text{cox}}}(log{\widehat{\mathrm{\Lambda }}}_{\tau }+{\widehat{\gamma }}_A{A}_{i0}+{\widehat{\gamma }}_{\mathbf{Z}}^{⊺}{\mathbf{z}}_{i0})$, from fitting a Cox model including A_i0 and Z_i0 = (Z_i1(A_i0), Z_i2(A_i0))^⊺ as covariates, where Λ̂_τ is the estimated baseline cumulative hazard function at τ, g_cox(x) = 1 – e^−ex, and ${({\widehat{\gamma }}_A,{\widehat{\gamma }}_Z^{⊺})}^{⊺}$ are the estimated log-hazard ratio for (A_i0, ${\mathbf{Z}}_{i0}^{⊺}$). For both of the risk estimates from our proposed method and the Cox model, we evaluate their age-specific prediction performance as well as the overall prediction performance based on AUC_τ. As shown in Figure 3, the age specific AUC, AUC_τ,a, of our proposed approach was generally higher than those from the Cox model. The overall AUC, AUC_τ, was about 0.817 for the proposed model and 0.77 for the Cox model. The average difference between the two overall AUC_τ's was 0.047 (SE = 0.007). These results highlight the improved prediction performance for using the proposed age-specific model.

Figure 3.

Age-specific AUCs for the proposed method (Age-specific) and Cox model (Cox). Values are averaged over 1000 repetitions with sample size n = 5000.

3.2 Simulations with Longitudinal Measurements

We also conducted simulation studies to examine the performance of the proposed procedures in longitudinal settings. To simulate the age at the occurrence of event T^𝔸 and longitudinal measurements of a risk factor Z(a), we generate two random effects α₀, α₁ from N(0, 1). The age of event T^𝔸 is obtained from

$$log(T^{\mathbb{A}})=0.5log(-log\in +3)/\{\mathrm{\Phi }({\alpha }_1)+0.5\}+3$$

where ∊ is generated from an uniform distribution over (0, 1) and Φ(·) is the cumulative distribution function of a standard normal distribution. We simulate age of entry to the study A₀ from Uniform (10, 70). Among the subjects who survive by the entry to the study, i.e., T^𝔸 > A₀, we randomly sample n subjects as our cohort. For the i-th subject in this cohort, the survival time since entry is $T_i=T_i^{\mathbb{A}}-A_{i0}$, and C_i is generated from a Uniform (10, 40), which leads to about 25% of censoring. The risk factor is measured at the entry age A_i0 and ages A_ij = A_i0 + Δ_ij after entering the study until event or censored, whichever comes first, where Δ_ij is generated from a N(4, 1) distribution. And the observed marker value at age A_ij is Z_i(A_ij) = α_0i + α_1i log(A_ij) + e_i(A_ij) where e_i(A_ij) ∼ N(0, 1.5²). We choose τ to be 10 years. The selected smoothing parameter h is around 5.0 and 1.3 for n = 2000 and 5000 respectively using 5-fold cross-validation scheme described in Section 2.3. We obtain β_τ(a) and AUC_τ,a estimates for ages from 20 to 60.

The average of the point estimates, average of the SE estimators, the empirical SEs and the coverage probabilities of the 95% CIs for the β_τ(a) and AUC_τ,a at a series of ages are shown in Figure 4. The proposed procedures yield estimators with negligible biases. The estimated SEs obtained through perturbation resampling are close to the empirical SEs.

Figure 4.

Average of estimates, average of the standard error estimates (ASE), empirical standard errors (ESE) and empirical coverage probabilities of the 95% CI. Longitudinal measurements. Each entry is based on 1000 simulated samples. The x-axis in all the plots is age.

For comparison, we also fit the data with a joint modeling (JM) approach (Tsiatis and Davidian 2004, Rizopoulos 2010) for longitudinal and survival data. In particular, we follow the setup in Rizopoulos (2010) to specify a linear mixed effects model with random intercept and slope for the longitudinal measurements Z_ij and a PH model relating the hazard function to the random slope, in which the log baseline hazard is approximated using B-splines. With the parameter estimates from the joint model, for subject i with measurement at A_ij, one can use a Monte Carlo approach to predict τ-year residual life risk given that the person has survived A_ij and we let ${\widehat{\mathrm{\Pi }}}_{\tau ,A_{ij}}^{JM}({\mathbf{Z}}_i)$ denote the resulting estimate of the risk function. We can estimate its corresponding AUC_τ,a as discussed in Section 2.2. We present the average of the estimated AUC_τ,a for the two modeling approaches at n = 5000 in Figure 5. The results suggest that the proposed approach improved prediction accuracy over a wide range of ages compared to the JM approach.

Figure 5.

Age-specific AUC for the proposed method (Age-specific) and JM with longitudinal measurements. Values are averaged over 1000 repetitions with sample size n = 5000.

4 Application

In this section, we apply the proposed methods to the Framingham Heart Study to develop and evaluate age-specific CVD or death risk prediction models. The original goal of the study was to identify the common factors that contribute to CVD by following its development over a long period of time in a large group of participants who had not yet developed CVD. Started in 1948 with 5,209 adult subjects, the study is now on its third generation of participants. Information on a wide spectrum of risk factors and disease outcomes is collected on each of the many followup visits during participants' lifetime. The dataset consists of 3,982 subjects (2,108 females and 1,874 males) with complete information on the risk factors at least one measurement time. Several traditional Framingham risk factors were collected on these subjects on each of their visits, including age, diastolic blood pressure, cholesterol, high-density lipoprotein (HDL), diabetes and smoking. In addition, an inflammation marker, C-reactive protein (CRP), was also measured at various visits. The median number of measurement times is 3. We use both Framingham risk factors and CRP to estimate age-specific 10-year risk of CVD or death for females and males separately. Thus we only include visits where all these risk factors are measured based on the study design. The outcome of interest is time to the onset of first major CVD event or death. Such a composite outcome avoids the issues of having to account for the competing risks from other causes of death. Neverthelss we note that our calculation of the probability of experiencing CVD or death within 10-years in this dataset can be regard approximately as 10-year CVD risk since majority of the observed events are CVD events, especially at younger age. In the study, there are 54 subjects who had CVD prior to death within 10 years. The cumulative incidence rate for CVD prior to death within 10 year is estimated to be about 1.4%. The median follow-up time was 32 years and the entry ages range from 5 to 70 with a median of 35.

We fit the proposed age-specific 10-year risk model (1) with a logistic link. Predictors include original scale of diastolic blood pressure, diabetes, smoking, the log scale of cholesterol, high-density lipoprotein and CRP. We use a gaussian kernel for K(·), with the smoothing parameter h selected as 5.2 for males and 3.8 for females using the 5-fold cross-validation scheme described in Section 2.3. In Figure 6(a) and (b), we demonstrate how the effects of major risk factors may vary by age for women and men respectively. For men, blood pressure does not show any significant association with risk of outcome for all the ages, whereas for women, higher blood pressure significantly increases the risk, especially for younger ages and older ages. For men, as expected, high cholesterol increases risk, and our analysis further reveals that for men, the effects decrease with ages. In other words, having high cholesterol for a younger man expose them to higher risk compared with older men. For women, diabetes status also shows an age-varying pattern: the effect increases with age initially, reaches a peak at 50, and goes down afterward. Smoking shows significant effect at age 45 for women, but the effect diminishes as age increases. However, for men, neither diabetes status nor smoking shows significant association with the risk at almost all ages. HDL shows no significant association with outcome for women, but is inversely associated with the risk between age 40 and 60 and no significant association at other ages for men. Finally CRP also exhibits different age-varying effects between women and men. For men, the effect of CRP is monotonically decreasing as age increases while for women, higher CRP increases risk before 60 years old and peaks at around 45 years old. In Figure 6(c), we show the age-specific AUC of the proposed age-specific risk scores for men and women. For women, the AUC_10,a does not vary substantially over age a with values fluctuating around 0.7. On the contrary, for men, AUC_10,a is substantially higher for younger ages with value as high as 0.9 and decreases to 0.6 at older ages. Thus, the age-specific risk model is highly accurate in predicting 10-year risk of CVD or death for younger males but only moderately accurate for middle aged or older males.

Figure 6.

Data analysis: Framingham Heart Study. The panels (a) and (b) show the point and interval estimates of the age-specific effects of the major risk factors for male and female, respectively. The panel (c) shows the point and interval estimates of the age-specific AUC for male and female respectively.

5 Discussion

When subjects are monitored over time for a clinical condition, it is highly desirable to dynamically recalculate risk estimates according to the updated risk factor information. Age is an important risk factor for many diseases such as CVD and the effects of other predictors on the disease risk may vary over age. Current risk prediction models used in clinical practice such as the FRS often incorporate age as an additive risk factor, which may limit the model prediction performance. Our proposed method estimates the age-specific absolute risk directly via a flexible varying-coefficient model that allows the predictor effects to vary over age and allows for easy incorporation of longitudinally collected risk factor information. We also provide procedures for non-parametrically assessing the prediction performance of such age-specific models, extending existing time-specific accuracy parameters to also incorporate the additional age domain.

Unlike the Cox model with time-varying covariates, our proposed model can easily provide age-specific absolute risk estimates without having to specify the full longitudinal marker processes. Compared to the JM approach, our method has the major advantage of allowing for non-linear effects and non-trivial number of time-varying continuous or discrete risk markers. Additionally our kernel-based procedures allows borrowing information across individuals of similar ages therefore provides a practical solution for situations where the longitudinal information is only measured sparsely and irregularly.

Our method allows for internal covariates in that it aims to make prediction for the residual life among event free subjects at age a, although it does require the availability of the marker information at age a for those with T^𝔸 > a. When the outcome is a non-terminal event that is subject to death as a competing risk, then one may easily modify the proposed procedures to instead make prediction of the disease risk for those who are still alive and have not yet developed the disease at age a. The age-specific accuracy parameters can also be modified to accommodate competing risks similar to those considered in Blanche and others (2013).

The proposed method employs a working model that requires the specification of g and ψ, both of which could potentially impact the model prediction performance. In general, the prediction performance is less sensitive to the choice of g due to the robustness properties such as the logistic likelihood as noted in Li and Duan (1989) and Eguchi and Copas (2002). One may choose appropriate ψ based on existing literature on the functional form of known risk factors or exploratory analyses. On the other hand, regardless of the choice of ψ or g, the fitted model may be mis-specified yet our proposed method could derive a risk model with good prediction performance. The flexible varying coefficient model is expected to perform well under mis-specification and the proposed inference procedures are always valid regardless of the potential mis-specification in the fitted model. In addition, while the proposed simple IPW method has the advantage of enabling robustness in inference under model mis-specification, it may come at a cost in efficiency loss. If there are auxiliary variables available at baseline, efficiency augmentation methods leveraging such information warrants further research.