Competing risk model with a nonparametric form of relative risks

Abstract

Competing risks, referring to multiple mutually exclusive cause-specific events in survival data, are commonly seen in medical and engineering studies. In the presence of covariates, the Cox proportional hazards model is the most common approach to modeling covariate effects as the relative risk component for each cause-specific hazard function. However, the linear form of covariate effects in the existing models can be too restrictive to satisfy in practice. In this work, we propose a Cox regression model for competing risk survival data where the relative risk of each cause-specific hazard takes a flexible nonparametric form. The effects of multiple covariates are modeled under a smoothing spline ANOVA framework allowing for nonparametric effect components among covariates. For each cause-specific hazard, we develop an estimation procedure based on the optimization of penalized partial likelihood. We show that our spline estimators of the log relative risk functions can achieved the optimal nonparametric rates of convergence. Through simulation studies, we demonstrate the numerical performance of the proposed method. We then apply our approach to a multiple myeloma dataset, where the effects of gene expressions on cancer and non-cancer related death are investigated. The nonlinear trend revealed in our analysis offers a strong support to the need of a nonparametric modeling approach like ours.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1007/s10985-026-09728-8.

Introduction

In survival analysis, it is common to observe subjects susceptible to multiple, mutually exclusive causes of failure, giving rise to competing risks. When covariates are present, various statistical models have been developed to account for the structure imposed by competing risks. Among these, the cause-specific hazards model based on the Cox proportional hazards framework, or the competing risk Cox model, is one of the most widely adopted approaches and is treated in detail in Prentice et al. (1978) and Chapter 8 of Kalbfleisch and Prentice (2002). Several alternative or extended approaches of this method have since been proposed. For example, Fine and Gray (1999) proposed a proportional subdistribution hazards model to directly assess the effect of each risk factor on the cumulative incidence function. Lunn and Mcneil (1995) proposed a data augmentation method to model multiple cause-specific hazards simultaneously using standard Cox regression. Modeling of correlated latent cause-specific event times using a copula is presented in Lo and Wilke (2010). For a comprehensive review of methodological developments in competing risks analysis, see Monterrubio-Gómez et al. (2024).

Meanwhile, nonparametric smooth modeling in survival analysis becomes essential when parametric assumptions become doubtful in practice. For example, kernel-based survival methods were considered in López-Cheda et al. (2017, 2023) for cure rate data and Akritas and Keilegom (2003) for bivariate survival data. Penalty-based regression spline methods were studied in Joly et al. (1998), Joly et al. (2002) and Commenges et al. (2014) for single event survival data with various censoring/truncation schemes, Rondeau et al. (2003) for correlated survival data, and Joly and Commenges (1999) for three-state data. Among nonparametric methods, smoothing spline survival models (Gu 2013) provide an especially appealing approach due to its rigorous mathematical foundation in the reproducing kernel Hilbert space framework, straightforward extension to the multi-dimensional setting through tensor product spaces and the functional ANOVA structure, and reliable performance even in estimating multivariate functions. Besides the single event survival models (Gu 2013), smoothing splines based methods have been studied in various other survival settings, such as high dimensional survival data (Du et al. 2010), recurrent event data (Du et al. 2011), and cure rate data (Wang et al. 2012).

Despite the rich literature in their respective areas, an integration of nonparametric smoothing into competing risk models have been rarely studied, with a few exceptions described below. For discrete-time competing risks survival data, Luo et al. (2016) proposed a Cox–logistic regression model of the cause-specific hazards, where only the baseline hazards are estimated nonparametrically by simple quadratic or cubic spline expansions but the covariate effects are modeled linearly. Fermanian (2003) proposed a joint distribution approach based on kernel methods. In particular, the joint survival function of the competing failure times is assumed to have a form defined by three components, an unknown cumulative distribution function, a set of cause-specific time functions, and a set of cause-specific covariate functions. Various constraints on the component functions, such as boundary conditions, monotonicity, and differenitability are necessary to make the model identifiable. Under these constraints, these three components are estimated separately by kernel methods. Despite its nonparametric nature, this method can be cumbersome to implement and its estimates are hard to interpret in practice due to its complex structure.

In this paper, we consider a nonparametric competing risk model based on the smoothing spline ANOVA framework. Instead of linear forms of covariate effects adopted in most parametric or semiparametric competing risk models, we consider cause-specific hazards with the true covariate effect functions assumed to lie in a tensor-product Sobolev space. This nonparametric formulation allows for flexible inclusion and decomposition of covariate effects. The estimation is through the optimization of a penalized partial likelihood consisting of the negative log partial likelihood representing the goodness-of-fit, a roughness penalty enforcing the smoothness constraint, and a smoothing parameter balancing the tradeoff. For the inference purpose, we consider the point-wise confidence intervals based on Baysian interpretation of the penalized partial likelihood. A smoothing parameter selection criterion based on the Kullback–Leibler distance is introduced. For theoretical properties, we establish the rates of convergence for the function parameters in the proposed model. The model is evaluated through simulations, and then applied to a multiple myeloma dataset studying the effect of genes on cause-specific hazards. The analysis reveals nonlinear and distinct effects across risk factors, demonstrating the advantage of the proposed model over existing methods that often neglect competing risks or rely on linear covariate effects. As one of the first nonparametric competing risk models, our method has the following advantages: (1) flexible model assumptions via the functional ANOVA decomposition without linearity restriction; (2) theoretically consistent estimation with the optimal convergence rate for nonparametric functions; (3) numerical stability with a convex objective for optimization; (4) efficient computation with a manageable dimension of parameter space; and (5) straightforward practical interpretation via the functional ANOVA structure and proper confidence interval estimates.

For the remainder of the paper, Sect. 2 presents the model formulation, function space structure, estimation procedure, and theoretical result. Section 3 provides simulation studies for the proposed model. Section 4 presents a real data application. Finally, Sect. 5 concludes the paper with a discussion of future research directions.

Smoothing spline competing risk cox model

Model structure

Let and be respectively the underlying event time and non-informative right-censoring time, with representing the ending time of the study. Let be a vector of d covariates. For a categorical covariate, is a finite set . For a continuous covariate, we set without loss of generality. Suppose there are K mutually exclusive risks, denoted by , that can be the cause of an event. Let represent the underlying risk associated with the event. We assume that a single subject can experience at most one event, and that and C are conditionally independent given X. Let be the observed time and be the censoring indicator. Let be the observed risk indicator. Note that G is 0 when the subject is censored and is the risk type otherwise.

Let be the underlying data, which are n independent and identically distributed copies from . The observations are then copies from . In competing risk models, for , the hazard function of given X is

where is the cause-specific hazard function of the k-th risk factor. The survival function is given by and the distribution function is then calculated as . The density function is , where . The distribution function can also be expressed as , where is the cumulative incidence function of the k-th risk factor. Define , where .

For each k, let denote the true covariate effect function for the k-th risk. We assume the proportional cause-specific hazards model, that is, the cause-specific hazard functions follow the Cox model

where is the baseline cause-specific hazard function, and .

Reproducing Kernel Hilbert spaces

The unknown covariate effect function is assumed to belong to a reproducing kernel Hilbert space (RKHS) of multivariate functions defined on the product domain of the covariates. For each covariate, the marginal space is constructed first according to the covariate type (e.g., continuous or categorical). Then these marginal spaces for all the covariates are assembled to form through a tensor product of spaces. This construction admits a decomposition of into effect subspaces corresponding to main and interaction effects, analogous to the classical ANOVA decomposition. The functional ANOVA represents as a sum of interpretable components, enabling separate estimation and inference for each effect.

For the jth covariate, let be the RKHS of functions defined on . For a discrete covariate, and it is easy to see that (Gu 2013), where the covariate effect is represented by level-specific parameters. For a continuous covariate, and is the Sobolev space of order , defined as

The marginal space can be decomposed into a constant subspace and a centered effect subspace through an averaging operator. The idea is similar to the zero sum constraint in the classical ANOVA where the average of all the treatment effects is assumed to be zero. Let be the averaging operator for . When , When , . Then can be decomposed as where

and . Thus, each marginal space is decomposed into a constant component and a centered effect component .

The tensor product Sobolev space on can be written as

where denotes the power set of and the components in the decomposition respectively represent the constant, the main effects, the two-way interactions, and so on. For each , the associated effect space is

For example, is the constant space, corresponds to the main effect of covariate j, while corresponds to the interaction effect between covariates j and j'.

In practice, not all effect spaces need to be included in the model. Let for some nonempty . Note that, since the Cox model does not include an intercept, is excluded from . A further selection of specifies the structure of . For instance, if , then adopts an additive form that involves only main effects. On the other hand, choosing allows to incorporate all the main effects and two-way interactions. For each risk k, we assume , and let its decomposition be where for every .

Next, we introduce the roughness penalty induced by . Given a semi-inner product on , can be decomposed as , where the null space forms a finite-dimensional subspace of , and forms an infinite-dimensional subspace of . Furthermore, is an inner product on , whose associated reproducing kernel (RK) is . For example, when d=1 and becomes a space of univariate functions, the squared-curvature penalty yields cubic smoothing splines. To distinguish the spaces used for different competing risks, we use and to represent the roughness penalty and its associated reproducing kernel for the estimation of .

Example 1

Consider d=2, , , and m=2. This corresponds to the scenario of two continuous covariates, a model with all the main and interaction effects, and a marginal squared-curvature penalty.

Let be the l-th scaled Bernoulli polynomial on [0, 1]. For instance, , , and . The null space , consisting of low-order polynomial components that are not penalized by . For and in , and ,

where , , and so on for the higher order derivatives. Here, are weighting parameters for subspaces which controls the relative amount of smoothing imposed on different subspace components.

Example 2

When and , we have one categorical covariate and one continuous covariate, and . For and in , and ,

where id is the identity mapping on .

Martingale structure

Here, we derive the martingale structure for our model, based on Fleming and Harrington (1991) and Kalbfleisch and Prentice (2002). Let . For each risk k, the counting process is defined as and the at-risk process is defined as For each subject i, and are defined in the same manner.

The filtration at t and are each defined as

where the σ -algebra generated by · is denoted by .

As and are conditionally independent given , we have

where . By the Doob-Meyer decomposition, we have where is a zero-mean martingale and is the compensator. The predictable variation process and covariation process are calculated respectively as and , where

where we regard dt · dt as 0. Furthermore,

for due to the continuity of the random variables (at most one jump at t among all n subjects) and the fact that a single subject cannot experience more than two risks.

Estimation

We now introduce the estimation procedure for the proposed model. For , the negative log-partial likelihood for the k-th risk is

To represent f, traditional smoothing splines require a full basis of spline functions with knots at all the observations. When n is large, this can generate significant computational burden. To address this issue, Kim and Gu (2004) proposed to use only observations as knots while preserving the same convergence rate as the full basis expansion. For each k, we randomly select subjects to serve as knots, so that the knot locations represent the overall covariate distribution without favoring particular subjects. Let their covariates be . Then, forms a finite-dimensional subspace of , which further defines , a finite-dimensional subspace of . Note that and the selected subjects can be different for each k. For each risk k, using the efficient approximation (Kim and Gu 2004), we define the estimator of to be

1

where is a roughness penalty and is a smoothing parameter. When , we have by the representer theorem.

The null space generally consists of lower-degree polynomials and is finite dimensional. Let be the basis functions of . Then estimator can have the expression . And the optimization in (1) becomes

2

where is the kernel matrix.

Here, two modeling choices are required. The first is the order m of for continuous covariates, where a larger m corresponds to smoother spline functions. A common choice is m=2 (Gu 2013), corresponding to cubic smoothing splines. The second is the number of knots . As shown in Kim and Gu (2004), is sufficient to guarantee the same convergence rate of the function estimate when tensor product cubic splines are used.

Since grows at a slower rate than n, the computational complexity of the proposed efficient approximation estimator in (2) is , which is substantially lower than the complexity of the full-knot estimator even for large sample sizes, supporting the computational efficiency emphasized in the introduction. In addition, the objective function in (2) is convex with a quadratic roughness penalty, yielding a well-behaved optimization problem that can be solved reliably using standard numerical procedures.

Bayesian confidence intervals

We now introduce Bayesian pointwise confidence intervals for the functional parameters in our model. The name “Bayesian" was coined by Wahba (1983) since the roughness penalty can be treated like a Gaussian prior, making the penalized likelihood interpretable as the posterior likelihood and its optimizer as the posterior mode in Bayesian analysis. For each k , define and the hessian matrix for (2) as

where , , , , and is a p× p zero matrix. Then the confidence interval for at is given by

where is the (1 - α /2)-quantile of the standard normal distribution.

Smoothing parameter selection

We propose a criterion for smoothing parameter selection. For each k, let represent the number of subjects who failed due to the k-th risk factor. Let be the subset of covariates out of n that satisfy .

Using the relative Kullback–Leibler distance (RKL) proxy for leave-one-out cross validation, as outlined in Du et al. (2010), we select the smoothing parameter that minimizes

3

where , is identity matrix, is length one vector, , and α >1 is a preselected constant to guard against under-smoothing. We set following the recommendation in Gu (2013).

Rates of convergence

We now present the rate of convergence result for our estimator in (1). For notational simplicity, we assume for all j, noting that any additional categorical domain only adds a finite dimensional component space which does not affect the convergence rates derived here. The notation and the assumptions are provided in Sects. A and B of the Appendix, respectively.

For each k, define a non-negative definite bivariate functional on by

for , where , , and for and . Let for .

Theorem 1

Under Assumption B.1 in Sect. B of the Appendix, for each k, as , we have

The proof of Theorem 1 is provided in section S1 of the Supplementary. One can show that Theorem 1 implies , where represents the norm. Here r > 1 characterizes the model parameter space . For example, r = 2m when d = 1 or d > 1 with an additive model; and r = 2m - δ when is a tensor product space for a model with interactions, where δ >0 can be arbitrarily small. See Gu (2013) for additional details. The constant represents the smoothness level of the true function , with corresponding to being super-smooth. The optimal rate of convergence is , achievable by setting . Note that this rate coincides with the optimal convergence rate for nonparametric function estimation in general.

Simulation study

Simulation study for additive models

We perform a simulation study for the proposed method under additive covariate effects. We set K = 2, m=2, d = 3, and impose an additive structure on ; that is, for each k, we assume , where

Note that some effect functions are adopted from Cho and Liu (2026). In addition, some effect functions are linear, reflecting the findings observed in the real data application presented in Sect. 4.

We assume a Weibull baseline cause-specific hazard functions; that is, for each k, . We set and . Similarly, the hazard function for the underlying censoring time is also assumed to be Weibull; that is, for a random variable , we have , where and . We set τ = 1.

Let be a multivariate normal random variable with mean vector and covariance matrix . We have , where \varPhi denotes the distribution function of the standard normal distribution. Each follows a uniform distribution on [0, 1] and are dependent.

Considering that for each k, we generate simulation data following Beyersmann et al. (2009). Specifically, for each subject i, (1) generate ; (2) generate ; (3) generate from ; (4) generate from

(5) compute , , and .

For the simulation setting, we vary the censoring levels by and the sample size , where smaller or smaller induces a heavier censoring rate. We repeat the simulation over 1000 replicates, where, for each replicate, we compute for each k. For each simulation setting, we evaluate the performance of the proposed method across all replicates, focusing on the rate of convergence and the Bayesian confidence interval.

Table 1 presents the summary of simulated data by settings. One can observe that events are approximately equally likely to occur for the two risks and that smaller values of or result in a higher proportion of censored observations.

Table 1.

The first subtable reports the simulated data summaries by settings, and the second subtable reports the losses by simulation settings

n			Data summary
250	0.8	0.7	134.89 (7.92)	58.18 (6.45)	56.93 (6.72)
	1.2	1.1	101.55 (8.05)	75.16 (7.29)	73.28 (7.16)
	1.6	1.5	80.19 (7.33)	86.20 (7.65)	83.62 (7.56)
500	0.8	0.7	269.81 (11.27)	115.78 (9.26)	114.41 (9.40)
	1.2	1.1	203.51 (10.67)	149.65 (10.46)	146.84 (10.26)
	1.6	1.5	160.77 (10.91)	171.12 (11.02)	168.11 (10.85)
1000	0.8	0.7	539.63 (16.17)	231.99 (13.36)	228.38 (13.66)
	1.2	1.1	407.71 (15.64)	299.42 (14.44)	292.87 (14.13)
	1.6	1.5	321.45 (14.90)	342.70 (15.44)	335.85 (15.12)

n
			j=1		j=2		j=3
			k=1	k=2	k=1	k=2	k=1	k=2
250	0.8	0.7	0.23 (0.11)	0.34 (0.15)	0.29 (0.15)	0.20 (0.18)	0.18 (0.15)	0.29 (0.16)
	1.2	1.1	0.20 (0.10)	0.30 (0.13)	0.25 (0.11)	0.18 (0.14)	0.16 (0.14)	0.25 (0.12)
	1.6	1.5	0.20 (0.09)	0.28 (0.11)	0.23 (0.10)	0.16 (0.12)	0.16 (0.12)	0.24 (0.11)
500	0.8	0.7	0.17 (0.08)	0.25 (0.10)	0.20 (0.09)	0.14 (0.11)	0.13 (0.09)	0.20 (0.09)
	1.2	1.1	0.15 (0.06)	0.22 (0.08)	0.18 (0.07)	0.12 (0.09)	0.11 (0.09)	0.18 (0.07)
	1.6	1.5	0.14 (0.06)	0.21 (0.08)	0.17 (0.07)	0.11 (0.08)	0.10 (0.08)	0.17 (0.07)
1000	0.8	0.7	0.12 (0.05)	0.19 (0.07)	0.15 (0.05)	0.10 (0.08)	0.09 (0.07)	0.15 (0.06)
	1.2	1.1	0.11 (0.04)	0.17 (0.06)	0.13 (0.05)	0.08 (0.07)	0.08 (0.06)	0.13 (0.05)
	1.6	1.5	0.10 (0.04)	0.16 (0.06)	0.12 (0.05)	0.08 (0.06)	0.08 (0.06)	0.12 (0.05)

Each cell shows the mean (standard deviation) over 1000 replicates

Estimation performances are reported in Table 1, while both estimation performances and the Bayesian confidence interval performances are shown in Figs. 1, 2, and 3 and Figs. S1–S6 in section S3 of the Supplementary. As expected, larger sample sizes or lower censoring levels yield improved estimation accuracy. Larger sample sizes also lead to narrower confidence intervals and generally better coverage rates. The proposed Bayesian confidence intervals align closely with empirical quantiles across replicates, demonstrating their reliability.

Fig. 1.

Simulation results for . For each simulation setting and each risk k, the top panel shows (solid line), average of (dashed line), average of the 95% Bayesian confidence interval for (dotted lines), and 2.5 and 97.5% empirical quantiles of (shaded area). The bottom panel displays the empirical coverage rate of the 95% Bayesian confidence interval for . Note that all averages and empirical quantiles are computed point-wise across all replicates

Fig. 2.

Simulation results for . For each simulation setting and each risk k, the top panel shows (solid line), average of (dashed line), average of the 95% Bayesian confidence interval for (dotted lines), and 2.5 and 97.5% empirical quantiles of (shaded area). The bottom panel displays the empirical coverage rate of the 95% Bayesian confidence interval for . Note that all averages and empirical quantiles are computed point-wise across all replicates

Fig. 3.

Simulation results for . For each simulation setting and each risk factor, the top panel shows (solid line), average of (dashed line), average of the 95% Bayesian confidence interval for (dotted lines), and 2.5 and 97.5% empirical quantiles of (shaded area). The bottom panel displays the empirical coverage rate of the 95% Bayesian confidence interval for . Note that all averages and empirical quantiles are computed point-wise across all replicates

Simulation study for models with interactions

We conduct a simulation study to evaluate the finite-sample performance of the proposed method in the presence of interaction effects. We set K=2, m=2, and d=2, and assume a tensor-product structure on . Specifically, for each k=1,2, we consider

The simulation study encompass a variety of effect structures, including nonlinear, linear, and categorical main effects, as well as nonlinear-by-nonlinear, nonlinear-by-categorical, and linear-by-categorical interactions. Several effect functions are adopted from Cho and Liu (2026). We consider two tensor-product effect structures.

Tensor-Product Structure 1. Let . Both covariates are continuous. The effect functions are specified as

This structure consists of two continuous covariates. All main and interaction effects are nonlinear except for , which is linear.

Tensor-Product Structure 2. Let and , where the second covariate is binary. The effect functions are specified as

This structure contains a nonlinear main effect, a categorical main effect, and a nonlinear-by-categorical interaction for Risk 1. For Risk 2, it contains a linear main effect, a categorical main effect, and a linear-by-categorical interaction.

Except for the tensor-product structure and the effect functions, the simulation data for structures 1 and 2 are generated in the same manner as in Sect. 3.1, with d=2 instead of d=3. We consider (1.6,1.5)\} and sample sizes . For each configuration, 1,000 replicates are generated.

Computational time was evaluated under Structure 1 with n=250 and 500, , and 50 replicates. All simulations were conducted on an Apple Mac equipped with an Apple M1 processor and 16 GB RAM. The average computational times were 17 seconds for n=250 and 97 s for n=500.

Tables S1–S4 in Section S3 of the Supplementary summarize the simulation results for Tensor-Product Effect Structures 1 and 2, respectively, including data summaries, estimation performances, and the coverage probabilities and lengths of the Bayesian confidence intervals. Overall, the proposed method performs well across a variety of effect structures. As the sample size n increases, for many cases, the estimation accuracy improves, the Bayesian confidence intervals become narrower, and the empirical coverage probabilities increases. As the censoring rate decreases, estimation accuracy improves in many cases, although the improvement is generally less pronounced than that resulting from an increase in the sample size n. Although the losses may differ across effects, this is largely attributable to differences in the scales of the underlying effect functions. In addition, linear, categorical, and interactions involving categorical covariates often exhibit better coverage performance than nonlinear effects, which is expected due to their lower complexity. Nevertheless, the proposed method demonstrates satisfactory estimation and the Bayesian interval inference performance across many tensor-product effect structures.

Real application

In this section, we analyze the Multiple Myeloma Research Foundation (MMRF) CoMMpass study, which provides clinical trial data and gene expression data, publicly available through the Genomic Data Commons (GDC) portal. Multiple myeloma is a type of blood cancer that originates in plasma cells and affects the bone marrow and immune function. The dataset includes, for each subject, time-to-event or censoring, a censoring indicator, cause of death (categorized as multiple myeloma-related or non-cancer-related), age, and gene expression profiles for various genes.

Most previous studies on time-to-event modeling in multiple myeloma have either used linear survival models (Simhal et al. 2023; Zhang et al. 2023), which focus on variable selection or interpretation but cannot accommodate nonlinear effects; or machine learning approaches (Shah et al. 2025; Ren et al. 2023), which aim for predictive performance but lack interpretability, especially with respect to how each covariate nonlinearly influences the event time. The study by Zhao et al. (2024), which focuses on survival modeling with omics covariates using The Cancer Genome Atlas (TCGA) data—also available through the GDC portal and formatted similarly to the MMRF CoMMpass data—mentions competing risks modeling as a potential approach for analyzing multiple causes of death, such as cancer-related and other-cause mortality. This suggestion motivates our use of competing risks modeling in the MMRF CoMMpass data analysis.

Despite the richness of the MMRF CoMMpass study, existing literature lacks nonparametric modeling frameworks that can flexibly capture nonlinear covariate effects, as well as jointly model multiple causes of death. Our proposed method addresses both limitations to analyze the MMRF CoMMpass data.

Under a competing risks framework, following previous studies on multiple myeloma (Shah et al. 2025; Simhal et al. 2023; Zhang et al. 2023), we focus on the effect of gene expression as covariates. Note that age is also an important factor (Zhao et al. 2024). After excluding subjects with missing age information, the numbers of observations in the censored group and two risk groups are respectively 577, 83 (multiple myeloma-related deaths), and 66 (non-multiple myeloma-related deaths).

The original gene expression data contain measurements on 60,660 genes. We preprocess the gene expression data by a logarithmic transformation and a scaling of each variable to [0, 1]. To reduce dimensionality, we first select the top 25 genes with the highest standard deviation among those with less than 10% missing values.

From this subset, with age (j=1) included as a covariate, gene covariates were added sequentially according to decreasing values of , where is the kth cause-specific log relative risk estimate from the proposed method with one covariate in the model at a time. Covariates were added until in (3), which serves as a criterion analogous to a cross-validation error (Gu 2013), began to increase. The value of decreases up to d=6 and increases thereafter. Therefore, we selected d=6 covariates. The selected genes are RPS6KA6 (j=2), IGKC (j=3), UCHL1 (j=4), DSG2 (j=5), and RPS4Y1 (j=6), listed in decreasing order of .

The final dataset, after excluding subjects with missing values for any of the three selected genes, consists of 498 censorings, 67 multiple myeloma-related deaths, and 56 non-multiple myeloma-related deaths, with d=6 covariates, consisting of age and the selected gene expression covariates described above. We fit our proposed model to this dataset, assuming an additive structure on and setting m=2.

In Figs. 4 and 5, we observe that for many genes, the fitted curves for multiple myeloma-related death and non-multiple myeloma-related death exhibit notably different shapes. This suggests the necessity of competing risk modeling for these risk factors. Interestingly, the effect of age differed between the cancer and non-cancer groups. In the non-cancer group, age exhibited a significant linear effect, whereas in the cancer group the estimated effect showed a non-significant nonlinear pattern, decreasing initially and increasing thereafter. This difference is clinically plausible, as age is a well-established risk factor for non-cancer mortality, whereas cancer-related mortality in multiple myeloma is likely driven more strongly by the underlying molecular and genetic characteristics of the disease than by age alone.

Fig. 4.

MMRF data analysis results. For each row, the title represents the covariate and the subtitles correspond to the associated risks. For each plot, the dashed line represents and the dotted lines indicate the 95% Bayesian confidence interval for . Note that the domains are in the original scales of gene expressions for better interpretability

Fig. 5.

MMRF data analysis results. For each row, the title represents the covariate and the subtitles correspond to the associated risks. For each plot, the dashed line represents and the dotted lines indicate the 95% Bayesian confidence interval for . Note that the domains are in the original scales of gene expressions for better interpretability

Additionally, for four genes, namely RPS6KA6, IGKC, UCHL1, and RPS4Y1, evidence of nonlinearity is observed for at least one of the two risks, namely multiple myeloma-related death or non-multiple myeloma-related death. For example, the fitted effect for multiple myeloma-related death in RPS6KA6 and IGKC, as well as the fitted effects for non-multiple myeloma-related death in RPS6KA6, IGKC, and UCHL1, exhibit nonlinear patterns. Moreover, the corresponding Bayesian confidence intervals exclude zero in certain regions of the covariate domain, suggesting significant nonlinear associations with the corresponding cause-specific hazards. Although the Bayesian confidence intervals for non-cancer death group in RPS4Y1 are relatively wide, its fitted curve still displays nonlinearity. These findings are not reported in the existing literature.

Overall, the analysis demonstrates that for several covariates, at least one cause-specific effect is nonlinear and varies across different causes of death. This highlights that the proposed model fills a gap in the existing literature, where competing risks are often overlooked or covariate effects are assumed to be linear. Furthermore, the estimated effect functions and their Bayesian confidence intervals provide a straightforward interpretation of covariate effects on the corresponding cause-specific hazards, including their direction, significance, and potential nonlinearity.

Conclusion

In this study, we introduced the smoothing spline competing risk Cox model, which combines the cause-specific Cox model with smoothing spline ANOVA to capture nonparametric covariate effects in competing risks. The proposed model bridges a gap in the literature by estimating nonlinear and distinct cause-specific covariate effects under the proportional cause-specific hazards assumption. For each risk, the estimation procedures are proposed, and the rate of convergence results are established. Our simulations and application to a multiple myeloma dataset demonstrate that the proposed model can estimate distinct effect functions across risk factors, in the presence of nonlinearity.

For future work, we plan to apply the smoothing spline ANOVA framework to various competing risk survival models. The proposed method in this paper combines the classical and fundamental proportional cause-specific hazards model with smoothing spline ANOVA. However, we aim to extend this framework by integrating alternative approaches to proportional cause-specific hazards modeling with smoothing spline ANOVA. For instance, we could combine the proportional subdistribution hazards model (Fine and Gray 1999) with smoothing spline ANOVA or integrate correlated latent cause-specific event times model by copula structure (Lo and Wilke 2010) with smoothing spline ANOVA.

There are a few other interesting directions that merit future research. One is the estimation and asymptotic analysis of the baseline hazard function. Under the proportional hazards framework, the baseline hazard does not enter the estimation of the covariate effects and can subsequently be estimated using a Breslow-type estimator (Breslow 1972). Although we do not investigate its asymptotic properties in the present work, such results are of practical interest and may be established using techniques in Kosorok (2008). The other is to determine whether a covariate effect is linear or nonlinear. As demonstrated in our application, visual inspect assisted by confidence intervals provides a simple way to identify nonlinear effects from linear ones. A more rigorous approach for this purpose would be an extension of the LAND method (Zhang et al. 2011) to the competing risk model setting.

Supplementary Information

Below is the link to the electronic supplementary material.

Notations

We introduce the notations used throughout our theory. We write for the distribution of W, and for the distribution of . We use the operator to indicate the Lebesgue integral taken with respect to the distribution of either W or . That is,

for any function g defined on or , where . The function g may be deterministic or a stochastic process. If g is deterministic,

For deterministic sequences for , we set () if there exists a constant and such that for all n ≥ N. Likewise, () is satisfied if for every constant , there exists such that for all . We adopt to mean that and .

For sequences for , with at least one being random, we refer to () if there is a constant such that as . In addition, () if as for any constant .

Here, if a constant does not have a subscript k, it is independent of k.

Let and be vector spaces that are subspaces of a vector space . We refer to , which can also be expressed as or , if the followings are satisfied:

1. ,

2. For every , there exist unique and such that .

For each j=1,… ,d, suppose that forms a basis of Hilbert spaces , where index set is either finite (when is finite-dimensional, ) or countably infinite (when is infinite-dimensional, ). We denote tensor product space by , which is the completion of with respect to some inner product meeting for all , j=1,… ,d, where for each j=1,… ,d, is some inner product on .

Assumptions

In this section, we provide assumptions needed in the rate of convergence for the proposed model. As in Sect. 2.7, we here assume for all j. Let be the density function of X . For and , let and .

Assumption B.1

Assume the followings:

1. There exist some constants and such that and for each k.

2. There exists some constant such that for all k, .

3. There exist some constants such that for all k, for all .

4. There exist some constants such that for all k, for all and all .

5. There exists some constant such that for all k, for all . Furthermore, for each k, there exists deterministic big convex neighborhood of , denoted by with some constant , which does not depend on n, such that there exist some constants such that for all , for all .

6. For each k , is completely continuous with respect to on , implying that there exists an eigensystem such that for all , the eigenfunction and the eigenvalue satisfy as , and , while for all . Furthermore, for any , we have , since for all , J(f) < ∞ , which is equivalent to .

   There exist some constants such that for each k , and as , where r>1. Note that r = 2m when d = 1, or when d > 1 and has an additive structure; and r = 2m - δ for any when d > 1 and contains at least one interaction.

7. For each k, there exists some constant such that .

8. For each k, there exists some constant such that as , , , and .

Let . By the strong law of large numbers and (1), which is common in the survival analysis literature, we have as . Given that n(· ) is non-increasing on [0, τ ], there are sufficient number of units at risk for any . Therefore, for simplicity, we assume , which is reasonable based on the assumption above.

Note that (2) indicates that the probability of each risk factor is above a certain threshold. This implies that, when n is sufficiently large, for each risk factor, we have a sufficiently large number of subjects with observed risk corresponding to that risk factor.

The uniformly boundedness assumptions in (3) and (4) are borrowed from Du et al. (2010). Specifically, (3) leads to . Observe that (4) is a uniformly boundedness condition used to show that Theorem 1 implies rate of convergence with respect to the norm.

The set in (5) is large enough to ensure that all estimators of in the technical proofs, including and , are elements of . This is a common assumption in the smoothing spline literature (Du et al. 2010; Gu 2013).

Constructing the eigensystem on for each k , as described in (6), is a standard approach in the smoothing spline literature (Du et al. 2010; Gu 2013).

The smoothness level of the true function is described by the parameter . When , we have , ensuring that condition (7) is automatically satisfied by .

The order of the number of knots required for efficient approximation, as well as the order of the smoothing parameter in relation to the number of knots, are discussed in (8). Both are common assumptions in the smoothing spline literature (Du et al. 2010; Gu 2013). When as , which guarantees the optimal convergence rate, as is also satisfied, thereby ensuring that (8) holds.

Declarations

Conflict of interest

The authors declare that they have no competing financial or non-financial interests that are directly or indirectly related to the work submitted for publication.