Large-Scale Parametric Survival Analysis

Abstract

Survival analysis has been a topic of active statistical research in the past few decades with applications spread across several areas. Traditional applications usually consider data with only small numbers of predictors with a few hundreds or thousands of observations. Recent advances in data acquisition techniques and computation power has led to considerable interest in analyzing very high-dimensional data where the number of predictor variables and the number of observations range between 10⁴ – 10⁶. In this paper, we present a tool for performing large-scale regularized parametric survival analysis using a variant of cyclic coordinate descent method. Through our experiments on two real data sets, we show that application of regularized models to high-dimensional data avoids overfitting and can provide improved predictive performance and calibration over corresponding low-dimensional models.

1. Introduction

Regression analysis of time-to-event data occupies a central role in statistical practice [1, 2] with applications spread across several fields including biostatistics, sociology, economics, demography, and engineering [3, 4, 5, 6]. Newer applications often gather high-dimensional data that present computational challenges to existing survival analysis methods. As an example, new technologies in genomics have led to high-dimensional microarray gene expression data where the number of predictor variables is of the order of 10⁵. Other large-scale applications include medical adverse event monitoring, longitudinal clinical trials, and business data mining tasks. All of these applications require methods for analyzing high-dimensional data in a survival analysis framework.

In this paper we consider high-dimensional parametric survival regression models involving both large numbers of predictor variables as well as large numbers of observations. While Cox models continue to attract much attention [7], parametric survival models have always been a popular choice among statisticians for analyzing time-to-event data [4, 6, 8, 9]. Parametric survival models feature prominently in commercial statistical software, are straightforward to interpret and can provide competitive predictive accuracy. To bring all these advantages to high-dimensional data analysis, these methods need to be scaled to data involving 10⁴ – 10⁶ predictor variables and even larger numbers of observations.

Computing the maximum likelihood fit of a parametric survival model requires solving a non-linear optimization problem. Standard implementations work well for small-scale problems. Because these approaches typically require matrix inversion, solving large-scale problems using standard software is typically impossible. One possible remedy is to perform feature selection as a pre-processing step. Although feature selection does reduce memory and computational requirements and also serves as a practical solution to overfitting, it introduces new problems. First, the statistical consequences of most feature selection methods remain unclear, making it difficult to choose the number of features for a given task in a principled way. Second, the most efficient feature selection methods are greedy and may choose redundant or ineffective combinations of features. Finally, it is often unclear how to combine heuristic feature selection methods with domain knowledge. Even when standard software does produce estimates, numerical ill conditioning can result in a lack of convergence, large estimated coefficient variances and poor predictive accuracy or calibration.

In this paper, we describe a regularized approach to parametric survival analysis [10]. The main idea is the use of a regularizing prior probability distribution for the model parameters that favors sparseness in the fitted model, leading to point estimates for many of the model parameters being zero. To solve the optimization problem, we use a variation of the cyclic coordinate descent method [11, 12, 13, 14]. We show that application of this type of model to high-dimensional data avoids overfitting, can provide improved predictive performance and calibration over corresponding low-dimensional models, and is efficient both during fitting and at prediction time.

In Section 2, we describe some of the other work that addresses similar problems. In Section 3, we describe the basics of regularized approach to parametric survival analysis. In section 4, we briefly describe the four parametric models used for survival analysis using the high-dimensional formulation. We also describe our optimization technique which is tailored for each specific model for computing point estimates of model parameters. More involved algorithmic details of the method pertaining to each of the four parametric models can be found in Appendix A. We describe the data sets and methods that we use in our experiments in Section 5, and provide experimental results in Section 6. The first application uses a large data set of hospitalized injured children for developing a model for predicting survival. Through our experiments, we establish that an analysis that uses our proposed approach can add significantly to predictive performance as compared to the traditional low-dimensional models. In the second application, we apply our method to a publicly available breast cancer gene expression data set and show that the high-dimensional parametric model can achieve similar performance to a low-dimensional Cox model while being better calibrated. Finally, we conclude in Section 8 with directions for future work.

We have publicly released the C++ implementation of our algorithm which can be downloaded from http://code.google.com/p/survival-analysis-cmake/. This code has been derived from the widely-used BBR/BXR software for performing large-scale Bayesian logistic regression (http://www.bayesianregression.org). All four parametric models discussed in this paper are included in the implementation, however, it is fairly straightforward to also extend it to other parametric models. Computation of various evaluation metrics discussed in Section 5 is also integrated into the code.

2. Related Work

Survival analysis is an old subject in statistics that continues to attract considerable research attention. Traditionally, survival analysis is concerned with the study of survival times in clinical and health related studies [4, 15]. However, over the past several decades, survival analysis has found an array of applications ranging from reliability studies in industrial engineering to analyses of inter-child birth times in demography and sociology [3]. Other application areas have also benefited from the use of these methods [8, 6].

Earlier applications usually consisted of a relatively small number of predictors (usually less than 20) with a few hundred or sometimes a few thousand examples. Recently, there has been considerable interest in analyzing high-dimensional time-to-event problems. For instance, a large body of work has focused on methodologies to handle the overwhelming amount of data generated by new technologies in biology such as gene expression microarrays and single nucleotide polymorphism (SNP) data. The goal of the work in high-dimensional survival analysis has been both to develop new statistical methods [16, 17, 18, 19, 20, 21] and to extend the existing methods to handle new data sets [22, 23, 13, 24]. For example, recent work [16, 18] has extended the traditional support vector machines used for regression to survival analysis by additionally penalizing discordant pairs of observations. Another method [19, 20] extends the use of random forests for variable selection for survival analysis. Similarly, other methods [22, 25] have used an elastic net approach for variable selection both under the Cox proportional hazards model and under an accelerated failure time (AFT) model. This method is similar to other work [24] that applies an efficient method to compute L1-penalized parameter estimates for Cox models. A recent review [26] provides a survey of the existing methods for variable selection and model estimation for high-dimensional data.

Some more recent tools such as coxnet [27] and fastcox [25] adopt optimization approaches that can scale to the high-dimensional high sample size data that we focus on. Although other models provided by the R package glmnet do support sparse formats, neither coxnet nor fastcox currently support a sparse matrix format for the input data. However, both coxnet and fastcox provide estimates for the Cox proportional hazards model and fastcox supports elastic net regularization.

3. Regularized Survival Analysis

Denote by n the number of individuals in the training data. We represent their survival times by y_i = min(t_i, c_i), i = 1, …, n, where t_i and c_i are the time to event (failure time) and right-censoring time for each individual respectively. Let δ_i = I(t_i ≤ c_i) be the indicator variable such that δ_i is one if the observation is not censored and zero otherwise. Further, let x_i = [x_i1, x_i2, …, x_ip]^⊤ be a p-vector of covariates. We assume that t_i and c_i are conditionally independent given x_i and that the censoring mechanism is non-informative. The observed data comprise triplets D = {(y_i, δ_i, x_i):i = 1, …, n}.

Let θ be the set of unknown, underlying model parameters. We assume that the survival times y₁, y₂, …, y_n arise in an independent and identically distributed fashion from density and survival functions f(y|θ) and S(y|θ) respectively, parametrized by θ. We are interested in the likelihood L(θ|D) of the parametric model, where

$$L(\theta \mid D)=\prod _{i=1}^{n}f{(y_i\mid \theta )}^{{\delta }_i}S{(y_i\mid \theta )}^{(1-{\delta }_i)}$$ (1)

We analyze and compare the performance of four different parametric models by modeling the distributions of the survival times using Exponential, Weibull, Log-Logistic or Log Normal distributions. Each of these distributions can be fully parametrized by the parameter pair θ = (λ, α). Typically, the parameter λ is re-parametrized in terms of the covariates x = [x₁, x₂, …, x_p]^⊤ and the vector β = [β₁, β₂, …, β_p]^⊤ such that λ_i = φ (β^⊤x_i), i = 1, …, n. In general, the mapping function φ(·) is different for each model and standard choices exist. The likelihood function of (1) in terms of the new parameters can be written as

$$L(\mathit{\beta },\alpha \mid D)=\prod _{i=1}^{n}f{(y_i\mid {\lambda }_i,\alpha )}^{{\delta }_i}S{(y_i\mid {\lambda }_i,\alpha )}^{(1-{\delta }_i)}.$$ (2)

The parameters β and α are estimated by maximizing their joint posterior density

$$L(\mathit{\beta },\alpha )\propto L(\mathit{\beta },\alpha \mid D)\pi (\mathit{\beta })\pi (\alpha ).$$ (3)

The joint posterior distribution of (β, α) does not usually have a closed form solution, but it can be shown that the conditional posterior distributions π(β|α, D) and π(α|β, D) are concave and therefore can be solved efficiently. In practice, it is sufficient to estimate β and α by maximizing the conditional posterior of just β

$$L(\mathit{\beta })\propto L(\mathit{\beta },\alpha \mid D)\pi (\mathit{\beta }).$$ (4)

For a model to generalize well to unseen test data, it is important to avoid overfitting the training data. In the Bayesian paradigm, this goal can be achieved by specifying appropriate prior distribution on β such that each β_j is likely to be near zero. As we will observe in the following sections, both Gaussian and Laplacian priors fall under this category. Since we focus on posterior mode estimation in this paper, one can view our procedure as Bayesian or simply as a form of regularization or penalization. We use the Bayesian terminology in part because we view fully Bayesian computation as the next desirable step in large-scale survival analysis. We return to this point in the conclusion section.

3.1. Gaussian Priors and Ridge Regression

For L2 regularization, we assume a Gaussian prior for each β_j with zero mean and variance τ_j, i.e.,

$$\pi ({\beta }_j\mid {\tau }_j)=\text{N}(0,{\tau }_j)=\frac{1}{\sqrt{2\pi {\tau }_j}}exp\left(-\frac{{\beta }_j^2}{2{\tau }_j}\right).$$ (5)

The mean of zero encodes a prior preference for values of β_j that are close to zero. The variances τ_j are positive constants that control the degree of regularization and are typically chosen through cross-validation. Smaller values of τ_j imply stronger belief that β_j is being close to zero while larger values impose a less-informative prior. In the simplest case we assume that τ₁ = τ₂ = … = τ_p. Assuming that the components of β are independent a priori, the overall prior for β can be expressed as the product of the priors of each individual β_j, i.e., $\pi (\mathit{\beta }\mid {\tau }_1,\dots ,{\tau }_p)={\prod }_{j=1}^p\pi ({\beta }_j\mid {\tau }_j)$. Finding the maximum a posteriori estimate of β with this prior is equivalent to performing ridge regression [28]. Note that although the Gaussian prior favors the value of β_j close to zero, posterior modes are generally not exactly equal to zero for any β_j.

3.2. Laplacian Prior and Lasso Regression

For L1 regularization, we again assume that each β_j follows a Gaussian distribution with mean zero and variance τ_j. However, instead of fixed τ_j’s, instead we assume that each τ_j arise from an exponential distribution parametrized using γ_j’s and having density

$$\pi ({\tau }_j\mid {\gamma }_j)=\frac{{\gamma }_j}{2}exp\left(-\frac{{\gamma }_j}{2}{\tau }_j\right).$$ (6)

Integrating out τ_j gives an equivalent non-hierarchical, double-exponential (Laplace) distribution with density

$$\pi ({\beta }_j\mid {\gamma }_j)=\frac{\sqrt{{\gamma }_j}}{2}exp(-\sqrt{{\gamma }_j}\mid {\beta }_j\mid ).$$ (7)

Again, in the simplest case, we assume that γ₁ = γ₂ = … = γ_p. Similar to the Gaussian prior, assuming that the components of β are independent, $\pi \mathit{\beta }\mid {\gamma }_1,\dots ,{\gamma }_p)={\prod }_{j=1}^p\pi ({\beta }_j\mid {\gamma }_j)$. Finding the maximum a posteriori estimate of β with the Laplacian prior is equivalent to performing a Lasso regression [29]. Using this approach, a sparse solution typically ensues, meaning the posterior mode for many components of the β vector will be zero.

4. Parametric Models

It can be shown that for both Gaussian and Laplacian regularization, the negated log-posterior of (4) for all the four parametric models considered in our work are log-convex. Therefore, a wide range of optimization algorithms can be used. Due to the high-dimensionality of our target applications, usual methods like Newton-Raphson cannot be used because of their high memory requirements. Many alternate optimization approaches have been proposed for maximizing MAP for high-dimensional regression problems [30, 31]. We use the Combined Local and Global (CLG) algorithm of [30] which is a type of cyclic coordinate descent algorithm due to CLG’s favorable property of scaling to high-dimensional data and ease of implementation. This method was successfully adapted to the lasso [14] for performing large-scale logistic regression and implemented in the widely-used BBR/BXR software (http://www.bayesianregression.org).

A cyclic coordinate descent algorithm begins by setting all p parameters β_j, j = 1, …, p to some initial value. It then sets the first parameter to a value that minimizes the objective function, holding all other parameters constant. This problem then is a one-dimensional optimization. The algorithm finds the minimizing value of a second parameter, while holding all other values constant (including the new value of the first parameter). The third parameter is then optimized, and so on. When all variables have been traversed, the algorithm returns to the first parameter and starts again. Multiple passes are made over the parameters until some convergence criterion is met. We note that similar to previous work [14], instead of iteratively updating each parameter till convergence, we do it only once before proceeding on to the next parameter. Since the optimal values of the other parameters are themselves changing, tuning a particular parameter to very high precision, in each pass of the algorithm, is not necessary. For more details of the CLG method, we refer readers to a relevant publication in this area [30, 14]. The details of our algorithm for different parametric models are described in Appendix A. We also note that for the case of the Laplacian prior, the derivative of the negated log-posterior is undefined for β_j = 0, j = 1, …, p. Section 4.3 of [14] describes the modification of the CLG algorithm that we utilize to address this issue.

5. Experiments

We tested our algorithm for large-scale parametric survival analysis on two different data sets. Below, we briefly describe the data sets and also motivate their choice for our work. In both cases, we compare the performance of the high-dimensional parametric model trained on all predictors with that of a low-dimensional model trained on a small subset of features.

International Classification of Disease, Ninth Revision

5.1. Pediatric Trauma Data

Trauma is the leading cause of death and acquired disability in children and adolescents. Injuries result in more deaths in children than all other causes combined [32]. To provide optimal care, children at high risk for mortality need to be identified and triaged to centers with the resources to manage these patients. The overall goal of our analysis is to develop a model for predicting mortality after pediatric injury. For prediction within the domain of trauma, the literature has traditionally focused on low-dimensional analysis, i.e., modeling using a small set of features from the injury scene or emergency department [33, 34]. While approaches that use low-dimensional data to predict outcome may be easier to implement, the trade-off may be poorer predictive performance.

We obtained our data set from National Trauma Data Bank (NTDB), a trauma database maintained by the American College of Surgeons. The data set includes 210, 555 patient records of injured children < 15 years old collected over five years (2006–2010). We divided these data into a training data set(153, 402 patients for years 2006–2009) and a testing data set (57, 153 patients for year 2010). The mortality rate of the training set is 1.68% while that of the test set is 1.44%. There are a total of 125, 952 binary predictors indicating the presence or absence of a particular attribute (or interaction among various attributes). The high-dimensional model was trained using all the 125, 952 predictors, while the low-dimensional model used only 41 predictors. The information about various predictors used for both high and low-dimensional models is summarized in Table 1.

Table 1.

Description of the predictors used for high and low-dimensional models for pediatric trauma data.

Predictor Type	# Predictors	Description	High-dim	Low-dim
Main Effects
ICD-9 Codes	1, 890	International Classification of Disease, Ninth Revision.	✓	✗
AIS Codes (predots)	349	Abbreviated Injury Scale codes that includes body region, anatomic structure associated with the injury and the level of injury.	✓	✗
Interactions/Combinations
ICD-9, ICD-9	102, 284	Co-occurrences of two ICD-9 injury codes.	✓	✗
AIS Code, AIS Code	20, 809	Co-occurrences of two AIS codes.	✓	✗
Body Region, AIS Score	41	Combinations of any of the 9 body regions associated to an injury with injury severity score (between 1–6) determined according to AIS coding scheme.	✓	✓
[Body Region, AIS Score], [Body Region, AIS Score]	579	Co-occurrences of two [Body Region, AIS Score] combinations.	✓	✗

5.2. Breast Cancer Gene Expression Data

For our second application, we analyzed the well-known breast cancer gene expression data set [35, 36]. This data set is publicly available and consists of cDNA expression profiles of 295 tumor samples of patients with breast cancer. These patients were analyzed with breast cancer between 1984 and 1995 at the Netherlands Cancer Institute and were aged 52 years or younger at the time of diagnosis. Overall, 79 (26.78%) patients died during the follow-up time and the remaining 216 are censored. The total number of predictors (number of genes) is 24, 885 each of which represents the log-ratio of the intensities of the two color dyes used for a specific gene. The high-dimensional model was trained using all the 24, 885 predictors. For the low-dimensional model, we used glmpath (coxpath) package of R [37] and generated the entire regularized paths and outputs predictors in the order of their relative importance. We picked the top 5 predictors from the list and used our model to train a low-dimensional model for comparisons. The data was then randomly split into training (67%) and testing (33%) data such that the mortality in both parts was equal to the original mortality rate.

5.3. Hyperparameter Selection

The Gaussian and Laplace priors both require a prior variance, ${\sigma }_j^2$, j = 1, …, p, for parameter values. The actual hyperparameters are $\tau ={\tau }_j={\sigma }_j^2$ for the Gaussian and $\sqrt{\gamma }=\sqrt{{\gamma }_j}=\sqrt{2}/{\sigma }_j$ for Laplace. For both the applications, the regularization parameter was selected using a four-fold cross validation on the training data. The variance σ was varied between 10⁻⁵ – 10⁶ by multiples of 10. This amounts to varying actual regularization parameter for the Gaussian loss between 10⁻⁵ – 10⁶ by multiples of 10 while that of the Laplacian between 0.0014 – 447.21 in multiples of $\sqrt{10}$. For each choice of the hyperparameter, we computed the sum of log-likelihoods for the patients in all the four validation sets and chose the hyperparameter value that maximized this sum.

5.4. Performance Evaluation

There are several ways to compare the performance of fitted parametric models. These evaluation metrics can be divided into two categories - ones that assess the discrimination quality of the model and the others that assess the calibration. As for any other regression model, the log-likelihood on the test data is an obvious choices for measuring the performance of penalized parametric survival regression. In the past several years, survival analysis-specific metrics have been proposed that take censoring into account. Among these metrics, area under the ROC curve (AUC), also known as the Harrell’s c-statistic [38, 39], a measure of discriminative accuracy of the model, has become one of the important criteria used to evaluate the performance of survival models [40]. Motivated from a similar test for logistic regression model proposed by Hosmer and Lemeshow, we evaluated calibration using an overall goodness-of-fit test that has been proposed for survival data [41, 42]. Although the original test was proposed for Cox models, its use for parametric models have also been described [6, Chapter 8]. Below, we briefly describe these two metrics.

5.4.1. Harrell’s c-statistic

Harrell’s c-statistic is an extension of the tradition area under the curve (AUC) statistic, but is more suited for time-to-event data since it is independent of any thresholding process. The method is based on the comparison of estimated versus the ground truth ordering of risks between pairs of comparable subjects. Two subjects are said to be comparable if at least one of the subjects in the pair has developed the event (e.g., death) and if the follow-up time duration for that subject is less than that of the other. Using the notation of Section 3, the comparability of an ordered pair of subjects (i, j) can be represented by the indicator variable ζ_ij = I(y_i < y_j, δ_i = 1), such that ζ_i,j is one when the two subjects are comparable and zero otherwise. The total number of comparable pairs in the test data containing n subjects can then be computed as

$$n_{\zeta }=\sum _{i=1}^n\sum _{\begin{array}{l}j=1\\ j\ne i\end{array}}^n{\zeta }_{i,j}.$$ (8)

To measure the predicted discrimination, the concordance of the comparable pairs is then estimated. A pair of comparable subjects (as defined above) is said to be concordant, if the estimated risk of the subject who developed the event earlier is more than that of the other subject. Therefore, the concordance of the ordered subject pair (i, j) can also be represented using the indicator variable ξ_ij = I(ζ_ij = 1, r_i > r_j), where r_i = −β^⊤ x_i and r_j = −β^⊤x_j are the relative risks scores of the i^th and j^th subjects. Therefore, ξ_ij is one when the two subjects are concordant and zero otherwise. The total number of concordant pairs can then be written as

$$n_{\xi }=\sum _{i=1}^n\sum _{\begin{array}{l}j=1\\ j\ne i\end{array}}^n{\xi }_{i,j}.$$ (9)

Finally, the c-statistic is given by

$$\text{c}-\text{statistic}=\frac{n_{\xi }}{n_{\zeta }}.$$ (10)

5.4.2. Hosmer-Lemeshow Statistic

To assess the overall goodness-of-fit using this test, the subjects are first sorted in an order of their relative risk score (r_i = −β^⊤ x_i, i = 1, …, n) and divided into G equal-sized groups. The observed number of events o_g of g^th group is obtained by summing the number of non-censored observations in that group while the number of expected events e_g is computed by summing the cumulative hazard of all the subjects in the same group. The χ² statistic for the overall goodness-of-fit is then by

$${\chi }^2=\sum _{g=1}^G\frac{{(o_g-e_g)}^2}{e_g}.$$ (11)

6. Results

We now compare the performance of the low and high-dimensional parametric models on both the data sets. For both applications, the hyperparameter σ² was selected by performing a four-fold cross-validation on the training data.

6.1. Pediatric Trauma Data

Tables 2 and 3 summarize the results for the pediatric trauma data set for low and high-dimensional models using all the four parametric models with Gaussian and Laplacian penalization. Note that under the L2 prior, the estimate for any β_j is never exactly zero and thus all variables contribute towards the final model. The number of selected predictors in Table 2 just refers to the number of significant predictors (|β_j| > 10⁻⁴). The Hosmer-Lameshow χ² index was computed using the value of G = 50. Both the discriminative measures (Log-likelihood, c-statistic) are always significantly better for high-dimensional models as compared to the corresponding low-dimensional models. In many cases, the high-dimensional model is also better calibrated.

Table 2.

Comparison of low and high-dimensional models for pediatric trauma data with Gaussian penalization. The number of selected predictors refers to the number of significant predictors (β_j > 10⁻⁴).

Model Type	Predictors		Log-likelihood	c-statistic	χ2
	Overall	Selected
Exponential
Low-dim	41	41	−4270.01	0.88	124.39
High-dim	125,952	101,733	−4372.41	0.94	543.28
Weibull
Low-dim	41	41	−4242.38	0.88	131.60
High-dim	125,952	101,794	−4557.10	0.94	749.22
Log-Logistic
Low-dim	41	41	−4120.66	0.89	95.37
High-dim	125,952	100,889	−3765.45	0.94	95.02
Log-Normal
Low-dim	41	41	−3234.00	0.89	76.95
High-dim	125,952	88,244	−3129.02	0.93	165.68

Table 3.

Comparison of low and high-dimensional models for pediatric trauma data with Laplacian penalization.

Model Type	Predictors		Log-likelihood	c-statistic	χ2
	Overall	Selected
Exponential
Low-dim	41	41	−4271.23	0.88	122.58
High-dim	125,952	153	−4034.67	0.92	94.34
Weibull
Low-dim	41	41	−4243.57	0.88	126.84
High-dim	125,952	151	−3997.28	0.92	107.99
Log-Logistic
Low-dim	41	41	−4122.07	0.89	94.73
High-dim	125,952	432	−3777.83	0.94	83.00
Log-Normal
Low-dim	41	41	−3236.79	0.89	80.71
High-dim	125,952	168	−2974.49	0.93	89.36

To provide further insight, we grouped the subjects into low, medium and high risk groups by sorting their relative risk scores in increasing order and using threshold values at the 33^rd and 66^th percentiles. For each group, we counted the number of events (number of non-censored observations). Tables 4 and 5 summarize the results for Gaussian and Laplacian penalization respectively. The results show that in almost all cases, the subjects assigned to high risk group by the high-dimensional models had more events than the ones assigned high-risk by the low-dimensional models. Although both kinds of models assign similar number of subjects to the low risk group, the mean observed survival time of the subjects having events is more for the high-dimensional models than for the corresponding low-dimensional ones. These findings also establish that in most cases the high-dimensional models are better calibrated than their low-dimensional counterparts.

Table 4.

Comparison of number of events in low, medium and high risk groups for various low and high-dimensional models using Gaussian penalization for pediatric trauma data. MST stands for mean observed survival time in days.

Model Type	Risk Group	# Subjects	Low-dim		High-dim
			# Events	MST	# Events	MST
Exponential	Low	18,860	10	2.00	8	3.38
	Medium	18,860	41	4.10	18	4.33
	High	19,433	774	3.66	799	3.65
Weibull	Low	18,860	11	2.00	7	3.43
	Medium	18,860	34	3.44	18	3.67
	High	19,433	780	3.69	800	3.66
Log-Logistic	Low	18,860	10	2.00	11	3.73
	Medium	18,860	32	4.13	21	4.14
	High	19,433	783	3.66	793	3.65
Log-Normal	Low	18,860	10	2.00	16	6.56
	Medium	18,860	31	4.10	39	3.23
	High	19,433	784	3.66	770	3.62

Table 5.

Comparison of number of events in low, medium and high risk groups for various low and high-dimensional models using Laplacian penalization for pediatric trauma data. MST stands for mean observed survival time in days.

Model Type	Risk Group	# Subjects	Low-dim		High-dim
			# Events	MST	# Events	MST
Exponential	Low	18,860	10	2.00	10	3.00
	Medium	18,860	40	3.95	28	7.75
	High	19,433	775	3.67	787	3.52
Weibull	Low	18,860	10	2.00	10	3.00
	Medium	18,860	37	3.32	20	9.80
	High	19,433	778	3.70	795	3.51
Log-Logistic	Low	18,860	10	2.00	4	4.50
	Medium	18,860	32	4.13	23	3.65
	High	19,433	783	3.66	798	3.66
Log-Normal	Low	18,860	10	2.00	12	3.00
	Medium	18,860	32	4.10	25	2.96
	High	19,433	783	3.67	788	3.69

6.2. Breast Cancer Gene Expression Data

The results of the gene expression data set for low and high-dimensional models using all the four parametric models with Gaussian and Laplacian penalization are summarized in Tables 6 and 7 respectively. The Hosmer-Lameshow χ² index was computed using G = 10 due to very few observations in the test set. While the discriminative performance of both methods is similar for most of the cases, the high-dimensional models are almost always better calibrated.

Table 6.

Comparison of low and high-dimensional models for gene expression data with Gaussian penalization. The number of selected predictors refers to the number of significant predictors (β_j > 10⁻⁴).

Model Type	Predictors		Log-likelihood	c-statistic	χ2
	Overall	Selected
Exponential
Low-dim	5	5	−86.26	0.71	16.83
High-dim	24,496	24,344	−98.72	0.75	112.69
Weibull
Low-dim	5	5	−85.64	0.71	12.79
High-dim	24,496	22,299	−85.56	0.70	9.34
Log-Logistic
Low-dim	5	5	−85.65	0.70	14.74
High-dim	24,496	22,090	−86.14	0.70	5.37
Log-Normal
Low-dim	5	5	−52.10	0.70	16.02
High-dim	24,496	24,154	−65.14	0.66	23.61

Table 7.

Comparison of low and high-dimensional models for gene expression data with Laplacian penalization.

Model Type	Predictors		Log-likelihood	c-statistic	χ2
	Overall	Selected
Exponential
Low-dim	5	5	−86.27	0.71	17.14
High-dim	24,496	13	−87.51	0.67	2.74
Weibull
Low-dim	5	5	−85.63	0.71	12.79
High-dim	24,496	13	−86.80	0.66	3.75
Log-Logistic
Low-dim	5	5	−85.65	0.70	14.71
High-dim	24,496	9	−86.20	0.68	3.66
Log-Normal
Low-dim	5	5	−52.10	0.70	15.98
High-dim	24,496	9	−53.46	0.66	8.57

7. Computation Time

Table 8 summarizes the training time taken for fitting different parametric models for low and high-dimensional data sets. All the experiments were performed on a system with Intel 2.4GHz processor with 8Gb of memory. Note that even though the time taken to fit high-dimensional models to pediatric trauma data set is much more than that taken to fit low-dimensional models, given the scale of the problem (153, 402 patients with 125, 952 predictors), the performance of the methods may be acceptable in many applications.

Table 8.

Computation time (in seconds) taken for training various low and high-dimensional models using Gaussian and Laplacian penalization for pediatric trauma and gene expression data.

Model Type	Pediatric Trauma				Gene Expression
	Gaussian		Laplacian		Gaussian		Laplacian
	Low-dim	High-dim	Low-dim	High-dim	Low-dim	High-dim	Low-dim	High-dim
Exponential	1	4,453	1	2,588	1	4	1	8
Weibull	3	3,406	2	2,600	1	12	1	30
Log-Logistic	2	2,794	2	3,278	1	13	1	38
Log-Normal	6	3,250	6	3,101	1	51	1	52

8. Conclusions

We present a method to perform regularized parametric survival analysis on data with 10⁴ – 10⁶ predictor variables and a large number of observations. Through our experiments in the context of two different applications, we have demonstrated the advantage of using high-dimensional survival analysis over the corresponding low-dimensional models. We have provided a freely available software tool that implements our proposed algorithm. Future work will provide the extension to Cox proportional hazards models in addition to accelerated failure time models and the Aalen’s additive hazard model. We have also developed software for high-dimensional regularized generalized linear models that utilizes inexpensive massively parallel devices known as graphics processing units (GPUs) [43]. This provides more than an order-of-magnitude speed-up and in principle could be further developed to include survival analysis. Fully Bayesian extensions to our current work could explore the hierarchical framework to simultaneously model multiple time-to-event endpoints, model multi-level structure such as patients-nested-within-hospitals, and incorporate prior information when available.

A. Appendix

Here we describe the details of our algorithm for Exponential, Weibull, Log-Logistic or Log-Normal distributions of the survival times.

A.1. Exponential

The exponential model is the simplest of all and can be parametrized using a single parameter λ, such that the density and survival functions can respectively be written as $f(y\mid \theta )=f(y\mid \lambda )=\frac{1}{\lambda }exp(-\frac{y}{\lambda })$ and $S(y\mid \theta )=S(y\mid \lambda )=exp(-\frac{y}{\lambda })$. The likelihood function of can be written as

$$L({\lambda }_1,\dots ,{\lambda }_n\mid D)=\prod _{i=1}^{n}f{(y_i\mid {\lambda }_i)}^{{\delta }_i}S{(y_i\mid {\lambda }_i)}^{(1-{\delta }_i)}=exp\left(\sum _{i=1}^n\left(-{\delta }_{i}log{\lambda }_i-\frac{y_i}{{\lambda }_i}\right)\right).$$ (12)

A common form for the mapping function φ(·) is to take the φ(β^⊤ x_i) = exp(β^⊤ x_i). The likelihood function is then

$$L(\mathit{\beta }\mid D)=exp\left(-\sum _{i=1}^n{\delta }_i{\mathit{\beta }}^{\top }{\mathbf{x}}_i-\sum _{i=1}^n{y}_{i}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)\right)$$ (13)

and the corresponding log-likelihood is

$$l(\mathit{\beta }\mid D)=-\sum _{i=1}^n{\delta }_i{\mathit{\beta }}^{\top }{\mathbf{x}}_i-\sum _{i=1}^n{y}_{i}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i).$$ (14)

Adding the Gaussian prior with mean zero and variance τ_j, the posterior density can be written as

$$l_G(\mathit{\beta })=l(\mathit{\beta }\mid D)+log(\pi (\mathit{\beta }\mid {\tau }_1,\dots ,{\tau }_p))=l(\mathit{\beta }\mid D)-\sum _{j=1}^p\left(log\sqrt{{\tau }_j}+\frac{1}{2}log2\pi +\frac{{\beta }_j^2}{2{\tau }_j}\right).$$ (15)

Similarly, for the Laplacian prior, the posterior density can be written as

$$l_L(\mathit{\beta })=l(\mathit{\beta }\mid D)+log(\pi (\mathit{\beta }\mid {\gamma }_1,\dots ,{\gamma }_p))=l(\mathit{\beta }\mid D)-\sum _{j=1}^p(log2-log{\gamma }_j+{\gamma }_j\mid {\beta }_j\mid ).$$ (16)

Using the CLG algorithm, the one-dimensional problem involves finding ${\beta }_j^{(\mathit{new})}$, the value of the j-th entry of β that minimizes −l(β), assuming that the other β_j’s are held at their current values. Therefore, using (14) and (15) for Gaussian prior (and ignoring the constants $log\sqrt{{\tau }_j}$ and $\frac{1}{2}log2\pi$), finding ${\beta }_j^{(\mathit{new})}$ is equivalent to the z that minimizes

$$g_G(z)=z\sum _{i=1}^n{x}_{ij}{\delta }_i+\sum _{i=1}^n{y}_{i}exp\left(-{zx}_{ij}-\sum _{\begin{array}{l}k=1\\ k\ne j\end{array}}^p{\beta }_k{x}_{ik}\right)+\frac{z^2}{2{\tau }_j}.$$ (17)

The classic Newton method approximates the objective function g(·) by the first three terms of its Taylor series at the current β_j

$$g(z)\approx g({\beta }_j)+g^{\prime }({\beta }_j)(z-{\beta }_j)+\frac{1}{2}{g}^{″}({\beta }_j){(z-{\beta }_j)}^2$$ (18)

where

$$g_G^{\prime }({\beta }_j)={\frac{{dg}_G(z)}{dz}|}_{z={\beta }_j}=\sum _{i=1}^n{x}_{ij}{\delta }_i-\sum _{i=1}^n{y}_i{x}_{ij}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)+\frac{{\beta }_j}{{\tau }_j}$$ (19)

$$g_G^{″}({\beta }_j)={\frac{d^2{g}_G(z)}{{dz}^2}|}_{z={\beta }_j}=\sum _{i=1}^n{y}_i{x}_{ij}^{2}exp(-{\beta }^{\top }{\mathbf{x}}_i)+\frac{1}{{\tau }_j}.$$ (20)

Similarly, for Laplacian prior,

$$g_L(z)=z\sum _{i=1}^n{\mathit{x}}_{ij}{\delta }_i+\sum _{i=1}^n{y}_{i}exp\left(-{zx}_{ij}-\sum _{\begin{array}{l}k=1\\ k\ne j\end{array}}^p{\beta }_k{x}_{ik}\right)+{\lambda }_j\mid z\mid$$ (21)

$$g_L^{\prime }({\beta }_j)={\frac{{dg}_L(z)}{dz}|}_{z={\beta }_j}=\sum _{i=1}^n{x}_{ij}{\delta }_i-\sum _{i=1}^n{y}_i{x}_{ij}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)+{\lambda }_j\text{sign}({\beta }_j),{\beta }_j\ne 0$$ (22)

$$g_L^{″}({\beta }_j)={\frac{d^2{g}_L(z)}{{dz}^2}|}_{z={\beta }_j}=\sum _{i=1}^n{y}_i{x}_{ij}^{2}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i),{\beta }_j\ne 0.$$ (23)

The value of ${\beta }_j^{(\mathit{new})}$ for both types of priors can then be computed as

$${\beta }_j^{(\mathit{new})}={\beta }_j+\mathrm{\Delta }{\beta }_j={\beta }_j-\frac{g^{\prime }({\beta }_j)}{g^{″}({\beta }_j)}.$$ (24)

A.2. Weibull

The Weibull model is a more general parametric model with density and survival functions given by $f(y\mid \theta )=f(y\mid \lambda ,\alpha )=\frac{\alpha y^{\alpha -1}}{\lambda }exp\left(-\frac{y^{\alpha }}{\lambda }\right)$ and $S(y\mid \theta )=S(y\mid \lambda ,\alpha )=exp\left(-\frac{y^{\alpha }}{\lambda }\right)$ respectively. The corresponding likelihood function can be written as

$$L({\lambda }_1,\dots ,{\lambda }_n,\alpha \mid D)=\prod _{i=1}^{n}f{(y_i\mid {\lambda }_i,\alpha )}^{{\delta }_i}S{(y_i\mid {\lambda }_i,\alpha )}^{(1-{\delta }_i)}={\alpha }^d\prod _{i=1}^n{\left(\frac{y_i^{\alpha -1}}{{\lambda }_i}exp\left(-\frac{y_i^{\alpha }}{{\lambda }_i}\right)\right)}^{{\delta }_i}{\left(exp\left(-\frac{y_i^{\alpha }}{{\lambda }_i}\right)\right)}^{(1-{\delta }_i)}$$ (25)

where $d={\sum }_{i=1}^n{\delta }_i$. Similar to the previous case, using λ_i = exp(β^⊤ x_i), the log-likelihood function can be written as

$$l(\alpha ,\mathit{\beta }\mid D)=dlog\alpha +\sum _{i=1}^n\left({\delta }_i(\alpha -1)log{y}_i-{\delta }_i{\mathit{\beta }}^{\top }{\mathbf{x}}_i-\frac{y_i^{\alpha }}{exp({\mathit{\beta }}^{\top }{\mathbf{x}}_i)}\right).$$ (26)

Similar to equations (15) and (16) conditional posterior corresponding to the Gaussian and Laplacian priors can be written as

$$l_G(\alpha ,\mathit{\beta })=l(\alpha ,\mathit{\beta }\mid D)+log(\pi (\mathit{\beta }\mid {\tau }_1,\dots ,{\tau }_p))=l(\alpha ,\mathit{\beta }\mid D)-\sum _{j=1}^p\left(log\sqrt{{\tau }_j}+\frac{1}{2}log2\pi +\frac{{\beta }_j^2}{2{\tau }_j}\right).$$ (27)

$$l_L(\alpha ,\mathit{\beta })=l(\alpha ,\mathit{\beta }\mid D)+log(\pi (\mathit{\beta }\mid {\gamma }_1,\dots ,{\gamma }_p))=l(\alpha ,\mathit{\beta }\mid D)-\sum _{j=1}^p(log2-log{\gamma }_j+{\gamma }_j\mid {\beta }_j\mid ).$$ (28)

Using the CLG algorithm, the one-dimensional problem involve finding ${\alpha }_j^{(\mathit{new})}$ and ${\beta }_j^{(\mathit{new})}$, the value of the j-th that gives the minimum value of −l(α, β), assuming that the other α_j’s and β_j’s are held at their current values. Therefore, for both Gaussian and Laplacian prior, finding α^(new) is equivalent to finding the z that minimizes

$$g(z)=-\mathit{d}logz-\sum _{i=1}^n\left({\delta }_i(z-1)log{y}_i-y_i^{z}exp\left(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i\right)\right).$$ (29)

Writing the Taylor series expansion for g(z) around z

$$g(z)\approx g(\alpha )+g^{\prime }(\alpha )(z-\alpha )+\frac{1}{2}{g}^{″}(\alpha ){(z-\alpha )}^2$$ (30)

where

$$g^{\prime }(\alpha )={\frac{dg(z)}{dz}|}_{z=\alpha }=-d/\alpha -\sum _{i=1}^n({\delta }_{i}log{y}_i-y_i^{\alpha }exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)log{y}_i)$$ (31)

$$g^{″}(\alpha )={\frac{d^{2}g(z)}{{dz}^2}|}_{z=\alpha }=d/{\alpha }^2+\sum _{i=1}^n{y}_i^{\alpha }{(log{y}_i)}^{2}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i).$$ (32)

The value of ${\alpha }_j^{(\mathit{new})}$ for both types of priors can then be computed as

$${\alpha }^{(\mathit{new})}=\alpha +\mathrm{\Delta }\alpha =\alpha -\frac{g^{\prime }(\alpha )}{g^{″}(\alpha )}.$$ (33)

The stepwise update for β for both Gaussian and Laplacian priors are similar to that of the exponential model and can be obtained by replacing y_i with $y_i^{\alpha }$ in equations (18)–(24).

A.3. Log-Logistic

For Log-Logistic model, the density and survival functions are given by

$$f(y\mid \theta )=f(y\mid \lambda ,\alpha )=\frac{\alpha y^{\alpha -1}}{\lambda {\left(1+\frac{y^{\alpha }}{\lambda }\right)}^2}\text{and}S(y\mid \theta )=S(y\mid \lambda ,\alpha )=\frac{1}{\left(1+\frac{y^{\alpha }}{\lambda }\right)}.$$ (34)

The corresponding likelihood function can be written as

$$L({\lambda }_1,\dots ,{\lambda }_n,\alpha \mid D)=\prod _{i=1}^{n}f{(y_i\mid {\lambda }_i,\alpha )}^{{\delta }_i}S{(y_i\mid {\lambda }_i,\alpha )}^{(1-{\delta }_i)}={\alpha }^d\prod _{i=1}^n{\left(\frac{y_i^{\alpha -1}}{{\lambda }_i{\left(1+\frac{y_i^{\alpha }}{{\lambda }_i}\right)}^2}\right)}^{{\delta }_i}{\left(\frac{1}{\left(1+\frac{y_i^{\alpha }}{{\lambda }_i}\right)}\right)}^{(1-{\delta }_i)}.$$ (35)

where $d={\sum }_{i=1}^n{\delta }_i$. Re-parameterizing, λi = exp(β^⊤ x_i), the log-likelihood function can be written as

$$l(\alpha ,\mathit{\beta }\mid D)=dlog\alpha +\sum _{i=1}^n\left({\delta }_i(\alpha -1)log{y}_i-{\delta }_i{\mathit{\beta }}^{\top }{\mathbf{x}}_i-(1+{\delta }_i)log\left(1+y_i^{\alpha }exp\left(-{\beta }^{\top }{\mathbf{x}}_i\right)\right)\right).$$ (36)

The posterior distributions corresponding to the Gaussian and Laplacian priors are similar to those for the Weibull model (27) (28). Using the CLG algorithm, for both types of priors, finding the update α^(new) is equivalent to finding the z that minimizes

$$g(z)=-\mathit{d}logz-\sum _{i=1}^n\left({\delta }_i(z-1)log{y}_i-(1+{\delta }_i)log\left(1+y_i^{z}exp\left(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i\right)\right)\right).$$ (37)

The value of ${\alpha }_j^{(\mathit{new})}$ for both types of priors can then be computed using (33) where

$$g^{\prime }(\alpha )={\frac{dg(z)}{dz}|}_{z=\alpha }=-d/\alpha -\sum _{i=1}^n\left({\delta }_{i}log{y}_i-\frac{(1+{\delta }_i)y_i^{\alpha }exp\left(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i\right)log{y}_i}{1+y_i^{\alpha }exp\left(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i\right)}\right)$$ (38)

$$g^{″}(\alpha )={\frac{d^{2}g(z)}{{dz}^2}|}_{z=\alpha }=d/{\alpha }^2+\sum _{i=1}^n\frac{(1+{\delta }_i)y_i^{\alpha }exp\left(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i\right){(log{y}_i)}^2}{{\left(1+y_i^{\alpha }exp\left(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i\right)\right)}^2}.$$ (39)

For Gaussian prior, finding ${\beta }_j^{(\mathit{new})}$ is equivalent to finding the z that minimizes

$$g_G(z)=z\sum _{i=1}^n{x}_{ij}{\delta }_i+\sum _{i=1}^n(1+{\delta }_i)log\left(1+y_i^{\alpha }exp\left(-{zx}_{ij}-\sum _{\begin{array}{l}k=1\\ k\ne j\end{array}}^p{\beta }_k{x}_{ik}\right)\right)+\frac{z^2}{2{\tau }_j}.$$ (40)

Similar to the previous cases, the updated value ${\beta }_j^{(\mathit{new})}$ can be computed using (24), where

$$g_G^{\prime }({\beta }_j)={\frac{{dg}_G(z)}{dz}|}_{z={\beta }_j}=\sum _{i=1}^n{x}_{ij}{\delta }_i-\sum _{i=1}^n\frac{(1+{\delta }_i)y_i^{\alpha }{x}_{ij}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)}{1+y_i^{\alpha }exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)}+\frac{{\beta }_j}{{\tau }_j}$$ (41)

$$g_G^{″}({\beta }_j)={\frac{d^2{g}_G(z)}{{dz}^2}|}_{z={\beta }_j}=\sum _{i=1}^n\frac{(1+{\delta }_i)y_i^{\alpha }{x}_{ij}^{2}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)}{{\left(1+y_i^{\alpha }exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)\right)}^2}+\frac{1}{{\tau }_j}.$$ (42)

Similarly, for Laplacian prior

$$g_L(z)=z\sum _{i=1}^n{x}_{ij}{\delta }_i+\sum _{i=1}^n(1+{\delta }_i)log\left(1+y_i^{\alpha }exp\left(-{zx}_{ij}-\sum _{\begin{array}{l}k=1\\ k\ne j\end{array}}^p{\beta }_k{x}_{ik}\right)\right)+{\lambda }_j\mid z\mid$$ (43)

$$g_L^{\prime }({\beta }_j)={\frac{{dg}_L(z)}{dz}|}_{z={\beta }_j}=\sum _{i=1}^n{x}_{ij}{\delta }_i-\sum _{i=1}^n\frac{(1+{\delta }_i)y_i^{\alpha }{x}_{ij}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)}{1+y_i^{\alpha }exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)}+{\lambda }_j\text{sign}({\beta }_j),{\beta }_j\ne 0$$ (44)

$$g_L^{″}({\beta }_j)={\frac{d^2{g}_L(z)}{{dz}^2}|}_{z={\beta }_j}=\sum _{i=1}^n\frac{(1+{\delta }_i)y_i^{\alpha }{x}_{ij}^{2}exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)}{{\left(1+y_i^{\alpha }exp(-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)\right)}^2},{\beta }_j\ne 0.$$ (45)

A.4. Log-Normal

Assuming that the survival times y_i, i = 1, …, n follow a log-normal distribution is equivalent to assuming that their logarithms w_i = log y_i, i = 1, …, n follow a normal N(μ, σ²) distribution with density and survival functions given by $f(y\mid \theta )=f(w\mid \mu ,\sigma )=\frac{1}{\sqrt{2\pi {\sigma }^2}}exp\left(-\frac{{(w-\mu )}^2}{2{\sigma }^2}\right)$ and $S(y\mid \theta )=S(w\mid \mu ,\sigma )=1-\mathrm{\Phi }\left(\frac{w-\mu }{\sigma }\right)$ respectively, where Φ(·) is the Gaussian cumulative distribution function. Following the convention, we replace λ and α with μ and σ respectively. The likelihood function of μ_i, μ_j, …, μ_n and σ can be written as

$$\begin{array}{l}L({\mu }_1,\dots ,{\mu }_n,\sigma \mid D)=\prod _{i=1}^{n}f{(y_i\mid {\mu }_i,\sigma )}^{{\delta }_i}S{(y_i\mid {\mu }_i,\sigma )}^{(1-{\delta }_i)}\\ ={\left(2\pi {\sigma }^2\right)}^{-d/2}\prod _{i=1}^n{\left(exp\left(-\frac{{(w-{\mu }_i)}^2}{2{\sigma }^2}\right)\right)}^{{\delta }_i}{\left(1-\mathrm{\Phi }\left(\frac{w-{\mu }_i}{\sigma }\right)\right)}^{(1-{\delta }_i)}\end{array}$$ (46)

where $d={\sum }_{i=1}^n{\delta }_i$. Re-parameterizing μ_i = β^⊤ x_i, the log-likelihood function can be written as

$$l(\sigma ,\mathit{\beta }\mid D)=log2\pi -dlog\sigma -\frac{1}{2{\sigma }^2}\sum _{i=1}^n{\delta }_i{(w_i-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)}^2+\sum _{i=1}^n(1-{\delta }_i)log\left(1-\mathrm{\Phi }\left(\frac{w-{\mathit{\beta }}^{\top }{\mathbf{x}}_i}{\sigma }\right)\right).$$ (47)

The conditional posterior densities corresponding to Gaussian and Laplacian priors can be written as

$$l_G(\sigma ,\mathit{\beta })=l(\sigma ,\mathit{\beta }\mid D)+log(\pi (\mathit{\beta }\mid {\tau }_1,\dots ,{\tau }_p))=l(\sigma ,\mathit{\beta }\mid D)-\sum _{j=1}^p\left(log\sqrt{{\tau }_j}+\frac{1}{2}log2\pi +\frac{{\beta }_j^2}{2{\tau }_j}\right).$$ (48)

$$l_L(\sigma ,\mathit{\beta })=l(\sigma ,\mathit{\beta }\mid D)+log(\pi (\mathit{\beta }\mid {\gamma }_1,\dots ,{\gamma }_p))=l(\sigma ,\mathit{\beta }\mid D)-\sum _{j=1}^p(log2-log{\gamma }_j+{\gamma }_j\mid {\beta }_j\mid ).$$ (49)

Assuming that the other parameters are held at their current values, the one-dimensional problems involve finding σ^(new) and ${\beta }_j^{(\mathit{new})}$, that minimize the posterior. Using the CLG algorithm for both types of priors, finding σ^(new) is equivalent to finding the z that minimizes

$$g(z)=dlog\sigma +\frac{1}{2{\sigma }^2}\sum _{i=1}^n{\delta }_i{(w_i-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)}^2-\sum _{i=1}^n(1-{\delta }_i)log\left(1-\mathrm{\Phi }\left(\frac{w-{\mathit{\beta }}^{\top }{\mathbf{x}}_i}{\sigma }\right)\right).$$ (50)

The value of σ^(new) can then be computed as

$${\sigma }^{(\mathit{new})}=\sigma +\mathrm{\Delta }\sigma =\sigma -\frac{g^{\prime }(\sigma )}{g^{″}(\sigma )}.$$ (51)

Here

$$g^{\prime }(\sigma )={\frac{dg(z)}{dz}|}_{z=\sigma }=\frac{d}{\sigma }-\frac{1}{\sigma }\sum _{i=1}^n{\delta }_i{p}_i^2-\frac{1}{\sigma }\sum _{i=1}^n(1-{\delta }_i)p_i{q}_i$$ (52)

$$g^{″}(\sigma )={\frac{d^{2}g(z)}{{dz}^2}|}_{z=\sigma }=-\frac{d}{{\sigma }^2}+\frac{3}{{\sigma }^2}\sum _{i=1}^n{\delta }_i{p}_i^2-\frac{1}{{\sigma }^2}\sum _{i=1}^n(1-{\delta }_i)(p_i^3{q}_i-p_i^2{q}_i^2-2p_i{q}_i).$$ (53)

where

$$p_i=\frac{w_i-{\mathit{\beta }}^{\top }{\mathbf{x}}_i}{\sigma }\text{and}{q}_i=\frac{\text{N}\left((w_i-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)/\sigma \right)}{1-\mathrm{\Phi }\left((w-{\mathit{\beta }}^{\top }{\mathbf{x}}_i)/\sigma \right)}.$$ (54)

For Gaussian prior, finding ${\beta }_j^{(\mathit{new})}$ is equivalent to finding the z that minimizes

$$g_G(z)=\frac{1}{2{\sigma }^2}\sum _{i=1}^n{\delta }_i{(r_i-x_{ij}z)}^2-\sum _{i=1}^n(1-{\delta }_i)log\left(1-\mathrm{\Phi }\left(\frac{r_i-x_{ij}z}{\sigma }\right)\right)+\frac{z^2}{2{\tau }_j}$$ (55)

where $r_i=w_i-{\sum }_{\begin{array}{l}k=1\\ k\ne j\end{array}}^p{\beta }_k{x}_{ik}$. The updated value ${\beta }_j^{(\mathit{new})}$ can be computed using (24), where

$$g_G^{\prime }({\beta }_j)={\frac{{dg}_G(z)}{dz}|}_{z={\beta }_j}=-\frac{1}{\sigma }\sum _{i=1}^n{x}_{ij}({\delta }_i{p}_i+(1-{\delta }_i)q_i)+\frac{{\beta }_j}{{\tau }_j}$$ (56)

$$g_G^{″}({\beta }_j)={\frac{d^2{g}_G(z)}{{dz}^2}|}_{z={\beta }_j}=\frac{1}{{\sigma }^2}\sum _{i=1}^n{x}_{ij}({\delta }_i-q_i(1-{\delta }_i)(p_i-q_i))+\frac{1}{{\tau }_j}$$ (57)

and p_i and q_i are same as (54). Similarly, for Laplacian prior

$$g_L(z)=\frac{1}{2{\sigma }^2}\sum _{i=1}^n{\delta }_i{(r_i-x_{ij}z)}^2-\sum _{i=1}^n(1-{\delta }_i)log\left(1-\mathrm{\Phi }\left(\frac{r_i-x_{ij}z}{\sigma }\right)\right)+{\lambda }_j\mid z\mid$$ (58)

$$g_L^{\prime }({\beta }_j)={\frac{{dg}_L(z)}{dz}|}_{z={\beta }_j}=-\frac{1}{\sigma }\sum _{i=1}^n{x}_{ij}({\delta }_i{p}_i+(1-{\delta }_i)q_i)+{\lambda }_j\text{sign}({\beta }_j),{\beta }_j\ne 0$$ (59)

$$g_L^{″}({\beta }_j)={\frac{d^2{g}_L(z)}{{dz}^2}|}_{z={\beta }_j}=\frac{1}{{\sigma }^2}\sum _{i=1}^n{x}_{ij}({\delta }_i-q_i(1-{\delta }_i)(p_i-q_i)),{\beta }_j\ne 0.$$ (60)