Sparse Regression in Cancer Genomics: Comparing Variable Selection and Predictions in Real World Data

Abstract

Background:

Evaluation of gene interaction models in cancer genomics is challenging, as the true distribution is uncertain. Previous analyses have benchmarked models using synthetic data or databases of experimentally verified interactions – approaches which are susceptible to misrepresentation and incompleteness, respectively. The objectives of this analysis are to (1) provide a real-world data-driven approach for comparing performance of genomic model inference algorithms, (2) compare the performance of LASSO, elastic net, best-subset selection, $L_0{L}_1$ penalisation and $L_0{L}_2$ penalisation in real genomic data and (3) compare algorithmic preselection according to performance in our benchmark datasets to algorithmic selection by internal cross-validation.

Methods:

Five large $(n4000)$ genomic datasets were extracted from Gene Expression Omnibus. ‘Gold-standard’ regression models were trained on subspaces of these datasets ( $n4000$ , $p=500$ ). Penalised regression models were trained on small samples from these subspaces ( $n\in \left\{25,75,150\right\},p=500$ ) and validated against the gold-standard models. Variable selection performance and out-of-sample prediction were assessed. Penalty ‘preselection’ according to test performance in the other 4 datasets was compared to selection internal cross-validation error minimisation.

Results:

$L_1{L}_2$ -penalisation achieved the highest cosine similarity between estimated coefficients and those of gold-standard models. $L_0{L}_2$ -penalised models explained the greatest proportion of variance in test responses, though performance was unreliable in low signal:noise conditions. $L_0{L}_2$ also attained the highest overall median variable selection F1 score. Penalty preselection significantly outperformed selection by internal cross-validation in each of 3 examined metrics.

Conclusions:

This analysis explores a novel approach for comparisons of model selection approaches in real genomic data from 5 cancers. Our benchmarking datasets have been made publicly available for use in future research. Our findings support the use of $L_0{L}_2$ penalisation for structural selection and $L_1{L}_2$ penalisation for coefficient recovery in genomic data. Evaluation of learning algorithms according to observed test performance in external genomic datasets yields valuable insights into actual test performance, providing a data-driven complement to internal cross-validation in genomic regression tasks.

Author Summary

Regression models are frequently used in cancer genomics, where they provide insight into the interactions between genes. Sparse regression models were developed to allow modelling of a large set of variables with a small number of samples – a scenario encountered frequently in genomics. However, evaluation of genomic model structures remains challenging, due to uncertainty regarding the true system of interactions. Previous studies have compared methods with synthetic data, which may not reflect the challenges of real-world data. In this study, genomic datasets were identified which contained enough samples to provide reasonable estimates of the true structures – which were used as ‘gold-standards’. Sparse regression methods were tasked with estimating the true structure given a small proportion of the available samples, allowing for comparison against the gold standards.

Our results show that the interaction strengths estimated by the $L_1{L}_2$ penalisation method correspond best with the gold standard models. Other penalisation methods, including the $L_0{L}_2$ penalisation method, may be unreliable in noisy data. We demonstrate that modelling decision may be supported by our evaluation method, an approach which may complement cross-validation.

Background

Regression models in cancer genomics

High-dimensional regression problems are ubiquitous in modern oncological research, as datasets often contain fewer observations than variables.^1-7 The tractability of penalised regression approaches in this setting has led to a large volume of research into their applications.^1,7-9 Penalised regression offers robust predictions in high dimensional data and mechanistic insights through the estimated coefficient vector.^1,7 $L_0$ and $L_1$ penalties perform variable selection inherently, by shrinking small dependencies to zero.^9-11 However, it is difficult to test the assumptions which penalised approaches require for valid model selection in real world datasets.^12,13 Furthermore, standard model selection approaches such as cross-validation and the Bayesian information criterion may be unreliable for model selection in the high-dimensional setting.^14,15

Penalised regression

The inverse covariance matrix, ${\left(X^{T}X\right)}^{-1}$ , is undefined if $n<p$ , precluding the use of ordinary least squared regression.^13,16 Penalised regression methods facilitate modelling in the high-dimensional setting through the addition of bias terms. $L_0$ , $L_1$ and $L_2$ penalised linear regression may be generally formulated such that:

$$\begin{array}{c}{\widehat{\beta }}^{L_0,L_1,L_2}:=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\left({\beta }_0,\beta \right)\in {\mathrm{ℝ}}^{p+1}}{\mathrm{m}\mathrm{i}\mathrm{n}}\left\{\begin{array}{l}\frac{1}{2}{\Vert y-{\beta }_0-X\beta \Vert }_2^2+{\lambda }_0{\Vert \beta \Vert }_0\hfill \\ +{\lambda }_1{\Vert \beta \Vert }_1+{\lambda }_2{\Vert \beta \Vert }_2^2\hfill \end{array}\right\}\hfill \end{array}$$ (1)

Here, notation is conventionally abused such that the $L_0$ ‘pseudo-norm’ counts the number of nonzero elements in $\beta$ . ¹⁰

$$\begin{array}{c}\Vert \beta {\Vert }_0:={\displaystyle \sum }_{i=1}^p\mathbb{I}\left\{{\beta }_i\ne 0\right\}\hfill \end{array}$$ (2)

Ridge regression ¹⁷ penalises the model by the $L_2$ norm of the coefficients ( ${\lambda }_0=0,{\lambda }_1=0,{\lambda }_2\ne 0)$ , balancing predictive error against coefficient magnitude. The imposed preference for smaller coefficients is termed ‘shrinkage’. The magnitude of the shrinkage effect is controlled by the ${\lambda }_2$ hyperparameter.

$${\widehat{\beta }}^{Ridge}:=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\left({\beta }_0,\beta \right)\in {\mathrm{ℝ}}^{p+1}}{\mathrm{m}\mathrm{i}\mathrm{n}}\left\{\frac{1}{2}{\Vert y-{\beta }_0-X\beta \Vert }_2^2+{\lambda }_2{\Vert \beta \Vert }_2^2\right\}$$ (3)

Ridge regression partially alleviates instability under collinearity by constraining coefficient magnitude. ¹⁶ The Least Absolute Selection and Shrinkage Operator (LASSO) ¹¹ penalty penalises the model by the $L_1$ norm of the coefficients, ( ${\lambda }_0=0,{\lambda }_1\ne 0,{\lambda }_2=0)$ .

$${\widehat{\beta }}^{LASSO}:=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\left({\beta }_0,\beta \right)\in {\mathrm{ℝ}}^{p+1}}{\mathrm{m}\mathrm{i}\mathrm{n}}\left\{\frac{1}{2}{\Vert y-{\beta }_0-X\beta \Vert }_2^2+{\lambda }_1{\Vert \beta \Vert }_1\right\}$$ (4)

The LASSO approach has ‘oracle’ properties under some conditions, meaning that predictions are nearly as good as if the true set of predictor variables were known.^18,19 An additional benefit of LASSO shrinkage is a tendency to shrink small coefficients to zero, leading to a ‘sparse’ $\widehat{\beta }$ , in which non-zero coefficients are deemed predictive. Thus, LASSO inherently performs variable selection. ¹¹ This behaviour is highly useful in bioinformatics, where analytic tasks often require the selection of a small number of predictive variables given a large candidate set. However, the lasso model structure is subject to inconsistency under subsampling. ¹² The Elastic Net ²⁰ is a combines the sparsity of $L_1$ penalisation with the consistency of $L_2$ penalisation ( ${\lambda }_0=0,{\lambda }_1\ne 0,{\lambda }_2\ne 0$ ), with improved results in several bioinformatic studies.^1,21 Penalties of ridge regression, LASSO and elastic net affect large coefficients more than small coefficients, biassing coefficient estimates. ‘Best subset selection’ provides a theoretical solution to this issue through the selection of the optimal model attainable with $k\in \mathbb{N}$ or fewer predictor variables, such that ¹⁰ :

$$\begin{array}{c}{\widehat{\beta }}^{BestSubset}:=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\left({\beta }_0,\beta \right)\in {\mathrm{ℝ}}^{p+1}}{\mathrm{m}\mathrm{i}\mathrm{n}}\left\{\frac{1}{2}{\Vert y-{\beta }_0-X\beta \Vert }_2^2\right\}\hfill \\ subjectto\left({\displaystyle \sum }_{i=1}^p\mathbb{I}\left\{{\beta }_i\ne 0\right\}\right)\le k\end{array}$$ (5)

Thus, for some ${\lambda }_0\in \mathrm{ℝ}$ , we have an equivalent Lagrangian expression:

$$\begin{array}{c}{\widehat{\beta }}^{BestSubset}:=\mathrm{a}\mathrm{r}\mathrm{g}\underset{\left({\beta }_0,\beta \right)\in {\mathrm{ℝ}}^{p+1}}{\mathrm{m}\mathrm{i}\mathrm{n}}\left\{\frac{1}{2}{\Vert y-{\beta }_0-X\beta \Vert }_2^2+{\lambda }_0{\Vert \beta \Vert }_0\right\}\hfill \end{array}$$ (6)

Best subset selection be may be approximated through $L_0$ penalisation in some conditions $({\lambda }_0\ne 0,{\lambda }_1=0,{\lambda }_2=0)$ . ¹⁰ $L_0$ penalisation applies no shrinkage to the selected predictors, resulting in unbiased coefficient estimates. ¹⁰ This combination of simplicity and unbiasedness has been described as a ‘holy grail’ of sparse modelling. ⁹ However, models suffer from inconsistency. ²² Furthermore, issues such as non-convexity and NP-hardness complicate best-subset model selection.^9,23 Recent developments such as mixed integer optimisation ¹⁰ have facilitated best subset model learning. Combinations of $L_0$ penalties with $L_1$ ( ${\lambda }_0\ne 0,{\lambda }_1\ne 0,{\lambda }_2=0$ ) or $L_2$ ( ${\lambda }_0\ne 0,{\lambda }_1=0,{\lambda }_2\ne 0$ ) have been suggested to increase the consistency of best subset selection whilst maintaining minimal bias. ²⁴

Assessing variable selection in genomic models

The true generating distribution for observational biological data is typically uncertain, complicating validation of estimated coefficient vectors. Consequently, many model assessments have employed synthetic^9,15,24-27 or semi-synthetic^1,10,28-30 datasets to assess variable selection performance. Real data analyses have focussed primarily on the models’ predictive capacity.^31-33 Accurate predictions may not guarantee correct model structure, especially in the highly collinear conditions commonly encountered in genomics. The representativeness of synthe-tic datasets is both uncertain and untestable. ²⁹ Further-more, results of these studies have been discordant, suggesting dependence on the benchmark datasets and validation techniques.^9,10

Genomic databases such as REACTOME ³⁴ and KEGG ³⁵ contain experimentally verified interactions, which may be used to externally validate genomic model structure. This approach has been used in previous analyses^27,29,36,37 and is limited by the uncertain completeness of such databases. Furthermore, the activity profile of interactions between a given set of genes may change with experimental conditions and unobserved confounders.^38,39 Consequently, the set of active predictors for a specific dataset may not align exactly with a static database. Finally, effect sizes may not be comparable between documented interactions, precluding the assessment of model coefficients by this method. Data-partitioning facilitates model validation without ground truth data, by assessing model generalisability to unseen observations. As training and validation observations are sampled from the same data, their distribution is asymptotically identical. However, the distribution may be difficult to estimate when $n\ll p$ , and data-partitioning favours excessively complex models in this setting.^14,15

Given the limitations of currently available methods for assessment of variable selection performance in genomic data, an urgent need exists for a novel approach.

Study objectives

The primary objectives of this study were to:

• Provide a real-world data-driven approach for comparing performance of high dimensional model inference algorithms in cancer genomics for both prediction and variable selection. We evaluate models by simulating $n\ll p$ conditions in real $n>p$ genomic datasets, allowing for robust evaluation of predictions in large-sample test partitions.

• Compare the performance of penalised linear regression methods for prediction and variable selection.

• Compare algorithmic selection by internal cross-validation to preselection according to performance in external test datasets under our validation approach.

These objectives are realised by subsampling real $n>p$ genomic datasets to simulate $n\ll p$ conditions, allowing for robust data-driven validation of model structure and predictions in large-sample test partitions.

Materials and Methods

Data

Five cancer genomics datasets were extracted from Gene Expression Omnibus ⁴⁰ with the GEOquery library. ⁴¹ Local institutional review board approval and informed participant consent were documented in each data publication.^42-46

GSE73002

GSE73002 ⁴² contains serum miRNA expression profiles for 4113 individuals; 1280 with breast cancer, 54 with benign breast disease, 63 with non-benign breast disease, 451 with various other cancers and 2836 non-cancer controls. Participants with breast cancer were recruited through admissions and referrals to the National Cancer Centre Hospital Japan between 2008 and 2014. Exclusion criteria were (1) administration of medication prior to serum sampling and (2) advanced cancer in other organs. Controls were recruited from (1) National Cancer Centre Biobank, Yokohama Minoru clinic and the Toray Industries staff. Samples from individuals with non-benign breast diseases and other cancers were extracted from the National Cancer Centre Biobank. miRNA expression was measured with was collected on the Toray Industries 3D-Gene Human miRNA Oligo Chip microarray.

GSE137140

GSE137140 ⁴³ contains serum miRNA expression profiles for lung cancer patients. About 1566 pre-operative and 180 post-operative samples are available, in addition to 2178 samples from patients without cancer, collected from the National Cancer Centre Japan and the Yokohama Minoru Clinic. Exclusion criteria were (1) miRNA expression quality check failure, (2) history of other malignancy, (3) missing clinical information, (4) pre-collection therapy and (5) over 180 days had passed between collection and surgery. miRNA expression was measured with was collected on the Toray Industries 3D-Gene Human miRNA Oligo Chip microarray.

GSE103322

GSE103322 ⁴⁴ contains full length single-cell RNAseq data from 5902 cells extracted from 18 patients with stage I to IV squamous cell carcinoma (SCC) of the oral cavity at the Massachusetts Eye and Ear Infirmary. Tissue samples were extracted from surgical biopsies of the primary tumour or lymph node. Sequencing was performed on the Illumina Nextseq 500 platform and transcript-per-million values reported.

GSE146026

GSE146026 ⁴⁵ contains single-cell RNAseq data from 22 ascites samples in 11 patients with high-grade serous ovarian cancer at Brigham and Women’s Hospital and the Dana-Farber Cancer Institute. About 9609 CD45+ depleted samples, profiled with 10× were included in this analysis. Sequencing was performed on the Illumina NextSeq 500 platform and transcript-per-million values reported.

GSE89567

GSE89567 ⁴⁶ contains 6341 single-cell RNAseq profiles from patients with isocitrate dehydrogenase mutant astrocytoma at Massachusetts General Hospital. Tumour tissue was collected from surgical resections and malignancy confirmed under frozen section. Following disaggregation, profiling was performed by Smart-seq2. Sequencing was performed on the Illumina NextSeq 500 and transcript-per-million values reported.

Data preprocessing

Where datasets had $>5000$ variables (GSE103322 and GSE146026), subspaces were extracted, retaining the 1000 variables with the fewest nonzero entries. Datasets were transformed with the Gaussian ECDF function^47,48:

$$\begin{array}{c}{X}_{i,j}:={\Phi }^{-1}\left(\frac{1}{n}{\displaystyle \sum }_{k=1}^n\mathbb{I}\left\{X_{k,j}\le X_{i,j}\right\}\right)\hfill \end{array}$$ (7)

Here $\Phi (\cdot )$ is the standard normal cumulative distribution function. To ensure uniqueness of the gold-standard model, QR-factorisation was performed, and perfectly collinear variables were removed.

$$\begin{array}{c}X=QR{P}^T\hfill \end{array}$$ (8)

Here $Q$ is an orthogonal matrix, $R$ is an upper triangular matrix and $P$ is a permutation matrix. A full-rank subspace was extracted from $X$ using QR factorisation, such that:

$$\begin{array}{c}X:=X{P}_{:,i\le rank\left(X\right)}^T\hfill \end{array}$$ (9)

Experiment setup

In each experiment, 500 design variables and a response were randomly selected from the available gene expression variables in 1 of the 5 datasets. A small proportion of the observations $(n\in \left\{25,75,150\right\})$ were randomly selected for training and the remainder held out for validation. $L_0,L_0{L}_1,L_0{L}_2,L_1$ and $L_1{L}_2$ penalised regression models were fitted using default library parameters (Table 1). Regularisation hyperparameters were selected by either 5-fold or 10-fold cross-validation on the training observations, optimising the mean squared error, a typical approach in genomic analyses.^1,6,7,49,50 The same cross-validation folds were employed for each penalisation method in a given experiment. Predictive performance and variable selection performance were assessed using the remaining test observations. Experiments were repeated for 100 different training samples, for each of 5 datasets and for both cross-validation routines, yielding 1000 experiments with which to compare penalisation methods for each sample size.

Table 1.

Penalised regression methods applied in this analysis.

Pseudonym	Notation	Penalty	Implementation	Reference
Best-subset selection	L 0	λ0 ≠ 0, λ1 = 0, λ2 = 0	L0Learn 1.2.0 24	Hastie et al 9 and Bertsimas et al 10
			Loss = ‘SquaredError’
			Penalty = ‘L0’
			Algorithm = ‘CD’
			Nlambda = 100
			nGamma = 10
			gammaMax = 10
			gammaMin = 1e-04
			partialSort = TRUE
			maxIters = 200
			tol = 1e-06
			activeset = TRUE
			activesetnum = 3
			maxswaps = 1000
			scaledownFactor = 0.8
			screenSize = 1000
			autoLambda = TRUE
			nFolds = 5
			excludeFirstK = 0
			intercept = FALSE
L 0 L 1	$L_0{L}_1$	${\lambda }_0\ne 0,{\lambda }_1\ne 0,{\lambda }_2=0$	L0Learn 1.2.0	Hazimeh and Mazumder 24
			Same as above except: Penalty = ‘L0L1’
L 0 L 1	$L_0{L}_2$	${\lambda }_0\ne 0,{\lambda }_1=0,{\lambda }_2\ne 0$	L0Learn 1.2.0	Hazimeh and Mazumder 24
			Same as above except: Penalty = ‘L0L2’
LASSO	$L_1$	${\lambda }_0=0,{\lambda }_1\ne 0,{\lambda }_2=0$	glmnet 4.2-051,52	Tibshirani 11
			family = ‘gaussian’
			alpha = 1
			weights = NULL
			offset = NULL
			lambda = NULL
			lambda.min.ratio = 1e-4
			type.measure = ‘mse’
			foldid = NULL
			alignment = ‘lambda’
			grouped = TRUE
			relax = FALSE
			alpha = 0
			parallel = FALSE
Elastic net	L 1 L 2	${\lambda }_0=0,{\lambda }_1\ne 0,{\lambda }_2\ne 0$	glmnet 4.2-0	Zou and Hastie 20
			Same as above except: alpha = {0, 0.11, 0.22, 0.33, 0.44, 0.56, 0.67, 0.78, 0.89, 1}

$\lambda$ Notation corresponds to the regularisation hyperparameters defined in equation (1).

Metrics

Model assessment metrics and notation followed previous comparative analyses.^9,10 As the true coefficient vector, $\beta \in {\mathrm{ℝ}}^p$ , was unknown in our experiments, it was estimated by ordinary least squares regression (without intercept) on the whole dataset $\left(n\approx 4000,p=500\right)$ , such that:

$$\begin{array}{c}\beta \approx {\beta }^{*}={\left(X^{T}X\right)}^{-1}{X}^{T}y\hfill \end{array}$$ (10)

Thus, ${\beta }^{*}$ represents a noisy gold-standard rather than strict ground truth. Here $x_0\in {\mathrm{ℝ}}^p$ denotes the test observations from the design matrix and $y_0\in \mathrm{ℝ}$ denotes the associated response. Hastie et al ⁹ measured 3 metrics of predictive performance – proportion of variance explained (PVE), relative risk (RR) and relative test error (RTE).

$$\begin{array}{c}PVE\left(\widehat{\beta }\right)=1-\frac{\mathbb{E}\left[{\left(y_0-x_0^T\widehat{\beta }\right)}^2\right]}{Var\left(y_0\right)}\hfill \end{array}$$ (11)

Higher PVE indicates superior fit, and PVE is limited by the signal to noise ratio (SNR) such that ⁹ :

$$\begin{array}{c}PVE\left(\widehat{\beta }\right)\le \frac{SNR}{1+SNR}\le 1\hfill \end{array}$$ (12)

Relative risk (RR) was employed as an performance metric in Bertsimas’ analysis. ¹⁰ Optimal relative risk is 0 and nullity is 1.

$$\begin{array}{c}RR\left(\widehat{\beta }\right)=\frac{\mathbb{E}\left[{\left(x_0^T\beta -x_0^T\widehat{\beta }\right)}^2\right]}{\mathbb{E}\left[{\left(x_0^T{\beta }^{*}\right)}^2\right]}\hfill \end{array}$$ (13)

Relative test error (RTE) compares error to the noise variance:

$$\begin{array}{c}RTE\left(\widehat{\beta }\right)=\frac{\mathbb{E}\left[{\left(y_0-x_0^T\widehat{\beta }\right)}^2\right]}{\mathbb{E}\left[{\left(y_0-x_0^T{\beta }^{*}\right)}^2\right]}\hfill \end{array}$$ (14)

Following calls for model coefficient similarity assessment, ⁹ we measured the cosine similarity of $\widehat{\beta }$ and ${\beta }^{*}$ , such that:

$$\begin{array}{c}CoefficientSimilarity\left(\widehat{\beta }\right)=\frac{〈\widehat{\beta },{\beta }^{*}〉}{\sqrt{〈\widehat{\beta },\widehat{\beta }〉〈{\beta }^{*},{\beta }^{*}〉}}\hfill \end{array}$$ (15)

Active (non-zero) variable selection performance was also estimated under ${\beta }^{*}.$ Coefficient significance of was estimated with t-tests:

$$\begin{array}{c}\mathrm{\mathbb{P}}\left({\beta }_i^{*}=0\right)~t_{n-p}\left({\beta }_i^{*}\right)=\frac{{\beta }_i^{*}}{SE\left({\beta }_i^{*}\right)}\hfill \end{array}$$ (16)

Significance was adjusted for multiple comparisons using false-discovery-rate (FDR) control ⁵³ and predictors were classified according to a cutoff $\alpha =0.05$ . Precision, recall, F1 score were measured. Hereafter, these metrics are referred to collectively as the ‘discrete’ variable selection metrics. Undefined variable selection results (due to division-by-zero errors) were replaced with zeros. Figure 1 depicts the variable selection validation method graphically.

Figure 2.

Test predictive performance. Medians are represented by boxplot centrelines; first and third quartiles by hinges; and minima and maxima by whiskers.

Figure 1.

Graphical visualisation of variable selection validation method. ‘Gold-standard’ regression models were trained on subspaces of large genomic datasets ( $n4000$ , $p=500$ ). T-tests were performed on gold standard coefficient estimates and significant coefficients were identified according to a false-discovery rate controlled alpha cutoff of .05. Penalised regression models were trained on small samples from these subspaces ( $n\in \left\{25,75,150\right\},p=500$ ) and validated against the gold-standard models.

To evaluate our model validation approach, we deployed it as a penalty preselection method, comparing it to traditional selection by minimisation of the internal cross-validation error. For each experiment, for each of 3 comparison metrics (PVE, F1 and coefficient similarity), a penalisation method was ‘preselected’ according to performance in experiments of equivalent sample size in the other 4 datasets. In each relevant experiment, penalisation methods’ performances were ranked and the method with the lowest rank aggregate performance was selected. The test performance of this method was compared to that of the penalisation method which yielded the lowest mean squared error on internal cross-validation. Overall performance of preselected penalties was compared to internal cross-validation selected penalties using a 2-sided paired t-test over all 3000 experiments.

Results

Experiment characteristics

Experiments represented a broad range of signal:noise ratios (Median: 0.94, IQR: [0.38, 2.68]), with high SNR in experiments sampled from GSE73002 (Median: 12.03, IQR: [4.69, 28.34]), intermediate SNR in GSE137140 (Median: 1.58, IQR: [1.16, 2.41]) and low SNRs in GSE103322 (Median: 0.47, IQR: [0.31, 0.85]), GSE146026 (Median: 0.39, IQR: [0.23, 0.77]) and GSE89567 (Median: 0.44, IQR: [0.31, 0.71]). The number of significant coefficients in each experiment was typically small (Median: 7.00, IQR: [1.00, 17.00]) and followed a right-skewed distribution (95th Quantile: 40.00, Max: 106.00). This is consistent with the scale-free property of genomic networks, in which a small number of genes have many interactions.

Predictive performance

Predictive performance metrics are provided in Figure 2 and Table 2. $L_0{L}_2$ -penalised models achieved the highest PVE overall (Median: 0.23, IQR: [0.04, 0.52]). However, this penalty performed unreliably in the n = 25 experiments, demonstrating strongly negative PVE values (ie, worse-than-random performance) in some cases (Min: −1.32, 5th Quantile: −0.33). Similarly, $L_0{L}_1$ -penalised models exhibited strong overall PVE (Median: 0.17, IQR: [−0.00, 0.50]) and variable performance in the n = 25 setting (Min: −1.70, 5th Quantile: −0.39). L1L2 penalised models achieved comparable overall PVE (Median: 0.19, IQR: [−0.00, 0.49]), with superior worst-case reliability in the n = 25 experiments (Min: −0.71, 5th Quantile: −0.01). Likewise, $L_1$ penalisation provided moderate overall PVE (Median: 0.13, IQR: [−0.00, 0.47]) and robust worst-case PVE scores in the n = 25 experiments (Min: −0.35, 5th Quantile: −0.01). $L_0$ penalisation selected null models in most experiments, returning null PVE (Median: 0.01, IQR: [−0.01, 0.40]). PVE was highly associated with SNR (: 0.61, 95% CI: [0.6, 0.62], P < 2e-16. PVE:SNR curves (Figure 3) demonstrate that $L_0{L}_1$ and $L_0{L}_2$ underperformance was mainly limited to the noisiest cases. $L_1$ and $L_1{L}_2$ penalisation were infrequently negative, even in noisy experiments. Conversely, $L_1$ and $L_1{L}_2$ penalisation demonstrated poorer PVE reliability than $L_0{L}_1$ and $L_0{L}_2$ penalisation in moderate SNR conditions. Relative risk performance distributions reflected those of PVE, with the best overall median performance observed in $L_0{L}_2$ (Median: 0.48, IQR: [0.24, 0.81]) and $L_0{L}_1$ -penalised models (Median: 0.58, IQR: [0.28, 1.00]), despite unreliable worst-case performance observed in n = 25 settings. Moderate relative risk performance was achieved through $L_1$ (Median: 0.68, IQR: [0.31, 1.00]) and $L_1{L}_2$ penalisation (Median: 0.23, IQR: [0.04, 0.52]), with superior worst-case reliability. RTE performance highlighted the shortcomings of L0 penalisation (Median: 1.79, IQR: [1.45, 2.33]).

Table 2.

Predictive performance of each penalisation method.

Penalty	N	Metric	Median	IQR
L 0	25	Proportion of variance explained	0	[0.00, 0.00]
L 0 L 1	25	Proportion of variance explained	0	[0.00, 0.06]
L 0 L 2	25	Proportion of variance explained	0	[0.00, 0.08]
L 1	25	Proportion of variance explained	0	[0.00, 0.00]
L 1 L 2	25	Proportion of variance explained	0	[0.00, 0.06]
L 0	75	Proportion of variance explained	0	[0.00, 0.20]
L 0 L 1	75	Proportion of variance explained	0	[0.00, 0.16]
L 0 L 2	75	Proportion of variance explained	0.02	[0.00, 0.14]
L 1	75	Proportion of variance explained	0	[0.00, 0.22]
L 1 L 2	75	Proportion of variance explained	0.02	[0.00, 0.09]
L 0	150	Proportion of variance explained	0	[0.00, 0.33]
L 0 L 1	150	Proportion of variance explained	0.05	[0.00, 0.20]
L 0 L 2	150	Proportion of variance explained	0.05	[0.00, 0.18]
L 1	150	Proportion of variance explained	0.07	[0.00, 0.29]
L 1 L 2	150	Proportion of variance explained	0.02	[0.00, 0.10]
L 0	25	Relative risk	1	[0.75, 1.75]
L 0 L 1	25	Relative risk	0.94	[0.45, 1.07]
L 0 L 2	25	Relative risk	0.67	[0.36, 1.04]
L 1	25	Relative risk	1	[0.48, 1.00]
L 1 L 2	25	Relative risk	0.75	[0.39, 1.00]
L 0	75	Relative risk	0.87	[0.40, 1.00]
L 0 L 1	75	Relative risk	0.56	[0.26, 1.00]
L 0 L 2	75	Relative risk	0.46	[0.23, 0.78]
L 1	75	Relative risk	0.63	[0.29, 1.00]
L 1 L 2	75	Relative risk	0.52	[0.26, 1.00]
L 0	150	Relative risk	0.62	[0.31, 1.00]
L 0 L 1	150	Relative risk	0.44	[0.20, 0.75]
L 0 L 2	150	Relative risk	0.37	[0.19, 0.61]
L 1	150	Relative risk	0.52	[0.24, 0.93]
L 1 L 2	150	Relative risk	0.44	[0.22, 0.77]
L 0	25	Relative test error	0	[−0.30, 0.17]
L 0 L 1	25	Relative test error	0.02	[−0.03, 0.38]
L 0 L 2	25	Relative test error	0.13	[−0.02, 0.44]
L 1	25	Relative test error	0	[−0.00, 0.34]
L 1 L 2	25	Relative test error	0.1	[−0.00, 0.41]
L 0	75	Relative test error	0.06	[−0.00, 0.42]
L 0 L 1	75	Relative test error	0.21	[−0.00, 0.52]
L 0 L 2	75	Relative test error	0.25	[0.05, 0.53]
L 1	75	Relative test error	0.18	[−0.00, 0.49]
L 1 L 2	75	Relative test error	0.22	[−0.00, 0.51]
L 0	150	Relative test error	0.18	[−0.00, 0.48]
L 0 L 1	150	Relative test error	0.26	[0.07, 0.56]
L 0 L 2	150	Relative test error	0.28	[0.10, 0.57]
L 1	150	Relative test error	0.22	[0.02, 0.52]
L 1 L 2	150	Relative test error	0.26	[0.06, 0.54]

Abbreviation: IQR, interquartile range.

For each sample size, 100 experiments were sampled from each of 5 datasets, for each of 2 cross-validation routines. IQR denotes interquartile range.

Figure 3.

Proportion of variance explained in test observations versus signal:noise ratio. Signal:noise ratio was estimated by the residuals of the gold standard models fitted to the complete dataset $(n4000$ ) with ordinary least squares regression. Medians are represented by boxplot centrelines; first and third quartiles by hinges; and minima and maxima by whiskers.

Variable selection

Variable selection performance metrics are provided in Figure 4 and Table 3. $L_1{L}_2$ -penalised models achieved high coefficient similarity overall (Median: 0.17, IQR: [0.09, 0.24]), although many nonzero coefficients were included (Median: 59.50, IQR: [11.00, 500.00]). Consequently, in n = 75 experiments, strong recall (Median: 0.33, IQR: [0.00, 1.00]) and poor precision were observed (Median: 0.33, IQR: [0.00, 1.00]). $L_0{L}_2$ -penalisation also achieved high coefficient similarity (Median: 0.13, IQR: [0.06, 0.20]), with fewer nonzero coefficients (Median: 25.00, IQR: [6.00, 67.00]). $L_0{L}_2$ penalisation achieved the highest F1 score in n = 75 (Median: 0.04, IQR: [0.00, 0.14]) and n = 150 experiments (Median: 0.07, IQR: [0.00, 0.19]). $L_0{L}_1$ -penalised models performed similarly in terms of coefficient similarity (Median: 0.08, IQR: [0.02, 0.15]) using fewer nonzero parameters (Median: 8.00, IQR: [2.00, 19.00]). Moderate F1 scores were achieved in n = 75 and (Median: 0.00, IQR: [0.00, 0.12]) and n = 150 experiments (Median: 0.07, IQR: [0.00, 0.18]) $L_1$ -penalised models achieved moderate coefficient similarity (Median: 0.05, IQR: [0.00, 0.15]) through models with very few nonzero coefficients (Median: 5.00, IQR: [0.00, 10.00]). Although $L_1$ -penalisation achieved moderate F1 score in n = 150 experiments (Median: 0.08, IQR: [0.00, 0.21]), it underperformed in n = 75 experiments (Median: 0.00, IQR: [0.00, 0.16]). $L_0$ -only penalisation produced highly parsimonious models, with very few nonzero coefficients (Max: 67.00, 95th Quantile: 10.05). However, variable selection performance was poor by every metric. Test performance summaries for prediction and variable selection are provided in Supplemental Table S1.

Figure 4.

Variable selection performance. Gold standard coefficient vectors were extracted from ordinary least squares regression models fitted to the full dataset $(n4000)$ . Coefficient significance was estimated with t-tests and true predictors were defined by FDR- $\alpha <0.05$ . Medians are represented by boxplot centrelines; first and third quartiles by hinges; and minima and maxima by whiskers.

Table 3.

Variable selection performance of each penalisation method.

Penalty	N	Metric	Median	IQR
L 0	25	Coefficient similarity	2	[0.00, 3.00]
L 0 L 1	25	Coefficient similarity	6	[1.00, 16.00]
L 0 L 2	25	Coefficient similarity	16	[4.00, 42.00]
L 1	25	Coefficient similarity	1	[0.00, 6.00]
L 1 L 2	25	Coefficient similarity	25.5	[4.00, 500.00]
L 0	75	Coefficient similarity	2	[0.00, 3.00]
L 0 L 1	75	Coefficient similarity	8	[2.00, 20.00]
L 0 L 2	75	Coefficient similarity	27.5	[6.00, 75.00]
L 1	75	Coefficient similarity	5	[0.00, 10.00]
L 1 L 2	75	Coefficient similarity	98.5	[13.00, 500.00]
L 0	150	Coefficient similarity	3	[1.00, 4.00]
L 0 L 1	150	Coefficient similarity	12	[3.00, 23.00]
L 0 L 2	150	Coefficient similarity	35	[10.00, 79.00]
L 1	150	Coefficient similarity	8	[1.00, 13.00]
L 1 L 2	150	Coefficient similarity	500	[20.00, 500.00]
L 0	25	F1 score	0.01	[0.00, 0.06]
L 0 L 1	25	F1 score	0.03	[0.00, 0.08]
L 0 L 2	25	F1 score	0.07	[0.02, 0.12]
L 1	25	F1 score	0	[0.00, 0.07]
L 1 L 2	25	F1 score	0.1	[0.02, 0.16]
L 0	75	F1 score	0.04	[0.00, 0.11]
L 0 L 1	75	F1 score	0.08	[0.03, 0.15]
L 0 L 2	75	F1 score	0.13	[0.08, 0.20]
L 1	75	F1 score	0.07	[0.00, 0.16]
L 1 L 2	75	F1 score	0.17	[0.11, 0.24]
L 0	150	F1 score	0.07	[0.00, 0.15]
L 0 L 1	150	F1 score	0.13	[0.07, 0.21]
L 0 L 2	150	F1 score	0.19	[0.12, 0.26]
L 1	150	F1 score	0.11	[0.00, 0.21]
L 1 L 2	150	F1 score	0.22	[0.16, 0.30]
L 0	25	Precision	0	[0.00, 0.00]
L 0 L 1	25	Precision	0	[0.00, 0.05]
L 0 L 2	25	Precision	0	[0.00, 0.12]
L 1	25	Precision	0	[0.00, 0.00]
L 1 L 2	25	Precision	0.08	[0.00, 1.00]
L 0	75	Precision	0	[0.00, 0.04]
L 0 L 1	75	Precision	0	[0.00, 0.12]
L 0 L 2	75	Precision	0.09	[0.00, 0.31]
L 1	75	Precision	0	[0.00, 0.12]
L 1 L 2	75	Precision	0.33	[0.00, 1.00]
L 0	150	Precision	0	[0.00, 0.07]
L 0 L 1	150	Precision	0.07	[0.00, 0.25]
L 0 L 2	150	Precision	0.17	[0.00, 0.43]
L 1	150	Precision	0.06	[0.00, 0.22]
L 1 L 2	150	Precision	0.5	[0.05, 1.00]
L 0	25	Recall	0	[0.00, 0.00]
L 0 L 1	25	Recall	0	[0.00, 0.06]
L 0 L 2	25	Recall	0	[0.00, 0.08]
L 1	25	Recall	0	[0.00, 0.00]
L 1 L 2	25	Recall	0.01	[0.00, 0.09]
L 0	75	Recall	0	[0.00, 0.06]
L 0 L 1	75	Recall	0	[0.00, 0.12]
L 0 L 2	75	Recall	0.04	[0.00, 0.14]
L 1	75	Recall	0	[0.00, 0.16]
L 1 L 2	75	Recall	0.04	[0.00, 0.12]
L 0	150	Recall	0	[0.00, 0.12]
L 0 L 1	150	Recall	0.07	[0.00, 0.18]
L 0 L 2	150	Recall	0.07	[0.00, 0.19]
L 1	150	Recall	0.08	[0.00, 0.21]
L 1 L 2	150	Recall	0.04	[0.00, 0.15]

Abbreviation: IQR, interquartile range.

For each sample size, 100 experiments were sampled from each of 5 datasets, for each of 2 cross-validation routines, yielding 1000 experiments for each comparison.

Comparing preselection to internal validation

Penalty preselection led to small, yet significant, performance gains in PVE ( $t_{2999}:$ 8.66, µ: 0.016, 95% CI: [0.012, 0.020], $P$ < 10⁻¹⁶), F1 score ( $t_{2999}:$ 4.66, µ: 0.016, 95% CI: 0.006 [0.003, 0.008], P = 3.3 × 10⁻⁶) and coefficient similarity ( $t_{2999}:$ 15.99, µ: 0.02, 95% CI: [0.018, 0.023], $P$ < 10⁻¹⁶) when compared to selection by internal cross-validation. In many cases the same penalisation method was selected under preselection and internal validation, leading to equivalent performance. Although aggregated improvements under preselection were statistically significant, internal validation outperformed in some experiments (Figure 5). Cumulative distribution functions of the performance improvements yielded under preselection are provided in (Figure 5). In other experiments internal validation outperformed preselection (Table 4).

Figure 5.

Cumulative distribution functions for performance improvement under penalty preselection compared with comparison to selection by internal cross-validation. For each experiment and each comparison metric, the penalisation method was selected with the best test performance in the other 4 datasets. This ‘preselected’ penalisation method was compared to that which minimised the mean squared error in internal cross-validation. About 3000 experiments were included in the comparison.

Table 4.

Paired-tests of mean performance difference using preselection compared to selection by internal cross-validation. Penalisation routines were ‘preselected’ according to performance in the other 4 datasets. Mean difference refers to preselected performance minus internal cross-validation performance. About 3000 experiments were included in the comparison.

Metric	t-Score (df = 2999)	Mean performance gain under preselection	95% CI	P-value (2-sided)
Proportion of variance explained	8.66	0.016	[0.012, 0.020]	$<{10}^{-16}$
F1	4.66	0.006	[0.003, 0.008]	$3.3\times {10}^{-6}$
Coefficient similarity	15.99	0.020	[0.018, 0.023]	$<{10}^{-16}$

Discussion

The optimal penalisation method for a particular dataset depends upon the project objectives, data distribution and noise levels. In most applications, reliability is paramount – the strong median predictive performance provided by $L_0{L}_1$ and $L_0{L}_2$ penalisation is unlikely to compensate for their worst-case performance, which may be undetectable in application. $L_1{L}_2$ penalisation offered strong coefficient similarity, though few coefficients are shrunk to zero, limiting its utility for the selection of parsimonious model structures. $L_1$ and $L_1{L}_2$ penalties also offered reliable test predictions in noisy data. $L_1$ is simpler to implement than combined penalties, requiring tuning of a single hyperparameter. Furthermore, the theory surrounding $L_1$ penalisation in the $n\ll p$ setting is well studied.^1,7,12,54 Various computational implementations of this method are available, and it is the fundamental building block for graph inference methods such as the graphical LASSO ⁵⁵ and the nodewise LASSO. ⁵⁶ $L_0$ penalisation resulted in weakly predictive models and poor variable selection, due primary to inadequate recall. These limitations overshadowed any potential advantage of theoretical unbiasedness. ¹⁰

Penalty preselection yielded small, yet significant improvements over internal cross-validation based selection in each examined metric, demonstrating the value of external data-driven preselection of model learning algorithms for $n\ll p$ datasets. This approach may serve as a complementary methodological validation measure for genomic datasets.

Related work

Bertsimas et al ¹⁰ found that $L_0$ penalisation outperformed the $L_1$ and forward stepwise regression in their comparisons. However, this result was contested in the comparisons of Hastie et al, ⁹ who concluded that $L_1$ outperformed $L_0$ in all but high signal-to-noise conditions. Hazimeh and Mazumder ²⁴ found that $L_0{L}_1$ and $L_0{L}_2$ penalties typically outperformed $L_1$ , ²⁴ a finding which concurs with our experiments.

Limitations

The primary limitation of this analysis is uncertainty regarding the true generating distributions of the datasets. In place of ground truth, a ‘gold-standard’ was set using a much larger number of observations. Thus, our analysis evaluates its capacity to recover the model which would have been found in a much larger study of the same population, a reasonable objective in many clinical studies. As the gold standard models were fitted to a finite number of observations, they were susceptible to some degree of overfitting.

Observations were not strictly partitioned on a patient-disjoint basis. In the typical clinical modelling scenario, estimation of model generalisability to new patients would require patient-disjoint partitioning and validation. ⁵⁷ However, distributional identicality of the training and test data would not have been guaranteed in such conditions, biassing assessment metrics in favour of underfitted models.

Bertsimas and Hastie both considered which SNR ranges were ‘realistic’; Bertsimas generated tasks with SNR $\in [2,10]$ and Hastie examined the SNR $\in [0.05,6]$ setting.^9,10 Our estimated SNRs align with those of Hastie. In the case that the gold standard models overfitted, noise levels would have been underestimated. Therefore, SNR estimates in this analysis are positively biassed. Nonzero coefficients were defined according to a traditional, yet arbitrary significance cutoff – therefore small effects may have been omitted erroneously. Likewise, some spuriously large coefficients may have been included.

Discrete variable selection metrics (precision, recall and F1 score) lacked the graduation required to compare penalisation methods at the n = 25 level. This limitation was particularly important in the setting of active variables estimated according to a sharp significance cutoff. The coefficient similarity metric proved useful in this regard, as it was continuous and independent of any significance cutoff. However, coefficient similarity provides little insight into on model complexity, a central aspect of genomic network inference. Indeed, although $L_1{L}_2$ penalisation optimised the coefficient similarity metric, it selected extremely complex models in most experiments, resulting in weak precision.

Real-world genomic datasets were employed in this analysis. Accordingly, our results are expected to be more representative of actual experimental modelling conditions. Data-driven model assessment was facilitated by the large number of observations available in these datasets. However, our results may not generalise to datasets with incomparably distributed signal or noise. Logistic and Cox regression tasks present addition challenges such as class imbalance and censoring, which are beyond the scope of this analysis.

Conclusions

$L_0{L}_2$ -penalised model provided the best test predictions, though performance was unreliable in noisy data. $L_0{L}_2$ also optimised discrete variable selection metrics. $L_1{L}_2$ -penalisation returned offered reliable test predictions in all settings and superior coefficient similarity. Further research is required to establish the performance of the penalties in classification and survival tasks. Evaluation of learning algorithms according to observed test performance in external genomic datasets yields valuable insights into actual test performance, providing a data-driven complement to internal cross-validation in genomic regression tasks.