Relationship between gene regulation network structure and prediction accuracy in high dimensional regression

Abstract

The least absolute shrinkage and selection operator (lasso) and principal component regression (PCR) are popular methods of estimating traits from high-dimensional omics data, such as transcriptomes. The prediction accuracy of these estimation methods is highly dependent on the covariance structure, which is characterized by gene regulation networks. However, the manner in which the structure of a gene regulation network together with the sample size affects prediction accuracy has not yet been sufficiently investigated. In this study, Monte Carlo simulations are conducted to investigate the prediction accuracy for several network structures under various sample sizes. When the gene regulation network is a random graph, a sufficiently large number of observations are required to ensure good prediction accuracy with the lasso. The PCR provided poor prediction accuracy regardless of the sample size. However, a real gene regulation network is likely to exhibit a scale-free structure. In such cases, the simulation indicates that a relatively small number of observations, such as $N=300$, is sufficient to allow the accurate prediction of traits from a transcriptome with the lasso.

Introduction

Technological advancements have enabled the collection of highly multidimensional data from biological systems^1–4. For example, RNA sequencing quantifies expression levels of thousands of genes. Such omics data is useful in predicting organismal traits, with empirical applications including diagnosis and classification of diseases and prediction of patient survival^5–8 and possible future applications in predicting crop yields⁹, insecticide resistance¹⁰, and environmental adaptation¹¹.

A common challenge in predicting traits from omics data is the dimension of the data far exceeding that of the sample size (known as high-dimensional regression). For example, if one is to apply least-squares estimation in multiple regression (e.g. $\text{trait}\approx {\beta }_0+{\beta }_1{\text{gene}}_1+{\beta }_2{\text{gene}}_2+\cdots$) to predict a trait value from a transcriptome, the sample size needs to be (at least) larger than the number of model parameters. However, because transcriptome studies typically observe thousands of genes, a sample size exceeding the number of genes is not realistic at present. In this case, high-dimensional regression modeling must be considered.

The least absolute shrinkage and selection operator (lasso¹²) is one of the most frequently used methods for high-dimensional regression. It simultaneously achieves variable selection and parameter estimation. Theoretically, the prediction accuracy of the lasso is highly dependent on the correlation structure among exploratory variables; it is high under certain strong conditions, such as the compatibility condition¹³. However, in practice, it is not easy to check whether the compatibility condition holds. Another popular estimation method for high-dimensional regression is principal component regression (PCR¹⁴). PCR is a two-stage procedure: first, principal component analysis is conducted for predictors, following which the regression model on which the principal components are used as predictors is fitted. This method may perform well when the exploratory variables are highly correlated.

It is reasonable to assume that gene regulation networks will result in conditional independence among the levels of gene expression^15–17. Here, two variables are conditionally independent when they are independent given other variables (e.g. two focal variables are independently influenced by a third variable¹⁸). When a random vector of exploratory variables follows a multivariate normal distribution, two variables are conditionally independent if and only if the corresponding element of the inverse covariance matrix is zero. Essentially, the networks are characterized by the nonzero pattern of the inverse covariance matrix.

One of the most notable characteristics of biological networks is their scale-free nature, that is, the degree distribution of the network follows a power-law expressed as $p(x)\propto x^{-\gamma }$ ($\gamma >1$)^19,20. Empirical studies suggest that biological networks are often scale-free^21–23, although exceptions have also been found²⁴. Therefore, it is reasonable to consider the problem of high-dimensional regression when the networks of exploratory variables are scale-free. Here, it should be noted that the relative performance of different high-dimensional regression techniques may depend on sample sizes. However, to the best of our knowledge, the effect of the gene regulation network structure together with sample size on prediction accuracy has not yet been sufficiently investigated.

This paper provides a general simulation framework to study the effects of correlation structure in explanatory variables. As an example, the prediction of ambient temperature from the transcriptome, for which good empirical data is available^11,25, is considered. It should be noted that the implementation of the proposed procedure is independent of the empirical data^11,25; the proposed framework may be applied to predict any consequence of gene expression differences. The proposed framework is based on the Monte Carlo simulations. Three datasets of transcriptome and their traits are generated. The datasets are characterized by the covariance structure of exploratory variables; one of the covariance structures corresponds to the scale-free gene regulation network. Both lasso and PCR are applied to these simulated datasets to investigate the prediction accuracy with different types of gene regulation networks. The sample size is also varied to examine its effect on the prediction accuracy.

The remainder of this paper is organized as follows. Section “Prediction methods for high-dimensional data” describes prediction methods for high-dimensional regression in the given simulation. Section “Simulation framework” discusses the proposed simulation framework. Finally, Section “Concluding remarks” presents the concluding remarks.

Prediction methods for high-dimensional data

Suppose that we have n observations $\{({\mathit{x}}_i,y_i)\mid i=1,\dots ,n\},$ where ${\mathit{x}}_i$ are p-dimensional vector of explanatory variables and $y_i$ are responses $(i=1,\dots ,n)$. Let $X={({\mathit{x}}_1,\dots ,{\mathit{x}}_n)}^T$ and $\mathit{Y}={(y_1,\dots ,y_n)}^T$. Consider the linear regression model:

$$\begin{array}{c}\hfill \mathit{Y}=X\mathit{\beta }+\mathit{ϵ},\end{array}$$

where $\mathit{ϵ}={({ϵ}_1,\dots ,{ϵ}_n)}^T$ is a vector of error variables with $E(\mathit{ϵ})=\mathbf{0}$ and $V(\mathit{ϵ})={\sigma }^2{I}_n$.

Lasso

The lasso minimizes a loss function that consists of quadratic loss with a penalty based on an $L_1$ norm of a parameter vector:

$$\begin{array}{c}\hfill \widehat{\mathit{\beta }}=\underset{\mathit{\beta }}{\text{arg}\text{min}}\frac{1}{2}{\Vert \mathit{Y}-X\mathit{\beta }\Vert }_2^2+\lambda {\Vert \mathit{\beta }\Vert }_1,\end{array}$$ 1

where $\lambda >0$ is a regularization parameter. Because of the nature of the $L_1$ norm in the penalty term, some of the elements of the coefficients are estimated to be exactly zero. Thus, variable selection is conducted, and only variables that correspond to nonzero coefficients affect the responses.

PCR

In some cases, the first few largest eigenvalues of the covariance matrix of predictors (i.e., proportional contributions of principle components) can be considerably large (e.g., spiked covariance model²⁶). In such a case, the lasso may not function effectively in terms of both prediction accuracy and consistency in model selection, because the conditions for its effective performance (e.g., compatibility condition²⁷) may not be satisfied. This issue could be addressed using PCR because it transforms data with a large number of highly correlated variables into a few principal components. In the first stage of PCR, principal component analysis is applied to predictors. The ith observation of predictor, ${\mathit{x}}_i$, is linearly mapped onto a $d(<p)$-dimensional vector, ${\mathit{z}}_i=A^T{\mathit{x}}_i$, where A is a $p\times d$ matrix. The matrix A is obtained by the following least squares optimization problem²⁸:

$$\begin{array}{c}\hfill A=\underset{A}{\text{arg}\text{min}}\sum _{i=1}^n{\Vert ({\mathit{x}}_i-\overline{\mathit{x}})-A{A}^T({\mathit{x}}_i-\overline{\mathit{x}})\Vert }_2^2\text{subject to}{A}^{T}A=I_d.\end{array}$$

here, $\overline{\mathit{x}}$ is the sample mean vector, that is, $\overline{\mathit{x}}={\sum }_{i=1}^n{\mathit{x}}_i/n$. In this work, the number of projected dimensions, d, was chosen such that d principle components collectively explain 90% or more variance (and $d-1$ principle components do not). Then, in the second stage, regression analysis is conducted, for which the principal components, $\{{\mathit{z}}_1,\dots ,{\mathit{z}}_n\}$, are used as predictors. Here, the regression coefficients in the second stage are estimated by the lasso.

Simulation framework

An overview of the simulation is presented in Fig. 1. First, the model that defines the relationship between the trait and the levels of gene expression was parameterized. This was done using the empirical data¹¹, which quantified the transcriptome of wild Arabidopsis halleri subsp. gemmifera weekly for two years in their natural habitat as well as bihourly on the equinoxes and solstices (p = 17,205 genes for $n=835$ observations). Three types of simulated data were generated using different covariance matrices of genes, denoted as ${\mathrm{\Sigma }}_j$ ($j=1,2,3$). ${\mathrm{\Sigma }}_1$ is the sample covariance matrix of genes. Generally, none of the elements of the inverse of sample covariance matrix are exactly zero, implying that each gene interacts with all the other genes. Such a fully connected network is ineffective in terms of interpretation of the mechanism of gene regulation. Thus, two other covariance matrices were produced to simulate sparse networks based on the sample covariance matrix ${\mathrm{\Sigma }}_1$. ${\mathrm{\Sigma }}_2$ is generated by the graphical lasso²⁹, which corresponds to the random graph. Although the graphical lasso is widely used because of its computational efficiency, real networks are often scale-free. Therefore, ${\mathrm{\Sigma }}_3$, which corresponds to the scale-free network, was generated here. The estimation of scale-free networks is achieved by the reweighted graphical lasso³⁰. Based on these three covariance matrices ${\mathrm{\Sigma }}_j$ ($j=1,2,3$), the simulated transcriptome data were generated from the multivariate normal distribution. The simulated trait data were generated from simulated transcriptome data. Finally, lasso and PCR were applied to these simulated data to compare their prediction accuracies. The sample sizes of the simulated data were varied to investigate the relationship between prediction accuracy and sample sizes.

Figure 1.

Overview of the simulation.

Evaluation of the estimation procedure

The performance of the estimation procedure is investigated by the following expected prediction error:

$$\begin{array}{c}\hfill E\left[{∥{\mathit{Y}}^{\ast },-,{(X^{\ast })}^T,\widehat{\mathit{\beta }}∥}_2^2\right],\end{array}$$

where $X^{\ast }$ and ${\mathit{Y}}^{\ast }$ follow $X^{\ast }\sim N(\mathbf{0},{\mathrm{\Sigma }}_j)$ $(j=1,2,$ or 3) and ${\mathit{Y}}^{\ast }\sim N({(X^{\ast })}^T\mathit{\beta },{\sigma }^2{I}_n)$, respectively. The estimator $\widehat{\mathit{\beta }}$ is obtained using current observations, while $X^{\ast }$ and ${\mathit{Y}}^{\ast }$ correspond to future observations. The ${\mathrm{\Sigma }}_j$ ($j=1,2,3$), $\mathit{\beta }$, and ${\sigma }^2$ are true values but unknown. In practice, these parameters are defined by using the actual dataset, (X, Y). Detail of setting of these parameters will be presented in the next subsection.

To estimate the expected prediction error, the Monte Carlo simulation is conducted. We first randomly generate training and test data, $({\tilde{X}}_{\mathit{train}},{\tilde{\mathit{Y}}}_{\mathit{train}})$ and $({\tilde{X}}_{\mathit{test}},{\tilde{\mathit{Y}}}_{\mathit{test}})$, respectively. Here, ${\tilde{X}}_{\mathit{train}}$ follows a multivariate normal distribution with mean vector ${\mu }_X$ and variance–covariance matrix ${\mathrm{\Sigma }}_j$, where ${\mu }_X$ is the sample mean of X. Then, ${\tilde{\mathit{Y}}}_{\mathit{train}}$ is generated by ${\tilde{\mathit{Y}}}_{\mathit{train}}={\tilde{X}}_{\mathit{train}}\mathit{\beta }+\mathit{ϵ}$, where $\mathit{ϵ}$ is a random sample from $N(\mathbf{0},{\sigma }^{2}I)$ with I being an identity matrix. The test data, $({\tilde{X}}_{\mathit{test}},{\tilde{\mathit{Y}}}_{\mathit{test}})$, are generated by the same procedure as $({\tilde{X}}_{\mathit{train}},{\tilde{\mathit{Y}}}_{\mathit{train}})$ but independent of $({\tilde{X}}_{\mathit{train}},{\tilde{\mathit{Y}}}_{\mathit{train}})$. The number of observations for the training and test data are N ($N=50,100,200,300,500,1000$) and 1000, respectively. The lasso and the PCR are performed with training data $({\tilde{X}}_{\mathit{train}},{\tilde{\mathit{Y}}}_{\mathit{train}})$, following which RMSE is calculated in (10). The above process, from random generation of data to RMSE calculation, was performed 100 times.

Parameter setting

Covariance structures

Here, the characterization of the network structure of predictors by conditional independence is considered. When the predictors follow a multivariate normal distribution, the network structure based on the conditional independence corresponds to the nonzero pattern of the inverse covariance (precision) matrix. In other words, the network structure is characterized by the inverse covariance matrix of predictors.

Let S be the sample covariance matrix of predictors, that is, $S={\sum }_{i=1}^n({\mathit{x}}_i-\overline{\mathit{x}}){({\mathit{x}}_i-\overline{\mathit{x}})}^T/n$. Let ${\mathrm{\Omega }}_j={\mathrm{\Sigma }}_j^{-1}$ $(j=1,2,3)$. ${\mathrm{\Sigma }}_1$ is a ridge estimator of the sample variance-covariance matrix, that is, ${\mathrm{\Sigma }}_1=S+\delta I$. Here $\delta$ is a small positive value (in this simulation, $\delta ={10}^{-5}$). The term $\delta I$ allows the existence of ${\mathrm{\Omega }}_1$. Note that because ${\mathrm{\Omega }}_1$ is not sparse, it leads to the complete graph, which is of no use in interpreting gene regulatory networks. To generate a covariance matrix whose inverse matrix is sparse, $L_1$ penalization is employed for the estimation of ${\mathrm{\Omega }}_2$ and ${\mathrm{\Omega }}_3$ as follows:

$$\begin{array}{c}\hfill {\widehat{\mathrm{\Omega }}}_j=\underset{\mathrm{\Omega }}{\text{arg}\text{max}}\left\{log|\mathrm{\Omega }|,-,\text{tr},(\mathrm{\Omega }S),-,P_j,(\mathrm{\Omega })\right\}(j=2,3),\end{array}$$ 2

where $P_j(\mathrm{\Omega })$ $(j=2,3)$ are penalty terms which enhance the sparsity of the inverse covariance matrix. To estimate the sparse inverse covariance matrix, the lasso penalty is typically used as follows:

$$\begin{array}{c}\hfill P_2(\mathrm{\Omega })=\rho \sum _{i=1}^p{\Vert {\mathit{\omega }}_{(-i,·)}\Vert }_1,\end{array}$$ 3

where ${\mathit{\omega }}_{(-i,·)}={({\omega }_{i1},{\omega }_{i2},\dots ,{\omega }_{i(i-1)},{\omega }_{i(i+1)},\dots ,{\omega }_{\mathit{ip}})}^T\in {\mathbb{R}}^{p-1}$. The problem (3) is referred to as the graphical lasso²⁹, and there exists several efficient algorithms to obtain the solution^31–33. The estimator of (2) with (3) corresponds to ${\mathrm{\Omega }}_2$ and ${\mathrm{\Sigma }}_2={\mathrm{\Omega }}_2^{-1}$.

The lasso penalty (3) does not enhance scale-free networks. It penalizes all edges equally so that the estimated graph is likely to be a random graph, that is, the degree distribution becomes a binomial distribution. To enhance scale-free networks (i.e., power-law distribution), the log penalty³⁰ is used as follows:

$$\begin{array}{c}\hfill P_3(\mathrm{\Omega })=\frac{\rho }{2}\sum _{i=1}^p\left\{log,\left(\Vert ,{\mathit{\omega }}_{(-i,·)},{\Vert }_1,+,a_i\right),+,log,\left(\Vert ,{\mathit{\omega }}_{(·,-i)},{\Vert }_1,+,a_i\right)\right\},\end{array}$$ 4

where ${\mathit{\omega }}_{(·,-i)}={({\omega }_{1i},{\omega }_{2i},\dots ,{\omega }_{(i-1)i},{\omega }_{(i+1)i},\dots ,{\omega }_{\mathit{pi}})}^T$ and $a_i>0$ are tuning parameters. We note that the penalty (4) is slightly different from original definition³⁰, expressed as

$$\begin{array}{c}\hfill P(\mathrm{\Omega })=\rho \sum _{i=1}^{p}log\left(\Vert ,{\mathit{\omega }}_{(-i,·)},{\Vert }_1,+,a_i\right).\end{array}$$ 5

When we do not assume that ${\omega }_{\mathit{ij}}={\omega }_{\mathit{ji}}$, the estimate of the inverse covariance matrix with (5) is not symmetric. Since the original graphical lasso algorithm does not assume that ${\omega }_{\mathit{ij}}={\omega }_{\mathit{ji}}$^31,34, we slightly modify the penalty as in Eq. (4). Notably, $P_3(\mathrm{\Omega })$ in (4) coincides with (5) when ${\omega }_{\mathit{ij}}={\omega }_{\mathit{ji}}$. From a Bayesian viewpoint, the prior distribution which corresponds to the log penalty becomes the power–law distribution³⁰; thus, the penalty (4) is likely to estimate the scale-free networks. The estimator of (2) with (4) corresponds to ${\mathrm{\Omega }}_3$.

Because the log-penalty (4) is nonconvex, it is not easy to directly optimize (2). To implement the maximization problem (2), the minorize-maximization (MM) algorithm³⁵ has been constructed³⁰, in which the weighted lasso penalty $P_M^{(t)}(\mathrm{\Omega })$ with current parameter ${\mathrm{\Omega }}_3^{(t)}$ is used:

$$\begin{array}{c}\hfill P_M^{(t)}(\mathrm{\Omega })=\sum _{i=1}^p\sum _{j\ne i}{\rho }_{\mathit{ij}}^{(t)}|{\omega }_{\mathit{ij}}|,\end{array}$$ 6

where ${\rho }_{\mathit{ij}}^{(t)}$ are the weights

$$\begin{array}{c}\hfill {\rho }_{\mathit{ij}}^{(t)}=\frac{1}{2}\left(\frac{\rho }{\Vert {\mathit{\omega }}_{(-i,·)}^{(t)}{\Vert }_1+a_i},+,\frac{\rho }{\Vert {\mathit{\omega }}_{(·,-j)}^{(t)}{\Vert }_1+a_j}\right).\end{array}$$ 7

In general, $\widehat{\mathrm{\Omega }}$ must be symmetric, so that Eq. (7) can be expressed as

$$\begin{array}{c}\hfill {\rho }_{\mathit{ij}}^{(t)}=\frac{1}{2}\left(\frac{\rho }{\Vert {\mathit{\omega }}_{(-i,·)}^{(t)}{\Vert }_1+a_i},+,\frac{\rho }{\Vert {\mathit{\omega }}_{(-j,·)}^{(t)}{\Vert }_1+a_j}\right).\end{array}$$ 8

Because the weighted graphical lasso can be implemented by a standard graphical lasso algorithm, the estimator is obtained as the following algorithm.

1. Set $t=0$. Get ${\mathrm{\Omega }}_3^{(0)}$ via ordinary graphical lasso. Repeat 2 to 4 until convergence.

2. Update weights ${\rho }_{\mathit{ij}}^{(t)}$ using (7).

3. Get ${\mathrm{\Omega }}_3^{(t+1)}$ via the weighted graphical lasso (2) with penalty (6).

4. $t\leftarrow t+1$.

To obtain ${\mathrm{\Sigma }}_2$ and ${\mathrm{\Sigma }}_3$, the tuning parameters $a_i$ $(i=1\cdots ,p)$ and $\rho$ must be determined. Following the experiments³⁰, $a_i=1$ was set for $i=1\cdots ,p$. To select the value of the regularization parameter $\rho$, several candidates were first prepared. In our simulation, the candidates were $\rho =0.3,0.4,0.5,0.6,0.7$. From these, the value of $\rho$ was selected such that the extended Bayesian information criterion (EBIC^36,37)

$$\begin{array}{c}\hfill \text{EBIC}=-n\left\{log|,{\mathrm{\Omega }}_2,|-\text{tr},({\mathrm{\Omega }}_{2}S)\right\}+qlogn+4q\delta logp\end{array}$$ 9

was minimized. Here, q is the number of nonzero parameters of the upper triangular matrix of $\widehat{\mathrm{\Omega }}$, and $\delta \in [0,1)$ is a tuning parameter. As the value of $\delta$ increases, a sparser graph is generated. Note that $\delta =0$ corresponds to the ordinary BIC³⁸. We set $\delta =0.5$ because $\delta =0.5$ is shown to yield good performance in both simulated and real data analyses³⁷. As a result, the EBIC selected $\rho =0.5$.

The upper triangular matrix ${\mathrm{\Omega }}_3$ must be estimated with the reweighted graphical lasso problem. A value of p = 17205 results in $p(p+1)/2\approx 148$ million parameters. As a result, with the machine used in this study (Intel Core Xeon 3 GHz, 128 GB memory), it would take several days to conduct the reweighted graphical lasso approach, even with a small number of iterations such as $T=5$. For this reason, $T=5$ iterations were employed to produce ${\mathrm{\Sigma }}_3$ here. Finally, ${\mathrm{\Sigma }}_2$ and ${\mathrm{\Sigma }}_3$ were scaled such that their signal-to-noise ratio became ${\mathrm{\Sigma }}_1$.

Figure 2 depicts the logarithm of the largest 30 eigenvalues of ${\mathrm{\Sigma }}_j$ ($j=1,2,3$). The first few largest eigenvalues of ${\mathrm{\Sigma }}_3$ are significantly larger than those of ${\mathrm{\Sigma }}_2$, implying that the scale-free networks tend to produce predictors with large correlations.

Figure 2.

Logarithm graph of the largest 30 eigenvalues of ${\mathrm{\Sigma }}_1$ (square), ${\mathrm{\Sigma }}_2$ (circle) and ${\mathrm{\Sigma }}_3$ (triangle). The horizontal axis expresses the index of eigenvalues arranged in descending order.

Regression parameters

The values of $\mathit{\beta }$ and ${\sigma }^2$ are determined as follows. First, 10-fold cross-validation is performed as described below, and the regularization parameter $\lambda$ in (1) is selected. The data (X, Y) are divided into ten datasets, $(X^{(j)},{\mathit{Y}}^{(j)})$ $(j=1,\dots ,10)$, which consist of almost equal sample sizes. Let $X^{(-j)}=(X^{(1)},\dots ,X^{(j-1)},X^{(j+1)},\dots ,X^{(10)})$, and ${\mathit{Y}}^{(-j)}=({\mathit{Y}}^{(1)},\dots ,{\mathit{Y}}^{(j-1)},{\mathit{Y}}^{(j+1)},\dots ,{\mathit{Y}}^{(10)})$ ($j=1,\dots ,10$). For each j ($j=1,\dots ,10$), the training and test data are defined by $(X^{(-j)},{\mathit{Y}}^{(-j)})$ and $(X^{(j)},{\mathit{Y}}^{(j)})$, respectively. Then, the parameter ${\widehat{\mathit{\beta }}}^{(j)}$ ($j=1,\dots ,10$) is found by the lasso:

$$\begin{array}{c}\hfill {\widehat{\mathit{\beta }}}^{(j)}=\underset{\mathit{\beta }}{\text{arg}\text{min}}\left(\Vert ,{\mathit{Y}}^{(-j)},-,X^{(-j)},{\mathit{\beta }\Vert }_2^2,+,\lambda ,{\Vert \mathit{\beta }\Vert }_1\right).\end{array}$$

For each j ($j=1,\dots ,10$), the verification error is calculated as follows:

$$\begin{array}{c}\hfill {\text{CV}}^{(j)}=\frac{1}{\#{\mathit{Y}}^{(j)}}{\Vert {\mathit{Y}}^{(j)}-X^{(j)}{\widehat{\mathit{\beta }}}^{(j)}\Vert }_2^2.\end{array}$$

Then, $\lambda$ is adopted such that it minimizes $\text{CV}=\frac{1}{10}{\sum }_{j=1}^{10}{\text{CV}}^{(j)}$, the mean of ${\text{CV}}^{(j)}$. Following this, the dataset (X, Y) is again randomly divided into two datasets: test data $(X_{\mathit{test}},{\mathit{Y}}_{\mathit{test}})$ and training data $(X_{\mathit{train}},{\mathit{Y}}_{\mathit{train}})$. Lasso estimation (1) is performed using the training data, with $\lambda$ obtained by the above 10-fold cross-validation. Then, $\mathit{\beta }$ is defined as the lasso estimator, resulting in the number of nonzero parameters of $\mathit{\beta }$ being 259. Figure 3 shows the histogram of nonzero parameters of $\mathit{\beta }$. It is seen that the majority of the nonzero coefficients were close to zero; only 15 parameters had absolute values larger than 0.1.

Figure 3.

Histogram of 259 nonzero parameters of $\mathit{\beta }$.

In addition, the root mean squared error (RMSE) is calculated as follows:

$$\begin{array}{c}\hfill \text{RMSE}=\frac{1}{\sqrt{\#{\mathit{Y}}_{\mathit{test}}}}{\Vert {\mathit{Y}}_{\mathit{test}}-X_{\mathit{test}}\widehat{\mathit{\beta }}\Vert }_2,\end{array}$$ 10

and the variance of errors, ${\sigma }^2$, is defined by ${\sigma }^2={(\text{RMSE})}^2$.

Results

The box and whisker plot of the RMSE and the coefficient of determination ($R^2$) are illustrated in Figs. 4 and 5. The horizontal axis is N (the number of observations of training data) and the vertical axis is the RMSE or $R^2$ based on 1000 observations of test data.

Figure 4.

Box and whisker plot of RMSE. The variance-covariance matrix used in the simulations is ${\mathrm{\Sigma }}_1$ in (a, b), ${\mathrm{\Sigma }}_2$ in (c, d), and ${\mathrm{\Sigma }}_3$ in (e, f). The regression model is estimated by the lasso in (a), (c), and (e) and by PCR in (b), (d), and (f).

Figure 5.

Box and whisker plot of $R^2$. The variance-covariance matrix used in the simulations is ${\mathrm{\Sigma }}_1$ in (a, b), ${\mathrm{\Sigma }}_2$ in (c, d), and ${\mathrm{\Sigma }}_3$ in (e, f). The regression model is estimated by the lasso in (a), (c), and (e) and by PCR in (b), (d), and (f).

We compared the performance of the lasso with that of the PCR. When ${\mathrm{\Sigma }}_1$ and ${\mathrm{\Sigma }}_3$ were used, the PCR performed worse than the lasso for small sample sizes. For ${\mathrm{\Sigma }}_2$, the prediction performance with PCR was unsatisfactory even when the sample size N increased. The poor performance of the PCR can be attributed to the predictors associated with small eigenvalues; these predictors affected the prediction performance. Figure 6 depicts a scatter plot of nonzero elements of $\mathit{\beta }$ and the eigenvector for the maximum eigenvalue of ${\mathrm{\Sigma }}_2$. As can be seen, only a significant amount of correlation existed; in fact, the correlation coefficient was only 0.068.

Figure 6.

Scatter plot of $\mathit{\beta }$ and the eigenvector corresponding to the maximum eigenvalue of ${\mathrm{\Sigma }}_2$. The nonzero elements of $\mathit{\beta }$ are not drawn.

The prediction accuracy was compared among the three covariance structures. In all the cases except PCR with ${\mathrm{\Sigma }}_2$, the values of RMSE decreased and $R^2$ increased with the increase in the value of N. Further, $R^2$ was unstable for small sample sizes for all the cases when the lasso was applied. For large sample sizes, the $R^2$ of ${\mathrm{\Sigma }}_1$ was better than that of ${\mathrm{\Sigma }}_2$ and ${\mathrm{\Sigma }}_3$. As described before, ${\mathrm{\Sigma }}_1$ was the sample covariance matrix, while ${\mathrm{\Sigma }}_3$ (and ${\mathrm{\Sigma }}_2$) was estimated using the graphical lasso. As the lasso-type regularization methods shrink parameters toward zero, the correlations among the exploratory variables reduce when the graphical lasso is used. Therefore, ${\mathrm{\Sigma }}_2$ and ${\mathrm{\Sigma }}_3$ resulted in smaller correlations as compared to ${\mathrm{\Sigma }}_1$. Consequently, the $R^2$ may increase with stronger correlations. We compared the RMSE results of ${\mathrm{\Sigma }}_2$ and ${\mathrm{\Sigma }}_3$. With ${\mathrm{\Sigma }}_2$, we found that a sufficiently large number of observations is required to yield a small RMSE with the lasso. Meanwhile, ${\mathrm{\Sigma }}_3$ resulted in a small RMSE with a relatively small number of observations, such as $N=300$.

Code availability

The proposed simulation is implemented in R package simrnet, which is available at https://github.com/keihirose/simrnet. Below is a sample code of the simrnet in R:

When $p=100$, it took less than 12 min to conduct the simulation with 100 replications using the machine employed herein (Intel Core Xeon 3 GHz, 128  GB memory). For high-dimensional data such as p = 17,205, which was used in the simulation presented in this paper, several days were required to complete the simulation task.

Concluding remarks

In a gene regulation network, a gene regulates a small portion of a genome, not all the genes in a genome. This indicates that gene regulation network is expected to be a sparse network rather than a complete graph. Therefore, two covariance matrices indicating sparse networks (${\mathrm{\Sigma }}_2$, ${\mathrm{\Sigma }}_3$) were prepared in addition to a covariance matrix derived from empirical data (${\mathrm{\Sigma }}_1$). Generally, although hundreds of genes contribute to defining a trait, their contributions are not equal. It is frequently observed that genes regulating a trait include a few large-effect genes and many small-effect genes. This property was reflected in the distribution of $\mathit{\beta }$ (Fig. 3). We considered the case where a limited number of regression coefficients significantly contributed to the definition of a trait. The Monte Carlo simulation result indicated that regardless of the network structure, the number of observations should be greater than at least 200 to accurately predict traits from a transcriptome (${\mathrm{\Sigma }}_1$, ${\mathrm{\Sigma }}_3$, Figs. 4 and 5). We also found that the lasso generally provided better accuracy than the PCR. In particular, when the gene regulation network was random (${\mathrm{\Sigma }}_2$), the prediction accuracy of the PCR was poor even if the sample size increased. In conclusion, it is important to sufficiently secure large sample sizes when performing regression analysis of data that exhibits either the random graph and the scale-free network. Additionally, we concluded that the lasso would be preferable to the PCR to ensure a good prediction accuracy.

Conventional theory on the relationship between RMSE and sample size has been developed under the assumption that the sample size exceeds the number of exploratory variables³⁹. However, omics data, which is rapidly being accumulated, results in high dimensional data with strong correlations. Thus, our simulation study considered more complicated settings than the traditional ones. Our simulation, or its extension, may be used in the future to find clues about theoretical aspects that may ultimately lead to the development of a sample size determination technique for omics data.

Other than the scale-free network, the small-world network is another notable property in the networks literature⁴⁰. The definition of the small-word networks is that the shortest path length between two randomly chosen variables is proportional to $logp$; that is, it is considerably small compared with the network size. The small-world networks have been investigated in various fields of research, including the biology^41–43. Some statistical properties of the small-world networks have also been studied^44–46. The investigation of the prediction accuracy in the small-world networks would be interesting but beyond the scope of this research. We would like to take this as a future research topic. The development of methods that provides better prediction accuracy than the lasso in various network structures with small sample sizes would also be an important future research topic.

Author contributions

Y.O. and K.H. created an R package and performed numerical experiments. Y.O. wrote most of this article, and D.K. and A.J.N. significantly revised Introduction and Simulation frameworks. K.H. wrote technical parts. S.K. first proposed to conduct the numerical simulation of high-dimensional regression on plant science.

Funding

This work was partially supported by the Japan Society for the Promotion of Science KAKENHI 19K11862 (KH) and JST CREST Grant number JPMJCR15O2 (AJN).

Competing interests

The authors declare no competing interests.