Technical note: an R package for fitting sparse neural networks with application in animal breeding

Abstract

Neural networks (NNs) have emerged as a new tool for genomic selection (GS) in animal breeding. However, the properties of NN used in GS for the prediction of phenotypic outcomes are not well characterized due to the problem of over-parameterization of NN and difficulties in using whole-genome marker sets as high-dimensional NN input. In this note, we have developed an R package called snnR that finds an optimal sparse structure of a NN by minimizing the square error subject to a penalty on the L₁-norm of the parameters (weights and biases), therefore solving the problem of over-parameterization in NN. We have also tested some models fitted in the snnR package to demonstrate their feasibility and effectiveness to be used in several cases as examples. In comparison of snnR to the R package brnn (the Bayesian regularized single layer NNs), with both using the entries of a genotype matrix or a genomic relationship matrix as inputs, snnR has greatly improved the computational efficiency and the prediction ability for the GS in animal breeding because snnR implements a sparse NN with many hidden layers.

INTRODUCTION

Genomic selection (GS) is an essential process in modern animal breeding for accelerating genetic progress (Meuwissen et al., 2001). The implementation of GS requires large data sets with both genomic and phenotypic information on a reference population so that predictive models can be built. However, the relationships between genomic information or genes and quantitative traits can be extremely complex, so that specific statistical models have been applied in such context (Gianola et al., 2008). Nonetheless, traditional models have drawbacks because they typically ignored those genes that have complicated interactions with each other or higher order nonlinearity in determining their corresponding phenotypes. To take possible nonlinearity into account in prediction, neural networks (NNs) have emerged as a new tool of GS for marker-based genomic predictions of complex traits in animal breeding (Gianola et al., 2011; Okut et al., 2013; Pérez-Rodríguez et al., 2013; Ehret et al., 2015).

Moreover, due to the problem of over-fitting when using whole-genome marker sets as high-dimensional NN input, it is not easy to construct an efficient NN for the prediction of future outcomes of GS (Ehret et al., 2015). Many dimension reduction strategies, such as genome-derived relationships among individuals (VanRaden, 2008) or singular value decomposition of whole-genome marker sets (Pérez-Rodríguez et al., 2013) as NN inputs, have been used to enhance prediction performance and to decrease computational cost (Ehret et al., 2015). It is believed that building an optimal sparse structure of a NN still remains as an effective way to reduce the amount of computational costs (Anders and Korn, 1999; Gripon and Berrou, 2011; Thomaidis et al., 2011). The sparse deep NN with high-dimensional inputs has been shown to be extremely powerful in connection with performing classification tasks and has been widely used in the area of machine learning (Gripon and Berrou, 2011; Scardapane et al., 2017). Examples of software for fitting deep NNs are the “deepnet” R package from Comprehensive R Archive Network (CRAN), the “DNN” function in Tensorflow Python package, and the “Deep Learning” with Matlab. However, very few software packages are available to fit sparse NN either in the public or commercial domains, not to mention for application in GS.

In this note, we have developed an R package called “snnR” that finds an optimal sparse structure of a NN to attenuate the over-fitting problem when a genomic relationship matrix or a SNP marker matrix is used as input to improve the predictive performance in GS applications. The method works by minimizing the square error subject to a penalty on the -norm of the parameters (weights and biases), which was introduced in the context of linear regression by Tibshirani (1996), known as the lasso estimator. To solve the problem of nondifferentiability of the L₁-norm penalty, we use a subgradient-based quasi-Newton method (Bertsekas et al., 2003). It is based on choosing a subgradient with minimum norm as a steepest descent direction and taking a step resembling Newton iteration in this direction with a Hessian approximation (Nocedal, 1980). An active-set method is adopted to set some parameters to exactly zero (Krishnan et al., 2007). The application of snnR is extended, such that additive and dominance effects can be fitted for genome-enabled prediction. snnR also implements a feed-forward NN with many hidden layers to improve the modeling power to predict complex traits more than when single layer architecture is used in NN. The package is available at the CRAN site (http://cran.r-project.org/mirrors.html).

METHODS

Linear, Nonlinear Models, and NNs

Meuwissen et al. (2001) introduced the use of linear regression models in genome-enabled predictions. The basic linear regression model for additive effects is

$$y_i=\mu +\sum _{j=1}^p{x}_{ij}{\beta }_j+{ϵ}_i\text{,}$$ (1)

where y_i is a target trait measured on individual i; µ is an intercept; β_j is the allele substitution effect of marker j; x_ij is the jth marker genotype observed in individual i; and ${ϵ}_i~NIID(0,{\sigma }_e^2)$, where ${\sigma }_e^2$ is the residual variance. A more general regression model suitable for capturing nonlinear patterns can be written as

$$y_i=\mu +g(x_i)+{ϵ}_i$$ (2)

where g(.)is a function that maps from the input space (p-dimensional) to the real line, x_i is the set of genotypic codes observed on i and ò_i is as in Eq. (1). It is well known that any nonlinear function can be exactly represented by a NN (Kurkova, 1992). The Single Hidden-Layer Feed-Forward Neural Networks (SLNN) for GS was introduced by Gianola et al. (2011):

$$y_i=\mu +\sum _{k=1}^S{W}_k{g}_k\left(b_k+\sum _{j=1}^p{x}_{ij}{\beta }_j^{[k]}\right)+{ϵ}_i$$ (3)

In terms of genome-enabled prediction using (3), in the hidden layer, the genomic covariates x_ij (for j = 1, …, p) of an individual i (for i = 1, …, n) are linearly combined with a vector of input weights ${\beta }_j^{[k]}$ that are specified in the training phase, plus an intercept b_k (in NN’s terminology also called “bias”), with k = 1, …, S denoting a neuron. The resulting linear score is then transformed using an activation function g_k(.) to produce the output of the single hidden neuron. To model nonlinear relationships between phenotype and input, the tangent hyperbolic function ($\text{tanh}(x)=(2/[1+\exp (-2x)])-1$) can be used in the hidden neurons. In the output layer, the S genotype-derived basis functions, resulting from the hidden layer, are also linearly combined by using the W₁, W₂, …, W_s weights.

In the context of GS, the NN model given by Eq. (3) can be fitted by using as predictors: (i) the incidence matrix for additive effects of genetic markers $X=\left\{x_{ij}\right\}\in \left\{-1,0,1\right\}$ with dimensions n × p, or (ii) a genomic relationship matrix (e.g., G), as pointed out by Gianola et al. (2011). For example, a genomic relationship matrix suggested by VanRaden (2008) as:

$$G=\frac{X{X}^{\text{'}}}{2{\sum }_{j=1}^p{p}_j(1-p_j)}\text{,}$$ (4)

where X is the incidence matrix for additive effects and p_j is the minor allele frequency for SNP j, j = 1, …, p.

A NN model that includes additive and dominance effects jointly was presented by Pérez-Rodríguez et al. (2013) as follows:

$$\begin{array}{l} y_i=\mu +\sum _{k=1}^{S_a}{W}_k^a{g}_k^a\left(b_k^a+\sum _{j=1}^p{x}_{ij}{\beta }_j^{a[k]}\right)\\ +\sum _{k=1}^{S_d}{W}_k^d{g}_k^d\left(b_k^d+\sum _{j=1}^p{z}_{ij}{\beta }_j^{d[k]}\right)+{ϵ}_i\text{,}\end{array}$$ (5)

where $Z=\left\{z_{ij}\right\}\in \{0,1\}$ is the incidence matrix for dominance effects of dimension n × p, if markers are used, z_ij = 1 if SNP j for individual i is heterozygous, and z_ij = 0 otherwise. Further, S_a and S_d are the numbers of neurons for the additive and dominance components in the hidden layers, respectively.

The NN model given by Eq. (5) can be fitted using as predictors: (i) the incidence matrix for additive effects of genetic markers X and the incidence matrix for dominance effects of markers Z, or (ii) the genomic additive relationship matrix G and the genomic dominance relationship matrix D using the same arguments of VanRaden (2008) as:

$$D=\frac{Z{Z}^{\text{'}}}{2{\sum }_{j=1}^p{p}_{j }{q}_j\left(1-2p_{j }{q}_j\right)}\text{,}$$ (6)

where q_j = 1 − p_j.

Sparse NNs

We obtain a sparse structure of model (3) by minimizing the negative logarithm of likelihood of the data with sparsity enforcing L₁-norm penalty on parameters$\left\{W_k,b_k,{\beta }_j^{[k]}\right\}$ (k = 1, …, S; j = 1, …, p) as follows:

$$\begin{array}{l}\underset{W_k,b_k,{\beta }_j^{\left[k\right]}}{\min }\tilde{F}\left(W_k,b_k,{\beta }_j^{\left[k\right]}\right)\triangleq \hat{\mathcal{L}}\left(W_k,b_k,{\beta }_j^{\left[k\right]}\right)\\ +\left(\sum _{k=1}^S\sum _{j=1}^p{\lambda }_{k,j}\left|{\beta }_j^{\left[k\right]}\right|+\sum _{k=1}^S{\lambda }_k\left|b_k\right|+\sum _{k=1}^S{\lambda }_k\left|W_k\right|\right)\text{,}\end{array}$$ (7)

where the approximate square error

$\hat{\mathcal{L}}\left(W_k,b_k,{\beta }_j^{[k]}\right)=\sum _{i=1}^n{\left(\sum _{k=1}^S{W}_k{g}_k\left(b_k+\sum _{j=1}^p{x}_{ij}{\beta }_j^{[k]}\right)-y_i\right)}^2,$ ${\lambda }_{k,j}({\lambda }_{k,j}>0)$ and ${\lambda }_k({\lambda }_k>0)$ are Lagrange multipliers that determine the amount of sparsity in ${\beta }_j^{[k]}$, W_k and b_k. Note that µ is not included in $\hat{\mathcal{L}}\left(W_k,b_k,{\beta }_j^{[k]}\right)$, as this parameters can be easily eliminated, for example, simply by centering the response vector.

The gradients of $\hat{\mathcal{L}}\left(W_k,b_k,{\beta }_j^{\left[k\right]}\right)$ with respect to $W_k,b_k,{\beta }_j^{\left[k\right]}$are given by

$$\begin{array}{l}{\nabla }_{W_k}\hat{\mathcal{L}}\left(W_k,b_k,{\beta }_j^{\left[k\right]}\right)\\ =2\sum _{i=1}^n\left(\sum _{k=1}^S{W}_k{g}_k\left(b_k+\sum _{j=1}^p{x}_{ij}{\beta }_j^{[k]}\right)-y_i\right)\\ g_k\left(b_k+\sum _{j=1}^p{x}_{ij}{\beta }_j^{[k]}\right)\text{,}\end{array}$$ (8)

$$\begin{array}{l}{\nabla }_{b_k}\hat{\mathcal{L}}\left(W_k,b_k,{\beta }_j^{\left[k\right]}\right)=\\ 2\sum _{i=1}^n\left(\sum _{k=1}^S{W}_k{g}_k\left(b_k+\sum _{j=1}^p{x}_{ij}{\beta }_j^{[k]}\right)-y_i\right)\\ W_k{\dot{g}}_k\left(b_k+\sum _{j=1}^p{x}_{ij}{\beta }_j^{[k]}\right)\text{,}\end{array}$$ (9)

where $\dot{g}$ denotes the derivative of g; and

$$\begin{array}{l}{\nabla }_{{\beta }_j^{\left[k\right]}}\hat{\mathcal{L}}\left(W_k,b_k,{\beta }_j^{\left[k\right]}\right)=\\ 2\sum _{i=1}^n\left(\sum _{k=1}^S{W}_k{g}_k\left(b_k+\sum _{j=1}^p{x}_{ij}{\beta }_j^{[k]}\right)-y_i\right)\\ W_k{\dot{g}}_k\left(b_k+\sum _{j=1}^p{x}_{ij}{\beta }_j^{[k]}\right)x_{ij}\text{.}\end{array}$$ (10)

Let $\begin{array}{l}\mathcal{X}=(W_1,\cdots ,W_S,b_1,\cdots ,b_S,{\beta }_1^{[1]},\cdots ,{\beta }_S^{[1]},\cdots ,{\beta }_1^{[p]},\dots {\beta }_S^{[p]})\text{'}\in {ℛ}^{S+S+S*p}\\ \end{array}$ in model (3) be the vector of weights and biases. Thus, the (7) can be written as

$$\underset{\mathcal{X}}{\min }\tilde{F}\left(\mathcal{X}\right)\triangleq \hat{\mathcal{L}}\left(\mathcal{X}\right)+{\lambda }_r\sum _{r=1}^{S+S+S*p}\left|{\mathcal{X}}_r\right|$$ (11)

where $\begin{array}{l}\mathcal{X}=(W_1,\cdots ,W_S,b_1,\cdots ,b_S,{\beta }_1^{[1]},\cdots ,{\beta }_S^{[1]},\cdots ,{\beta }_1^{[p]},\dots {\beta }_S^{[p]})\text{'}\in {ℛ}^{S+S+S*p}\\ \end{array}$.

Subgradient-based Method to Solve the Sparse NNs

In (11), $\hat{\mathcal{L}}\left(\mathcal{X}\right)$ of $\tilde{F}\left(\mathcal{X}\right)$ is not convex and differentiable with respect to X_r. Solving (11) is complicated by the nondifferentiability of $\left|{\mathcal{X}}_r\right|$ at X_r = 0. Subgradient methods are among the most popular method for nondifferentiable optimization (Bertsekas et al., 2003). We use the subgradient with minimum norm (Bertsekas et al., 2003) of $\tilde{F}\left(\mathcal{X}\right)$ in (11) as the steepest descent direction and take a step resembling a Newton iteration in this direction with a Hessian approximation (Gill et al., 1984) to solve the above problem.

The subgradient of $\tilde{F}\left(\mathcal{X}\right)$ in (11) with respect to a variable X_r is given by

$$\partial {}_r\tilde{F}\left(\mathcal{X}\right)={\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right)+{\lambda }_{r}sgn({\mathcal{X}}_r)\text{,}$$ (12)

where $\partial {}_r\tilde{F}\left(\mathcal{X}\right)=\partial {}_{{\mathcal{X}}_r}\tilde{F}\left(\mathcal{X}\right)$, ${\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right)={\nabla }_{{\mathcal{X}}_r}\hat{\mathcal{L}}\left(\mathcal{X}\right)$, and the set-valued function sgn(X_r) (Bertsekas et al., 2003) is given by

$$sgn({\mathcal{X}}_r)\triangleq \{\begin{array}{c}sign\left({\mathcal{X}}_r\right),{\mathcal{X}}_r\ne 0\\ \left[-1,1\right],{\mathcal{X}}_r=0\end{array}\text{.}$$ (13)

Because the subgradient of $\tilde{F}\left(\mathcal{X}\right)$ is separable in the variables, the problems of computing the minimum-norm element of the subgradient is also separable. Hence, we can solve the minimum-norm problem coordinate-wise to yield that the element of the minimum-norm subgradient with respect to a variable X_r is:

$$\begin{array}{l} {\tilde{\nabla }}_r\tilde{F}\left(\mathcal{X}\right)\\ \triangleq \{\begin{array}{c}{\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right), \lambda =0\\ {\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right)+{\lambda }_{r}sign\left({\mathcal{X}}_r\right), \lambda >0, \left|{\mathcal{X}}_r\right|>0 \\ {\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right)+{\lambda }_r, \lambda >0, {\mathcal{X}}_r=0, {\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right)<- {\lambda }_r \\ {\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right)-{\lambda }_r, \lambda >0, {\mathcal{X}}_r=0, {\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right)> {\lambda }_r \\ 0, \lambda >0, {\mathcal{X}}_r=0,| {\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right)|\le {\lambda }_r \end{array} \end{array}$$ (14)

where ${\tilde{\nabla }}_r\tilde{F}\left(\mathcal{X}\right)={\tilde{\nabla }}_{{\mathcal{X}}_r}\tilde{F}\left(\mathcal{X}\right)$ and $\lambda =\{{\lambda }_1,{\lambda }_2,\cdots ,{\lambda }_{N_p}\}$ (N_p is the number of parameters in X). In the last case of (14), the element of the minimum-norm subgradient is zero, because we can set sign(X_r) to $-{\nabla }_r\hat{\mathcal{L}}\left(\mathcal{X}\right)/{\lambda }_r$ to achieve a norm of zero.

A search direction to solve the sparse NNs is as follows

$$h\left[\kappa \right]\triangleq -{\mathcal{H}}^{-1}\left[\kappa \right]\tilde{\nabla } \tilde{F}\left(\mathcal{X}\left[\kappa \right]\right)$$ (15)

In dealing with practical problems, quasi-Newton methods (Dennis et al., 1997) allow us to replace the Hessian matrix H[k] by an approximation B[k]. So, the search direction is changed as follows:

$$h\left[\kappa \right]\triangleq -{\mathcal{B}}^{-1}\left[\kappa \right]\tilde{\nabla } \tilde{F}\left(\mathcal{X}\left[\kappa \right]\right)\text{,}$$ (16)

Details on the calculation of the search direction are given in Supplementary A.

Active-set Method to Set Some Parameters to Zero

To set variables X[k] to exactly zero, we use one of the active-set methods which are widely used for solving L₁-norm regulation problems (Krishnan et al., 2007). The variables are split into two sets: the working set W containing the sufficiently nonzero variables, and the active-set A containing the sufficiently zero-values variables.

The working set is defined as follows:

$$\mathcal{W}\triangleq \left\{r \left|{\mathcal{X}}_r\right|>E\right\}\text{\hspace{0.17em}.}$$ (17)

On each iteration, we take a projection of the Newton step along the working set using the Hessian matrix approximation B⁻¹[k] and a projected minimum norm subgradient for the active-set variables:

$${\mathcal{X}}_{\mathcal{W}}\left[\kappa +1\right]\leftarrow {\mathcal{P}}_{\mathcal{O}}\left({\mathcal{X}}_{\mathcal{W}}\left[\kappa \right]-\iota {\mathcal{B}}_{\mathcal{W}}^{-1}\left[\kappa \right]\tilde{\nabla }{ }_{\mathcal{W}}\tilde{F}\left(\mathcal{X}\left[\kappa \right]\right)\right)\text{,}$$

$${\mathcal{X}}_{\mathcal{A}}\left[\kappa +1\right]\leftarrow {\mathcal{P}}_{\mathcal{O}}\left({\mathcal{X}}_{\mathcal{A}}\left[\kappa \right]-\iota \mathcal{D}\left[\kappa \right]{\tilde{\nabla }}_{\mathcal{A}} \tilde{F}\left(\mathcal{X}\left[\kappa \right]\right)\right)\text{,}$$ (18)

where we use ${\tilde{\nabla }}_{\mathcal{A}} \tilde{F}\left(\mathcal{X}\left[\kappa \right]\right)$ to denote the subvector of $\tilde{\nabla } \tilde{F}\left(\mathcal{X}\left[\kappa \right]\right)$ corresponding to elements of A and ${\mathcal{B}}_{\mathcal{W}}^{-1}\left[\kappa \right]$ to denote the submatrix of B⁻¹[k] with all rows and columns of W; P_o is an orthant projection, which is effective at sparsifying the parameter vector and ensures that the line search does not cross points of nondifferentiability, and is defined as:

$${\mathcal{P}}_{\mathcal{O}}\left(\mathcal{X}+h\right)\triangleq \{\begin{array}{c} 0 if {\mathcal{X}}_r\left({\mathcal{X}}_r+h_r\right)<0 \\ {\mathcal{X}}_r+h_r \text{otherwise} \end{array}\text{,}$$ (19)

and D[k] is the diagonal scaling matrix and set to σ⁻¹[к]I with the Barzilai–Borwein scaling ${\sigma }^{-1}[\kappa ](\sigma [\kappa ]\triangleq ({\mathcal{I}}^{\text{'}}[\kappa ]\mathcal{I}[\kappa ]/{\mathcal{I}}^{\text{'}}[\kappa ]s[\kappa ]))$ (Fletcher, 2001).

We outline an algorithm that can be used as a basis for implementing a subgradient-based quasi-Newton method for solving the problem (7) (Supplementary B).

EXAMPLES

Jersey Data Set

The first data set, described in detail in Gianola et al. (2011) and made publically available by Pérez-Rodríguez et al. (2013) in the brnn package, consists of milk production records for 3 traits (fat yield, milk yield, and protein yield) in Jersey cows. The incidence matrices with marker codes for additive and dominance effects (X and Z, respectively) were derived from 33,262 SNP on each of 297 individually genotyped cows. The necessary data are stored as R objects that can be accessed once the library snnR is loaded. The phenotypic information is stored in the data frame object pheno, and the additive genomic relationship matrix and the dominance genomic relationship matrix are stored in objects G and D, respectively. The data set also includes a vector (partitions) that assigns observations to 10 disjoint sets; this vector could be used to perform a 10-fold cross validation. We can fit Eq. (3) using the entries of the G matrix as inputs, and fat yield, milk yield or protein yield as response variable using the function snnR. A NN model that includes additive and dominance effects jointly (5) was fitted using the G and D matrices of the Jersey data. The model can be fitted using the function snnR_extended. Details of the commands are in the manual of snnR.

Simulated Data Set

The second data set consists of a simulated data including 10,000 F2 individuals from 2 founder lines with information on 1,000 biallelic SNP loci evenly distributed over 10 chromosomes of length 1M each. The mutation rate for both SNP markers and QTL was set to 2.5 * 10⁻⁵. The cumulative effect of the simulated QTL equaled the genetic value of the individuals, while the phenotypes of the individuals were obtained as the sum of the individual’s genetic value and a random draw from a normal distribution with mean zero and a variance set to produce different heritabilities: 0.1, 0.3, 0.5, and 0.7. The data also includes a file that assigned observations to 10 disjoint sets. We have also provided a simulated data including 10,000 F2 individuals from 2 founder lines with 20,000 biallelic SNP loci at the heritability 0.7 for testing the predictive performance of our snnR package for big samples and big input dimension. The data are available with a quick-start guide at: http://mgb.ouc.edu.cn/novegene/html/sdnn.php.

RESULTS

Evaluation of the Predictive Power

Jersey data set.

We have compared the predictive performance of snnR, brnn (Pérez-Rodríguez et al., 2013), and RR-GBLUP (Meuwissen et al., 2001) with the 2 data sets. Table 1 and Supplementary Table S1 and S2 present the results of evaluation of the predictive power of the models including additive and dominance effects with a 10-fold cross-validation with the genomic relationship G matrix and D matrix in Jersey Data set as inputs for milk yield, fat yield, and protein yield. For different values of λ, we trained a model on a training set and selected the optimal value of λ that gives the smallest prediction error on a validation set. The correlation between observed and predicted values for each of the folds was obtained as the average of 10 runs in the case of snnR software to mitigate the effect of the starting values for all parameters in the net. The last 2 rows of the tables show the average correlation and the average root mean square error (RMSE).

Table 1.

Correlations between observed and predicted values of milk yield in a testing set of Jersey cows^a

Fold			Additive			Additive + dominance
	RR-GBLUP	brnn1–2–1	snnR1–2–1	snnR1–5–5–5–1	brnn1–2:2–1	snnR1–2:2–1	snnR1–5–5–5:5–5–5–1
1	0.5328	0.3455	0.4269	0.4459	0.2715	0.4189	0.4707
2	0.5894	0.5211	0.5862	0.6163	0.5546	0.6371	0.6486
3	0.4782	0.2553	0.4125	0.4808	0.4264	0.4063	0.5215
4	0.6965	0.5241	0.6594	0.6861	0.5051	0.6217	0.6763
5	0.2887	0.1118	0.2716	0.3115	0.0989	0.2984	0.2817
6	0.4880	0.3821	0.4336	0.4513	0.4211	0.4335	0.4878
7	-0.0668	-0.1365	-0.0622	-0.0464	-0.0756	-0.0132	-0.0115
8	0.7364	0.4533	0.6379	0.6886	0.5387	0.6578	0.7761
9	0.1415	0.1361	0.1322	0.1917	0.1974	0.1446	0.1897
10	0.5231	0.4334	0.5034	0.5451	0.4920	0.5328	0.5774
Avg. COR	0.4407	0.3026	0.4002	0.4372	0.3430	0.4138	0.4619
Avg.RMSE	50.73	68.79	54.28	52.17	67.93	53.84	48.89

^abrnn 1–2–1 = Bayesian regularized neural network with two neurons in 1-hidden layer for the additive component; snnR 1–2–1 = Sparse neural network with 1-hidden layer and two neurons in the hidden layer for the additive component; snnR 1–5–5–5–1 = Sparse neural network with 3-hidden layers and five neurons in each hidden layer for the additive component; brnn 1–2:2–1 =Bayesian regularized neural network with two neurons for each of the additive and dominance components; snnR 1–2:2–1 = Sparse neural network with 1-hidden layer and two neurons in the hidden layer for each of the additive and dominance components; snnR 1–5–5–5:5–5–5–1 = Sparse neural network with 3-hidden layers and five neurons in each hidden layer for each of the additive and dominance components; COR = correlation; RMSE = root mean square error.

For the Jersey Data set with only 297 samples, predictive ability of RR-GBLUP showed among the greatest in the case of predictions for the fat yield and the protein yield. However, for the remainder trait (milk yield), snnR with 3-hidden layers and five neurons in each hidden layer for each of the additive and dominance components produced the most accurate predictions, about 2% greater than RR-GBLUP. Overall with this data set, RR-GBLUP was found to be an efficient prediction method. Wimmer et al. (2013) compared the performance of four commonly used methods such as RR-GBLUP and BayesB (Meuwissen et al., 2001), LASSO (Tibshirani, 1996), and the elastic net (Zou and Hastie, 2005). They also found a good performance of RR-GBLUP due to long-range LD, medium heritabilities, and small sample sizes.

For the same NN methods, the predictions obtained by using snnR were better than those obtained by brnn with the same network architectures (Table 1 and Supplementary Table S1 and S2). Our data show that the deep-structured NN have an advantage for predicting complex traits over shallow-structured single layer NN. Furthermore, on average, inclusion of the dominance component yielded slightly better predictions for milk yield (Table 1), even though milk yield is believed to be controlled mostly by additive genetic effects. This hypothesis is also supported by results of nonadditive effects on milk yield in Jersey cows, which reported that individual nonadditive effects made a small contribution to the genetic variation of milk yield (Aliloo et al., 2015).

A great variability in the results was observed on the folds 7 and 9 with null or negative correlations for fat yield, milk yield, and protein yield using all three methods (Table 1 and Supplementary Table S1 and S2). A possible explanation for such variability could be overfitting caused by sampling in the small sample size of Jersey.

Simulated data set.

Table 2 shows the results of evaluation of the predictive power of the models with additive effects for the Simulated Data set. RR-GBLUP achieved predictive abilities of 0.3417, 0.6068, 0.6791, and 0.8695 for the traits with heritabilities 0.1, 0.3, 0.5, and 0.7, respectively. For all traits, snnR and brnn reached predictive abilities between 7.91% and 19.37% and between 7.63% and 9.12% greater than RR-GBLUP, respectively. One possible explanation for the worse performance of RR-GBLUP could be the fact that RR-GBLUP was fitting with an inverse of G matrix (Meuwissen et al., 2001), while the dimension of the inverse of G matrix was too big (close to 10,000 in this example) to calculate correctly. For the trait with high heritability 0.7, snnR and brnn reached also the same good predictive abilities. With the heritability decreasing, the predictive ability of snnR obviously outperformed brnn and deep-structured snnR had an advantage for predicting complex traits over shallow-structured snnR. For an example, snnR achieved a predictive ability of about 0.8872 when using the simulated data set with 10,000 samples and 20,000 SNP genotype inputs.

Table 2.

Correlations between observed and predicted values for Simulated Data set for traits with different heritabilities: 0.1, 0.3, 0.5, and 0.7^a

Fold	Additive h2 = 0.1				Additive h2 = 0.5
	RR-GBLUP	brnn1–2–1	snnR1–2–1	snnR1–5–5–5–1	RR-GBLUP	brnn1–2–1	snnR1–2–1	snnR1–5–5–5–1
1	0.3068	0.4109	0.4869	0.4993	0.6812	0.7614	0.8145	0.8477
2	0.3549	0.4052	0.4535	0.4606	0.6489	0.7774	0.8368	0.8668
3	0.3859	0.4277	0.4771	0.5166	0.7007	0.7754	0.8248	0.8716
4	0.3126	0.3736	0.4330	0.4985	0.6924	0.7601	0.8577	0.8896
5	0.3306	0.4680	0.4278	0.4931	0.6976	0.7780	0.8529	0.8831
6	0.3056	0.4729	0.4403	0.4976	0.7082	0.7542	0.8736	0.8969
7	0.3458	0.3864	0.4265	0.4501	0.6501	0.7687	0.8401	0.8531
8	0.3885	0.4131	0.4971	0.4432	0.6807	0.7719	0.8561	0.8724
9	0.3786	0.4401	0.4843	0.5516	0.6871	0.7696	0.8508	0.8774
10	0.3078	0.3961	0.4470	0.4905	0.6448	0.7863	0.8446	0.8691
Avg COR	0.3417	0.4194	0.4573	0.4912	0.6791	0.7703	0.8451	0.8728
Avg.RMSE	10.22	9.74	9.12	8.97	5.7125	3.29	1.71	1.52
Fold	Additive h2 = 0.3				Additive h2 = 0.7
	RR-GBLUP	brnn1–2–1	snnR1–2–1	snnR1–5–5–5–1	RR-GBLUP	brnn1–2–1	snnR1–2–1	snnR1–5–5–5–1
1	0.6112	0.6745	0.7665	0.7651	0.8742	0.9484	0.9483	0.9449
2	0.6025	0.6516	0.7201	0.7371	0.8528	0.9388	0.9412	0.9516
3	0.6050	0.6762	0.7331	0.7611	0.8813	0.9499	0.9527	0.9445
4	0.6292	0.7105	0.7285	0.7969	0.8817	0.9428	0.9516	0.9531
5	0.6238	0.6853	0.7573	0.7221	0.8750	0.9454	0.9477	0.9465
6	0.6204	0.7199	0.7853	0.7583	0.8909	0.9439	0.9457	0.9460
7	0.5755	0.6772	0.7125	0.7997	0.8382	0.9454	0.9486	0.9376
8	0.6086	0.6504	0.7901	0.7446	0.8948	0.9525	0.9530	0.9642
9	0.6084	0.6664	0.7695	0.7751	0.8798	0.9502	0.9534	0.9585
10	0.5838	0.7197	0.7357	0.7419	0.8263	0.9427	0.9431	0.9428
Avg COR	0.6068	0.6831	0.7498	0.7601	0.8695	0.9460	0.9485	0.9489
Avg.RMSE	7.39	5.34	4.16	3.57	1.97	0.95	0.98	0.98

^abrnn 1–2–1 = Bayesian regularized neural network with two neurons in 1-hidden layer for the additive component; snnR 1–2–1 = Sparse neural network with 1-hidden layer and two neurons in the hidden layer for the additive component; snnR 1–5–5–5–1 = Sparse neural network with 3-hidden layers and five neurons in each hidden layer for the additive component; COR = correlation; RMSE = root mean square error.

Evaluation of the Computation Time

Both the NN methods were implemented in R and ran on a Linux machine with an Intel Core i5-3230M CPU processor @2.60GHz with 4 GB of RAM memory. Figure 1 presents the results of RMSE corresponding to the training computation time (in seconds) for brnn and snnR with different network architectures (2 to 15 neurons in 1-hidden layer) for genome-enabled predictions of milk traits in Jersey cows using the G matrix as input. We noticed that (1) RMSE of our snnR rapidly decreased as the network was trained with a relatively simple NN architecture (2–5 neurons in the hidden layer, Figure 1a–d) and even with complex NN architectures (over five neurons in the hidden layer, Figure 1e and f); (2) however, RMSE of brnn decreased very slowly with the increased number of neurons in the hidden layer of NN (Figure 1a–f). For each simulated data set at the different heritabilities in Table 2, the brnn software took over 12 h to fit the model, while our snnR completed it within 30 min when both packages use the same simple NN architecture with two neurons in a single hidden layer.

Figure 1.

Results of RMSE corresponding to the training computation time (in seconds) for brnn and snnR with different network architectures for genome-enabled predictions of milk traits in Jersey cows using the G matrix as input. (a)–(f) structures with brnn 1–n–1 = Bayesian regularized neural network with n from 2 to 15 neurons in 1-hidden layer; snnR 1–n–1 = Sparse neural network with 1-hidden layer and n from 2 to 15 neurons in the hidden layer. RMSE = root mean square error.

Optimal Sparse Structure

Consider the nonlinear function depicted in Figure 2a. The predicted results of brnn with 1-hidden-layer architecture, snnR with 5-hidden-layer architecture, and snnR with 7-hidden-layer architecture (each layer have five neurons) are shown in Figure 2b, c, and e, respectively. The optimal sparse structures of snnR with 5-hidden-layer architecture and 7-hidden-layer architecture, which were obtained by passing the parameters $wDNNs of snnR into the function plotnet of NeuralNetTools (Beck, 2015), are showed in Figure 2d and f. The results demonstrated that deep-structured sparse NN have obvious greater accuracy in nonlinear function regression than shallow-structured single layer NN of brnn. Details of the commands are in the manual of snnR.

Figure 2.

Nonlinear regression. (a) a nonlinear function, (b) the predicted results of brnn with 1-hidden-layer architecture five neurons in the hidden layer, (c) the predicted results of snnR with 5-hidden-layer and five neurons in each hidden layer, (d) the optimal sparse structure of snnR with 5-hidden-layer, (e) the predicted results of snnR with 7-hidden-layer and five neurons in each hidden layer, (f) the optimal sparse structure of a snnR with 7-hidden-layer.

Building an appropriate and compact structure for a network, which involves making an optimal trade-off between the training data and model complexity, is a difficult task, due to the huge search space of possible models. Akaike Information Criteria (AIC; $\text{AIC}=-2/T\text{ln(}\hat{\mathcal{L}}\text{(}\mathcal{X}\text{))}+2K/T$, where T denotes the number of observations and K denotes the number of parameters; Akaike, 1973) and Bayesian Information Criteria (BIC; $\text{BIC}=-2/T\text{ln(}\hat{\mathcal{L}}\text{(}\mathcal{X}\text{))}+2K\text{ln}(T)/T$; Schwarz, 1978) for NNs are still widely used to find an optimal trade-off between an unbiased approximation of the underlying model and the loss of accuracy caused by estimating an increasing number of parameters (Anders and Korn, 1999). Both AIC and BIC are effective in testing these models with the smaller their values indicating the higher effective structure of a network.

Figure 3 presents the combined results of AIC and BIC for all prediction scenarios tested. The single panels (a–h) show the dependency of the average AIC and BIC results of cross-validation runs on the optimal sparse NN models with different initialized fully connected NN architecture (1 to 8 hidden layers and 2 to 10 neurons in each hidden layer) for milk yield as response and G used as input for snnR. Consistency of the changes of AIC and BIC in responses to neuron numbers in all the single panels (Figure 3) indicates that the sparse NN models are suitable to build an appropriate and compact structure for a network using AIC or BIC. The smaller values of AIC and BIC are found in the 3-hidden-layer and 4-hidden-layer (Figure 3c and d), implying that some deep-structured NN models have an advantage for predicting milk yield over shallow-structured single layer NN models. The big values of AIC or BIC are seen in the NN with over nine neurons and over seven of hidden layers (Figure 3g and h), suggesting that the NN will learn irrelevant details of the data due to overfitting.

Figure 3.

AIC and BIC evaluated using initialized NN with different hidden layers and different neurons in each layer for predicting values of milk yield in a testing set of Jersey cows. Panels (a–h) show the dependency of the average AIC and BIC results of cross-validation runs on the optimal sparse NN models with different initialized fully connected NN architecture(1 to 8 hidden layers and 2 to 10 neurons in each hidden layer) for milk yield as response and G used as input to our snnR.

Therefore, our strategy for setting the number of hidden layers and the number of hidden neurons in each layer is as follows: Step 1, add an additional hidden neuron, obtain a new optimal sparse structure, and the new value of either AIC or BIC is calculated. The hidden unit is accepted when the AIC or BIC criteria show an improvement with their decreasing values. Step 2, when this is the case an enlarged network is estimated, and new residuals and information criteria are computed. The procedure stops when an additional hidden neuron does not lead to further improvements based on the new information criteria AIC or BIC. Step 3, add a new hidden layer and repeat Step 1 and Step 2 until it does not lead to further improvements, then the appropriate and compact structure for a network has been obtained.

Concluding Remarks

We have developed an open source software snnR that allows fitting sparse NN in R and that works in Windows and UNIX-like environments. The main features of the package are: (1) It implements sparse NN to improve the prediction performance with input variables in high-dimensional space. Our method has resulted in lower prediction errors than the method of Bayesian regularized single layer NNs for prediction of complex traits, especially with the bigger sample sizes. (2) It finds the optimal sparse network structure by minimizing the square error subject to a penalty on the L₁-norm of the parameters of a NN to aggregate the high generalization capability and automatic parameter selection. (3) It raises computational efficiency due to using a smaller number of parameters in a sparse structure than in the brnn, especially dealing with input variables in high-dimensional space and the bigger sample sizes. (4) It extends the deep architecture feed-forward NNs with many hidden layers to improve the modeling power to predict complex traits than when single layer architecture is used in NN. From Table 1 and Figure 2, we know deep-structured NN have better performance for predicting complex traits over shallow-structured single layer NN. As such, this software seems to have potential use in genomic selection and animal breeding.

A main problem of snnR is probably computing time. Considering that computing the minimum-norm element of the subgradient is separable, our algorithm can be parallelized, hence saving save computation time. In addition, we will continue working to make our software even more efficient by parallelizing over many machines with an asynchronous mode or over multiple graphics processing units (Lecun et al., 2015). For the optimization problem of sparse NN learning, it is obvious that better parameter initialization techniques will lead to better prediction and computational efficiency. Thus, the random initialization for the parameters of a NN in this software has plenty of room for improvement. The unsupervised pretraining technique which has attracted a wide attention in the parameter initialization may be a good alternative (Lecun et al., 2015). Moreover, we will also implement the group spare regularization (Scardapane et al., 2017) in our package snnR in the near future.

SUPPLEMENTARY DATA

Supplementary data are available at Journal of Animal Science online.