Comparing artificial and convolutional neural networks with traditional models for Genomic prediction in wheat

Abstract

With the rapid development of sequencing technology, the application of genomic prediction has become more and more common in breeding schemes of livestocks and crops. Selecting an appropriate statistical model is of central importance to achieve high prediction accuracy. Recently, machine learning models have been expected to upgrade genomic prediction into a new era. However, the perspective still suffers from lack of evidence that machine learning models can generally outperform the traditional ones on empirical data sets. In this study, we compared two machine learning models based on artificial neural network (ANN) and convolutional neural network (CNN) with four traditional models, including genomic best linear unbiased prediction (GBLUP), Bayesian ridge regression (BRR), BayesA and BayesB, using three published data sets for grain yield in wheat. For each model, we considered two variants: modeling and ignoring the genotype-by-environment () interaction. In the comparison, we considered two strategies of cross-validation: predicting genotypes that have not been evaluated in any environment (CV1) and predicting genotypes that have been tested in other environments (CV2). Our results showed that traditional Bayesian models (BayesA, BayesB, and BRR) outperformed GBLUP, ANN and CNN when considering interaction. The accuracies of ANN and CNN were higher than traditional models only in CV1 and when interaction was ignored. It was also found that the performance of the two machine learning models was significantly affected by the interaction between the CV strategy and the way of treating the interaction, while that of the four traditional models was only influenced by whether the interaction was considered or not. Thus, machine learning models can be a powerful complementary to the traditional ones and their superiority may depend on the prediction scenario. Among the two machine learning models, we observed that the accuracy of ANN was higher than CNN in most cases, indicating that it is still challenging to adapt complex machine learning models such as CNN to genomic prediction.

Introduction

Genomic prediction (GP) was introduced by Theo Meuwissen in 2001 (Meuwissen et al. 2001) and has been extensively applied in animal and plant breeding. It uses high-density molecular markers covering the whole genome to predict the genomic estimated breeding value (GEBV) of individuals. Based on the GEBV, breeders can select superior genotypes without observing their phenotypic performance, thereby shortening the generation interval (Ghosh et al. 2018), accelerating the breeding process (Watson et al. 2019), saving costs (Bernardo 2008), and promoting the development of precision and efficiency in modern breeding. Therefore, it is crucial to build reliable prediction models in order to obtain accurate GEBVs. Sine the choice of model could greatly affect the accuracy and efficiency of genomic prediction, it has been an active research area to compare the prediction accuracies of different models with different data sets.

For over a decade, the genomic best linear unbiased prediction (GBLUP) model (VanRaden 2008) has been the most popular approach applied in genomic prediction. It predicts the phenotypic performance of the individuals in the test set, in which only the genomic data is available, by building a genomic relationship matrix from molecular markers. This relationship matrix serves as a bridge between the individuals in the test and those in the training set, in which both genomic and phenotypic data are available. The GBLUP model is statistically equivalent to the ridge regression best linear unbiased prediction (RRBLUP) model (Meuwissen et al. 2001), which is in turn equivalent to the Bayesian ridge regression (BRR) model (de Los Campos et al. 2013). In the BRR model, the effect of each marker is explicitly modeled as a random variable and the variances of all marker effects are assumed to be equal. The estimation of these marker effects is done within the training set, and the estimated marker effects are then used to predict the individuals in the test set. From biological point of view, the contributions of different markers across the genome to the total genetic variance are not necessarily the same. Motivated by this, Meuwissen et al. modified the assumptions and proposed two alternative models: BayesA and BayesB (Meuwissen et al. 2001). In both models, each marker effect has its own variance. But in BayesB, there is an additional assumption that a proportion of markers has zero effects, which may better fit the reality that many genetic markers do not contribute to the trait at all. Meuwissen’s early study compared the accuracy of least squares, RRBLUP, BayesA, and BayesB, which showed that BayesB had the highest accuracy and the accuracy of the two Bayesian models was higher than that of RRBLUP (Meuwissen et al. 2001). Clark et al. compared the accuracy of BayesB and GBLUP using simulated data sets, and found that BayesB was more accurate than GBLUP in the case that most markers contributing to the trait have rare alleles (Clark et al. 2011). On the other hand, it was found that the computational load of BayesB was higher than RRBLUP and BayesA (Verbyla et al. 2009). In this study, these models are called traditional models as they are among the earliest models for genomic prediction and have been extensively studied by the research community.

All traditional models introduced above are parametric models, and more precisely, linear models. Thus, the models assume that each marker contributes to the trait separately and the final trait value is a linear combination of all marker effects. However, biological evidences suggest that markers may interact with each other. To take this into account, semi-parametric and machine learning models were introduced to genomic prediction and have gained popularity. Due to the successful application in many research areas (Egmont-Petersen et al. 2002; LeCun et al. 2015), the artificial neural network (ANN) and the convolutional neural network (CNN) were expected to be promising in genomic prediction, and there has been a few studies comparing them with the traditional models. McDowell compared some traditional models with several models based on ANN by using data sets in Arabidopsis, maize, and wheat, and found that, among the six traits (two for each species), the ANN-based models achieved the highest accuracy in three traits, and the traditional models were more accurate for the other three traits (McDowell 2016). Rachmatia et al. found that the deep belief network model (a type of ANN) outperformed GBLUP, for traits with a non-additive genetic architecture in maize (Rachmatia et al. 2017). Ma et al. compared CNN with two traditional models (RR-BLUP and GBLUP) and three ANN-based models for eight traits in wheat, and found that the accuracies of CNN, RR-BLUP, and GBLUP were comparable with each other and were higher than the three ANN-based models (Ma et al. 2018. In Liu et al. (2019), it was shown that CNN was superior to traditional models (RRBLUP, BRR, and BayesA) in predicting five traits in soybean.

As reviewed in Montesinos-López et al. (2021), there has been no strong evidence that machine learning models have better predictive power than traditional models generally across data sets. Thus, to gain more insights into the prospect of applying machine learning models in genomic prediction, we compared two machine learning models (ANN and CNN) with four traditional models (GBLUP, BRR, BayesA and BayesB) using three published wheat data sets in this study. Since all data sets were generated with multiple environments, and the environment may impact genetic traits through genotype-by-environment () interaction (DeLacy et al. 1996; Heslot et al. 2014), we evaluated two situations for all models: considering and ignoring the interaction effects.

Materials and methods

Data Set

The three wheat datasets used in this article were developed by CIMMYT’s global wheat program and obtained from its wheat breeding station at Cd. Obregon, Sonora, Mexico. The environments were three irrigation regimes (moderate drought stress, optimal irrigation, and high drought stress), two planting systems (bed and flat planting), and two planting dates (normal and late). All phenotype and genotype data can be obtained in Montesinos-López et al. (2018). Two datasets contain four environments (, , , ), and one dataset contains five environments (, ). The number of wheat lines evaluated in the three data sets are 693, 670 and 807, respectively. The phenotypic traits of all three datasets were adjusted based on experimental design for grain yield (GY, tons/hectare). The lines were genotyped using genotyping-by-sequencing technology (GBS) (Poland et al. 2012). For all data sets, 15, 744 single nucleotide polymorphism (SNP) markers were used in this study.

Genomic best linear unbiased prediction (GBLUP)

The GBLUP model considering interaction is as follows:

This model was proposed by Montesinos-López et al. (2018). Here is the response variable of the ith line in the jth environment (, ). The is the fixed effects of the jth environment, and is the random genetic effect of the ith line, with , where is the genotypic variance and is the genomic relationship matrix (GRM). Each is the random interaction term between the genomic effect of the ith line and the jth environment, and , where is the interaction variance. The symbol represents a random residual associated with the ith line in the jth environment distributed as , where is the residual variance. When interaction is not considered, the term is omitted.

Bayesian ridge regression (BRR), BayesA, and BayesB

The BRR, BayesA, and BayesB models are extensions of the standard normal linear regression models based on the Bayesian theory framework. Different Bayesian models can be obtained by assuming different prior distributions of parameters. For the BRR model (cf. Montesinos-López et al. (2022)), marker effects are assumed to be independent and have the same prior distributions. More precisely, let be the effects of p markers, then , where is the prior variance of marker effects. The random error term follows a prior distribution . Both variance components, and , are assigned inverse Chi-square distributions: , . It can be shown that the posterior mean of marker effects is the best linear unbiased predictor (BLUP) of marker effects, so Bayes ridge regression is also referred to as RRBLUP (de Los Campos et al. 2013).

The BayesA model (cf. Montesinos-López et al. (2022)) slightly modifies the BRR model by assuming different prior variances for the marker effects. Specifically, the marker effects are distributed as , where . The BayesB model can be considered a direct extension of the BRR and BayesA models by adding a parameter that represents the prior proportion of markers with null effects. Therefore, for the BayesB model, the prior distribution of marker effects is as follows (Pérez-Rodríguez et al. 2012):

The Gibbs sampler is then used to estimate the posterior distribution of the parameters. All models were implemented by the BGLR package (v 1.1.0, (de los Campos et al. 2015)) in R (v 4.3.1) with default parameters.

Artificial neural network (ANN)

Neural networks are based on perceptron extensions, and ANNs can be understood as neural networks with many hidden layers, so ANNs are sometimes called multi-layer perceptron (MLP). According to the location of different layers in ANN, the internal neural network layers can be divided into three categories: input layer, hidden layer, and output layer. The ANN diagram used in this study is shown in Fig. 1. The first layer is the input layer, then the hidden layer and the last layer is the output layer. The output of the previous layer is used as the input value of the later layer, and when ANN is used as regression, there is only one neuron in the output layer.

Fig. 1.

Outline of ANN. The input layer has six neurons, the hidden layer has four neurons, and the output layer has one neuron

The ANN model in the figure above can be expressed by the formulas as follows:

1

2

In the formulas Eqs. 1 and 2, the function represents the activation function of the ith layer, refers to the ith element of the input vector x, d represents the number of neurons in the input layer, and N denotes the number of neurons in the hidden layer. The symbols and denote the weight of neurons in the input layer and the hidden layer, respectively. After the weighted sum of each layer of neurons, the offsets are denoted by and b. The learning process of the neural network is to update the weight and bias so that the loss function is minimized, i.e.,

The form of the function J varies depending on the problem. The variables predicted in this paper are continuous, so MSE was chosen as the loss function.

Overfitting is a common problem in deep learning. Overfitting refers to a state where one can only fit the training data but not well fit other data that is not included in the training data. Regularization prevents overfitting. The regularization method used in this paper is Dropout, proposed by Srivastava et al. (2014). It randomly deletes neurons during the learning process so they do not participate in the next layer of the neuron meter. This method is one of the most effective and commonly used regularization techniques in neural networks. Dropout values range from 0 to 1 (Weidman 2019). In this study, the dropout value is fixed as 0.05. The learning rate determines whether and when the objective function converges to the local minimum, which is essential in deep learning. If the learning rate is too low, the loss function will change slowly, significantly increasing the convergence complexity of the network, and it is easy to get trapped in the local minimum value. If the learning rate is too high, the loss function may directly pass the global optimal value, the gradient oscillates around the minimum value, and may even fail to converge. We set two learning rate values (0.001, 0.005) to prevent these two situations from training the model. The ReLU activation function is utilized for ANN due to its simplicity and effectiveness. Generally, it refers to the ramp function in mathematics, defined as . This can be expressed as a piecewise function as shown below:

Convolutional neural network (CNN)

Convolutional neural networks are widely used algorithms in deep learning, suitable for situations where input variables are distributed along spatial patterns, such as 1D-CNN (e.g., SNP or text), 2D or 3D (e.g., images). The CNN model is a particular type of neural network that uses convolution instead of full matrix multiplication in hidden layers (Goodfellow et al. 2016. In one-dimensional convolutional neural networks, from input to output, the model can be expressed as , where x is the input vector, y is the output vector, and w is the parameter vector (convolution operator). The length of y is , and each element in y is evaluated as

The structure of CNN is divided into three layers: the convolution layer, the pooling layer, and the fully connected layer. Each neuron in the latter convolutional layer is connected only to neurons in a small rectangular region in the previous convolutional layer. The former convolutional layer of CNN focuses on relatively low-level features, while the latter convolutional layer integrates the features of the previous convolutional layer to form relatively complex features. Pooling is an operation to reduce the dimensionality of the input object in terms of height and length, such as Max pooling (selecting the maximum value from the target area) and average pooling (calculating the average value of the target area). Pooling can also reduce the complexity of the network without damaging the recognition results and, to some extent, prevent overfitting. In the fully connected layer, the neurons of adjacent layers are all connected, and the number of outputs can be arbitrarily determined.

The CNN used in this paper combines multi-layer convolution, max pooling layers and nonlinear activation functions, as shown in Fig. 2. There are three convolutional layers in Fig. 2, and each layer will be pooled after the convolution. Because the dependent variable analyzed in this paper is a scalar, the last layer is a fully connected layer with one neuron output. The activation function ReLU is also commonly used in convolutional neural networks.

Fig. 2.

Outline of CNN. The architecture consists of three convolutional layers, each followed by a ReLU activation and max pooling, and concludes with a fully connected layer and a ReLU activation

The ANN and CNN models utilized identical structured inputs. We integrated environmental, genotypic, and genotype by environment interaction components into a unified design matrix , constructed as follows:

• the design matrix for environmental effects , a incidence matrix allocating the kp observed phenotypic values to k environments.

• the design matrix for genomic effects , which was derived from the genomic relationship matrix (GRM) :

  
• the design matrix for genotype-by-environment interaction effects , which was formulated through Kronecker product :

  

When the interaction is disregarded, the input becomes: . In both the ANN and CNN models, the input vector is a row vector extracted from the design matrix . A detailed description of the design matrix can be found in Montesinos-López et al. (2022). To simplify the calculation of deep learning models, we used Keras, the most popular advanced deep learning framework (Allaire 2018; Géron 2022).

Cross-validation and predictive accuracy

We considered the prediction problems for two different scenarios. The first type of cross-validation is denoted as CV1. We assessed the prediction accuracy of the models when training (TRN) and testing (TST) data consist of disjoint sets of lines. This method mimics the problem of predicting lines that have not been evaluated in any field trials. To generate TRN and TST sets in CV1, we randomly assigned of the lines to TRN and the remaining to TST. To ensure robustness, this partitioning procedure was independently repeated five times. While this differs from conventional 5-fold cross-validation (where folds are mutually exclusive and exhaustive), the results are not anticipated to diverge significantly from standard cross-validation protocols under sufficiently large sample sizes.

The second cross-validation strategy CV2 comes from Lopez-Cruz et al. (2015). We considered predicting the performance of lines in environments in which lines have not been evaluated. This approach mimics the prediction problem in incomplete field trials where lines are evaluated in some but not all target environments. There are m environments denoted as , and w lines denoted as , so there are records in each data set. To select the entries in the TST data set, we randomly chose lines and then randomly picked one environment per line. The resulting data was used as the TST set, and the unselected data was used as the TRN set. As the data in this paper satisfies , some lines contain multiple environments when sampling. We used the Pearson correlation coefficient between the predicted results and observed values in the TST dataset to determine the prediction accuracy of each model. CV2 adopted the identical five-repetition framework established in CV1, maintaining independent data partitioning across all iterations.

In the deep learning model, we applied an inner CV strategy to determine the hyperparameters using grid search. The inner CV strategy requires splitting each training set of the (outer) CV strategy above. In each training set (TRN), of genotypes was assigned as the inner TRN and the remaining as the inner TST. For the ANN model, the hyperparameters to be tuned are the number of neurons, batch size and learning rate value. There are two values (30 and 90) for the number of neurons, two values (28 and 56) for batch size, and there are also two learning rate values (0.001, 0.005). For other hyperparameters, only one value was used. So there are 8 hyperparameter combinations to train the artificial network model. The inner TST set was used to test the accuracy of 8 hyperparameter combinations. Then we selected the hyperparameter combination with the highest accuracy for outer cross-validation. Finally, we used the obtained hyperparameters to construct the model and calculate its accuracy. For the CNN model, the inner five-fold cross-validation was used to choose dropout, the number of unit and filter. In grid search, the CNN model also has 8 hyperparameter combinations: two values (0, 0.05) of dropout, two units (33, 57), and two values (32, 64) of filters. This inner CV strategy can be simplified using Keras’ (version 2.11.1) internal function and the validation_split parameter on the fit() function.

In the paper, statistical analysis of variance (two-way ANOVA) was performed to assess whether the accuracy of each model was affected by interaction or cross-validation strategies. As a remark, we emphasize that all models described in the previous subsections were implemented by assuming independent environmental effects, i.e., there is no environmental covariance. Of course, the assumption does not necessarily hold in reality. Nevertheless, it is not possible to explicitly model the covariance as no environmental covariates are available in the datasets used in this study. Although there is the option of assuming a general covariance matrix among the environments and let the model itself estimate the missing covariance parameters, it would probably lead to an over-parametrized model. Therefore, we decided to maintain this assumption for all models throughout this study.

Results

Comparing the prediction accuracies between the traditional and deep learning models

The prediction accuracies of all models conducted with and without interaction, in each environment for each data set and for each CV strategy were shown in Figs. 3-5. In the following, if there are no special instructions, the accuracy of each model refers to the mean performance across all environments (Fig. 6). In CV1, when the interaction was not considered, the ANN and CNN models were better than traditional models. In each dataset, the accuracy of the ANN and CNN models were higher than the average accuracy of all traditional models; on average across data sets, the accuracy of ANN (CNN) was () higher than the average accuracy of traditional models. When considering interaction, the traditional Bayesian models (BA, BB and BRR) outperformed the GBLUP, ANN and CNN models. In each dataset, the average accuracy of traditional Bayesian models was higher than the accuracy of the ANN and CNN models; on average across data sets, the average accuracy of traditional Bayesian models was higher than the accuracy of ANN, and higher than the accuracy of CNN. However, in CV2, when the interaction was not considered, the difference between the accuracies of ANN and traditional models was not significant (T-test, ). And the accuracy of the CNN model was lower than the other five models. In CV2, when considering interaction, the traditional Bayesian models outperformed the GBLUP, ANN and CNN models. In each dataset, the average accuracy of traditional Bayesian models was higher than the accuracy of the ANN and CNN models; on average across data sets, the average accuracy of traditional Bayesian models was higher than the accuracy of ANN, and higher than the accuracy of CNN.

Fig. 4.

Prediction accuracy of all models in different scenarios for Data2. (a) Estimated prediction accuracy of all models in four environments with no interaction for Data2 in CV1; (b) Estimated prediction accuracy of all models in four environments with interaction for Data2 in CV1; (c) Estimated prediction accuracy of all models in four environments with no interaction for Data2 in CV2; (d) Estimated prediction accuracy of all models in four environments with interaction for Data2 in CV2. BA, BayesA; BB, BayesB; BRR, Bayesian Ridge Regression; GBLUP, Genomic Best Linear Unbiased Prediction; ANN, Artificial Neural Network; CNN, Convolutional Neural Network; SE, Standard Error; PA, Prediction Accuracy

Fig. 3.

Prediction accuracy of all models in different scenarios for Data1. (a). Estimated prediction accuracy of all models in four environments with no interaction for Data1 in CV1; (b). Estimated prediction accuracy of all models in four environments with interaction for Data1 in CV1; (c). Estimated prediction accuracy of all models in four environments with no interaction for Data1 in CV2; (d). Estimated prediction accuracy of all models in four environments with interaction for Data1 in CV2. BA, BayesA; BB, BayesB; BRR, Bayesian Ridge Regression; GBLUP, Genomic Best Linear Unbiased Prediction; ANN, Artificial Neural Network; CNN, Convolutional Neural Network; SE, Standard Error; PA, Prediction Accuracy

Fig. 5.

Prediction accuracy of all models in different scenarios for Data3. (a) Estimated prediction accuracy of all models in five environments with no interaction for Data3 in CV1; (b) Estimated prediction accuracy of all models in five environments with interaction for Data3 in CV1; (c) Estimated prediction accuracy of all models in five environments with no interaction for Data3 in CV2; (d) Estimated prediction accuracy of all models in five environments with interaction for Data3 in CV2. BA, BayesA; BB, BayesB; BRR, Bayesian Ridge Regression; GBLUP, Genomic Best Linear Unbiased Prediction; ANN, Artificial Neural Network; CNN, Convolutional Neural Network; SE, Standard Error; PA, Prediction Accuracy

Fig. 6.

Mean prediction accuracy across environments for all models. (a) Mean prediction accuracy across environments for all models conducted with and no interaction in Data1 and for each CV; (b) Mean prediction accuracy across environments for all models conducted with and no interaction in Data2 and for each CV; (c) Mean prediction accuracy across environments for all models conducted with and no interaction in Data3 and for each CV

Based on the above observations, it is clear that the ranking of model performance depends on the CV strategy and was affected by whether the interaction was considered. We can conclude that, regardless of the CV strategy, the traditional Bayesian models were better than ANN and CNN when considering interaction. The ANN and CNN models were superior to the traditional models only when the interaction was not considered and the CV1 strategy was adopted.

The influence of CV strategy and interaction on the accuracies of different models

The Table 1 presents the results of a two-way ANOVA analyzing the interaction between CV strategy and interaction across all models. It was shown that, neither the CV strategy nor its interaction with the interaction was significant () for traditional models, indicating no significant change in the accuracy between CV1 and CV2, regardless of whether the interaction was considered or not. But the way of treating interaction significantly () affected the accuracy. In both CV strategies, considering interaction increased the accuracy of traditional Bayesian models. Across three traditional Bayesian models in three datasets, the average accuracy improved by in CV1 and by in CV2. In contrast, the accuracy of the ANN and CNN models was significantly () affected by the interaction between the CV strategy and the way of treating interaction. In CV1, considering interaction decreased the accuracy of the ANN and CNN models. On average across three datasets, the accuracy of ANN decreased by , and that of CNN decreased by . In CV2, considering interaction improved the accuracy of the ANN and CNN models. On average across three datasets, the accuracy of ANN improved by , and that of CNN improved by . When the interaction was not considered, the accuracy of the ANN and CNN models was significantly () higher in CV1 than in CV2, while the opposite was observed when the interaction was considered (Fig. 7).

Table 1.

The results of the two-way ANOVA for the interaction between cross-validation and interaction.

	Source	SS	df	MS	F statistic	P-value
BA	CV	0.000602	1	0.000602	0.062674	0.803387
		0.283717	1	0.283717	29.51403	0.00000182
	Interaction	0.00000556	1	0.00000556	0.000578	0.980917
	Error	0.461422	48	0.009613
BB	CV	0.00512	1	0.00512	0.820607	0.369528
		0.24702	1	0.24702	39.58876	8.96E-08
	Interaction	0.023609	1	0.023609	3.783689	0.057625
	Error	0.299504	48	0.00624
BRR	CV	0.00128	1	0.00128	0.100535	0.752564
		0.277985	1	0.277985	21.83233	0.0000244
	Interaction	0.000192	1	0.000192	0.015103	0.902703
	Error	0.61117	48	0.012733
GBLUP	CV	0.00221	1	0.00221	0.098446	0.755062
		0.06732	1	0.06732	2.998793	0.08975
	Interaction	1.92E-08	1	1.92E-08	0.000000857	0.999265
	Error	1.077554	48	0.022449
ANN	CV	0.005838	1	0.005838	0.480445	0.491561
		0.023481	1	0.023481	1.932257	0.170925
	Interaction	0.123581	1	0.123581	10.16942	0.002515
	Error	0.583308	48	0.012152
CNN	CV	0.012836	1	0.012836	0.865273	0.356922
		0.02628	1	0.02628	1.771488	0.189486
	Interaction	0.307538	1	0.307538	20.73061	0.0000362
	Error	0.71208	48	0.014835

SS, Sum of Squares; df, degree of freedom; MS, Mean Squares

Fig. 7.

Prediction accuracy of all models in different environments for all data sets combined. (a) Prediction accuracy of all models with interaction; (b) Prediction accuracy of all models with no interaction. Boxplots contain 13 prediction values. Red boxes show results of CV1, while blue boxes show results of CV2

A detailed comparison between the accuracies of ANN and CNN models

Figure 8 shows the prediction accuracy of the ANN and CNN models in each environment within each data set, and for each of the four combinations of CV strategies (CV1 and CV2) and means of treating the interaction (considering and ignoring). In CV1, regardless of whether the interaction was considered, the CNN model exhibited higher accuracy than the ANN model across environments , , and for both Data1 and Data3, with the sole exception of in Data1 when the interaction was ignored. In CV2, when considering the interaction, the CNN outperformed the ANN in , , and for Data2, as well as in for Data3. In all other scenarios, the ANN model outperformed the CNN model. On average across the four combinations in all environments, the prediction accuracy of ANN was higher than CNN in all three data sets. The paired samples t-test showed that the difference was significant (i.e. ) in Data1 () and Data2 (), and was not significant in Data3 (). Regardless of whether the interaction was considered, in three out of four environments (, and ), the average accuracy of ANN across all datasets was significantly () higher than that of CNN. Therefore, we conclude that in our case, the ANN model was superior to the CNN model.

Fig. 8.

Prediction accuracy of ANN and CNN in different scenarios. (a) The prediction accuracy of ANN and CNN in Data1; (b) The prediction accuracy of ANN and CNN in Data2; (c) The prediction accuracy of ANN and CNN in Data3

Discussion

There are many instances of the effective use of machine learning in genomic prediction (Ma et al. 2018; González-Camacho et al. 2012; Libbrecht and Noble 2015; Okser et al. 2014; Pérez-Enciso and Zingaretti 2019; Sandhu et al. 2021b). It was expected that ML models are superior to traditional genomic selection models in terms of prediction accuracy. However, there is not enough empirical evidence in the literature to support this claim (Montesinos-López et al. 2019). Thus, we investigated the applicability of established machine learning models in genomic prediction by comparing the ANN and CNN methods with traditional prediction models (BA, BB, BRR, and GBLUP) using public datasets in wheat.

In this study, we found that the accuracy of ANN and CNN models was similar to or lower than traditional models when considering the interaction. Only in CV1, when the interaction was not considered, the ANN and CNN models outperformed traditional models. This was unexpected because using deep learning methods may help to capture complex interaction, which often helps to improve prediction accuracy. But our findings are supported by results of previous studies. Montesinos-López et al. showed that a deep learning method with densely connected network architecture was as effective as the GBLUP method, but generally speaking, the GBLUP method outperformed the deep learning method when considering interaction (Montesinos-López et al. 2018). However, when ignoring the interaction, the deep learning method was better than the GBLUP method in six out of the nine datasets. Montesinos-López et al. compared the prediction accuracy of MLP, support vector machine (SVM), and the Bayesian threshold genomic best linear unbiased prediction (TGBLUP) model in four datasets with and without interaction (Montesinos-López et al. 2019). When the interaction was considered, the TGBLUP model produced the best prediction in all four datasets. When the interaction was ignored, the accuracy of TGBLUP was similar to or lower than that of MLP and SVM in all four datasets. As discussed in Montesinos-López et al. (2018), deep learning models could capture complex relationships hidden in the data without a need for strong assumptions about underlying mechanisms, which were frequently unknown or insufficiently defined. This might be the reason for the superiority of deep learning models over the GBLUP model when interaction was not considered. On the other hand, the deep learning models often require a large amount of data to train effectively. When considering the interaction, the dimensionality of the data increases, but the sample size does not increase. Thus, in this case, it would be difficult to exploit the advantage of deep learning models. This might explain why the traditional prediction models outperformed the deep learning models when interaction was considered.

It is important to note that all models considered in this study were implemented with the assumption of no environmental covariance, as was already mentioned in Materials and Methods. As a consequence, the performance of the evaluated models and their rankings in different scenarios (with or without interaction) were strongly affected by this assumption. If additional data measuring the environments, such as climate and soil parameters, are available, a better approach is to model the environmental covariance by using these factors (Jarquín et al. 2014). In such a case, the machine learning models may be able to capture the implicit patterns of phenotype-envirotype-genotype relations, and hence have the potential to increase the prediction accuracy (Crossa et al. 2022). This is also relevant for the application of genomic prediction in reality, since for breeders, the most critical scenario is predicting new genotypes in new environments. But this scenario cannot be evaluated without data specifically describing the new environments. While investigating these issues is beyond the scope of the current study, it is certainly a promising and interesting direction for further research.

There were also studies in which deep learning models were reported to have the best performance or at lease one of the best models in terms of prediction accuracy. To predict grain protein content and grain yield in wheat, Sandhu et al. evaluated nine models, including GBLUP, four Bayesian models, Random Forest (RF), SVM, MLP, and CNN (Sandhu et al. 2021a). The results indicated that MLP performed the best. To predict grain length in wheat, Ma et al. evaluated CNN and two popular genomic prediction models (RR-BLUP and GBLUP) and three versions of the MLP (Ma et al. 2018). They concluded that the best models were CNN, RR-BLUP and GBLUP. Thus, it seems that the relative advantage between the traditional and deep learning models depends on the trait, and even on the specific data set. This is justified because of the No Free Lunch (NFL) Theorem (Wolpert and Macready 1995), which states that for any algorithm, any elevated performance over one class of problems is exactly paid for in performance over another class.

The commonly used deep learning models in plant breeding are MLP and CNN, as reported by Pérez-Enciso and Zingaretti (2019). The ANN model used in this paper is the MLP model with one hidden layer. The results of our study showed that the ANN model was generally better than the CNN model in three data sets. This finding is supported by Sandhu et al. (2021b) in which the performance of MLP and CNN for predicting five traits in wheat was evaluated. And the results showed that the prediction accuracy of the MLP model was higher than that of the CNN model for both grain yield and grain protein content in wheat. However, there were also contrasting results in wheat reported in the literature. According to Pérez-Enciso and Zingaretti (2019), the CNN model showed better performance than MLP for predicting grain yield in wheat. Ma et al. reported that the performance of the CNN model was slightly better than the GBLUP model which in turn was more accurate than the MLP model in predicting grain length in wheat (Ma et al. 2018). Thus the relative advantage of ANN and CNN are still not clear and further studies are needed.

Both traditional prediction models and deep learning models are potent tools for genomic selection. Traditional predictive models have the advantage of providing a transparent build process and interpretability, avoiding adjusting hyperparameters, but sometimes the resulting accuracy may not be satisfactory. For example, the accuracy of GBLUP in this paper is relatively low in most cases. As datasets grow in size and feature richness, using deep learning models becomes increasingly feasible (Jubair and Domaratzki 2023). However, it is also prone to overfitting, and it requires training a large number of parameters, such as the initial values of network topology, weights, and thresholds. And one cannot observe the learning process in using a deep learning model. It is challenging to interpret the process of prediction (the output results are the predicted values, which does not require interpretations), which will affect the credibility and acceptability of the results. Hence, depending on the species/trait in consideration and the scenario of prediction, deep learning methods can be a powerful complement to traditional genomic prediction tools.

Funding

No funding was received.

Data Availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval

All authors approved the submission.

Consent to participate

Not applicable.

Consent for publication

Not applicable.

Conflict of interest

The authors declare no competing interests.