Random forest imputation and genomic prediction for missing egg production time-series data in yellow-feathered broiler breeders

Abstract

Background

Egg production is a key economic trait in poultry breeding, and its longitudinal records are essential for accurate genetic evaluation. However, time-series egg production data often contain missing values due to sensor failure, recording interruption, or operational error. Such missingness not only reduces the accuracy of phenotypic reconstruction but may also affect downstream breeding-value estimation. This study aimed to compare multiple imputation strategies for missing egg production time-series data and to evaluate them from two complementary perspectives: imputation accuracy and downstream genomic and pedigree-based prediction performance.

Results

We benchmarked six imputation strategies—Forward–Backward Mean, piecewise linear regression, spline regression, K-Nearest Neighbors, Random Forest (RF), and Long Short-Term Memory (LSTM) networks—using datasets from 4,390 yellow-feathered broilers, encompassing more than 100,000 egg production records and 463,000 SNPs. In terms of imputation accuracy, RF consistently outperformed the alternative methods across simulated missingness rates of 5%, 10%, 15%, and 20%, reducing RMSE by 15.50% – 31.22% and MSE by 28.31% – 52.69%, while improving R² by 7.42% – 23.41% relative to the other methods. In downstream evaluation, the accuracy of Genomic Estimated Breeding Values (GEBV) without imputation ranged from 0.239 to 0.277, whereas imputation improved it to 0.288 – 0.293. The Random Forest method emerged as the most robust approach, delivering significant accuracy improvements of 5.78% to 21.76% (p < 0.05).

Conclusions

This study demonstrates that Random Forest imputation is a highly effective tool for resolving missing data challenges in egg production time-series. By bridging the gap between raw data processing and genomic prediction, these findings provide a practical computational framework for enhancing the reliability of breeding programs in the poultry industry and other livestock species with longitudinal data.

Background

In poultry breeding, particularly within the broiler industry, production performance serves as the core metric for evaluating economic efficiency and breeding outcomes. Egg production, as a critical longitudinal phenotypic trait, requires complete and accurate time-series records to effectively assess flock physiological status and uncover genetic potential. Unlike static phenotypic snapshots, egg production is a dynamic physiological process; thus, missing data in these sequences can obscure the genetic merit trajectory of the individual. However, in large-scale commercial production environments, missing time-series data for egg production frequently occurs due to factors such as sensor malfunctions, data transmission interruptions, or human operational errors [1, 2]. Missing data compromises the accuracy of production performance evaluations and complicates downstream genetic parameter estimation and genomic prediction.

To address missing data, a range of imputation methods has been developed, from simple statistical approaches to machine-learning-based methods. Traditional approaches, such as neighboring mean imputation and smoothing-based fitting [3, 4], are computationally efficient but may fail to capture the non-linear trends and temporal dependencies of egg production trajectories. In contrast, machine-learning-based methods, including Random Forest and Long Short-Term Memory (LSTM) networks [5, 6], offer greater flexibility for modeling complex patterns in longitudinal data and have shown promising performance in missing-data recovery in several biological and agricultural applications.

However, despite the broad development of imputation methods, their application to poultry egg production time-series data remains insufficiently evaluated. In particular, previous studies have often focused primarily on the numerical accuracy of the imputed values, while paying less attention to whether the repaired phenotypes remain suitable for downstream genetic evaluation. This distinction is important because high imputation accuracy does not necessarily guarantee preservation of the covariance structure required for reliable breeding-value estimation. If the imputed data are numerically close to the true values but distort or dilute the association between phenotype and genotype, they may severely mislead breeding decisions [7]. Moreover, while extensive research has focused on imputation methods and algorithm optimization for genotype data [8–11], corresponding studies on phenotypic data—especially regarding how data completeness influences genomic prediction—remain limited. Therefore, both the recovery of original phenotypic values and the impact of imputed phenotypes on downstream prediction should be considered when evaluating imputation methods in breeding applications.

To address these gaps, this study establishes a comparative benchmark for handling missing longitudinal egg production data in yellow-feathered broilers. We evaluated six representative imputation strategies, ranging from traditional statistical approaches to machine-learning-based methods. The present work evaluates imputation methods from two complementary perspectives: the accuracy of phenotypic reconstruction and the influence of repaired phenotypes on downstream genomic and pedigree-based breeding-value prediction. Our findings aim to provide a practical basis for improving the preprocessing of longitudinal phenotypes in poultry breeding.

Methods

Material

This study used time-series data on egg production from the 27th generation of caged yellow-feathered broilers, provided by Guangdong Wiz Agricultural Science & Technology Co. Ltd. The cumulative egg production from 20 to 43 weeks was selected as the phenotype for the single-trait model. For data cleaning, outliers were removed from the dataset. Outliers were defined as records deviating by more than three standard deviations [12] from the mean cumulative egg production from 20 to 43 weeks. This procedure was performed using R (v4.4.2) [13]. Individuals with missing cage information were also removed. Ultimately, 105,360 egg production phenotypic records from 4390 birds remained as the complete dataset for subsequent analysis. All individuals used in the analysis had complete egg production records, batch, and cage. The pedigree data used in the analysis contained a total of 112,690 records, with a pedigree depth of four generations. A summary of statistics for the cumulative egg production data from 20 to 43 weeks is presented in Table 1, and the average weekly egg production curve is shown in Fig. 1. To simulate missing data scenarios, we processed the complete dataset by introducing data missing completely at random (MCAR) at rates of 5%, 10%, 15%, and 20%. These levels were selected to represent mild to moderately severe missing-data scenarios and to enable a systematic comparison of method performance under increasing levels of missingness. In practical production settings, missing rates are typically moderate, and the selected range broadly reflects realistic conditions while also including more challenging scenarios. For each missing rate, the simulation was repeated 20 times using different random seeds, generating a total of 80 missing datasets.

Table 1.

Summary of statistics for the cumulative egg production data

Batch	Count	Min (eggs)	Max (eggs)	Mean (eggs)	SD (eggs)
1	1040	73.00	130.00	97.77	10.62
2	1180	73.00	134.00	99.70	11.33
3	1106	73.00	140.00	99.47	12.30
4	1064	73.00	139.00	99.27	13.63

Fig. 1.

Average weekly egg production of the 27th generation birds

In this study, a total of 11494 yellow-feathered broilers, including all individuals from the 27th generation, were genotyped using low-depth resequencing (463182 SNPs), and genotype imputation was performed with Beagle [14]. Genomic data quality control (QC) was performed using PLINK (v1.07) [15]. The criteria were as follows: SNP markers were removed if they had genotype-missing rates > 0.10, minor allele frequencies (MAF) < 0.05, or a Hardy–Weinberg equilibrium (HWE) p-value ≤ 10^–5 (based on a Chi-squared test). Samples with a call rate lower than 90% were also removed. After QC, 11,411 genotyped individuals (including 5037 from the 27th generation) and 463,182 high-quality SNPs remained for the subsequent study.

The evaluation involved eight types of datasets, which were grouped into three categories: the complete dataset, the dataset with missing values, and six imputed datasets. These six imputed datasets were generated by applying their respective imputation methods to the incomplete dataset. For the complete dataset, the Estimated Breeding Values (EBV) were calculated directly using two genetic evaluation models. The analyses for the datasets with missing values and the imputed datasets were conducted at the four different missing rate levels. For each missing rate, we employed a repeated five-fold cross-validation strategy, where within each run, the data were randomly partitioned into five folds(80% training, 20% validation) [16], and each fold served as the validation set once; this entire five-fold procedure was then repeated ten times with different random seeds to ensure robustness of the results.

For the genetic evaluation models, we used two methods: Pedigree-based Best Linear Unbiased Prediction (PBLUP) [17] and single-step Genomic Best Linear Unbiased Prediction (ssGBLUP) [18, 19]. Both models were fitted with the same fixed effects, including batch and cage effects, which were previously tested and found to have a significant influence on the egg production. The estimation of genetic parameters was conducted using the Average Information-Restricted Maximum Likelihood (AI-REML) algorithm [20]. All models were constructed and computations were performed using the DMU (v5.2) software [21]. The single-trait models for PBLUP and ssGBLUP are as follows:

Where, is a column vector containing all phenotypic observations for all individuals; is a vector containing all fixed effects; is a vector of random effects containing the additive genetic effects for all individuals; is a vector containing all residual effects; and are incidence matrices that relate the effect vectors and to the observation vector . In the PBLUP and ssGBLUP models, the distributions of the additive genetic effects, , the assumptions are as follows:

where is the pedigree relationship matrix and is an identity matrix. The matrix seamlessly integrates the pedigree relationship matrix () with the genomic relationship matrix (). The elements of are constructed from pedigree as follows:

The diagonal elements are computed as , where is the inbreeding coefficient of individual , calculated as . The off-diagonal elements are computed as , where and are the parents of individual .

In the mixed model equations, its inverse () is used, which is constructed as follows:

where: is the inverse of the pedigree relationship matrix for all animals (both genotyped and non-genotyped); is the inverse of the tuned or weighted genomic relationship matrix for genotyped animals. is often tuned (e.g., , = 0.05) [22] to align with and account for differences in scale or missing pedigree; is the inverse of the pedigree relationship matrix for the genotyped animals only. The block matrix adds the genomic information component only to the subset of genotyped individuals.

Modelling

Traditional methods

Forward–Backward Mean Imputation (FBM), a simple form of linear interpolation, is a data preprocessing method based on the statistical characteristics of adjacent observations [23]. The core principle of this method is to replace a missing value with the average of its nearest preceding and succeeding values in the time series. For a missing value , its imputation formula is as follows:

Where, represents the window of available observations neighboring the missing value ; denotes the number of valid observations within the window. If consecutive missing values are encountered, the process strictly follows the time-series sequence, adopting a recursive method for iterative filling.

Statistical model methods

Piecewise Linear Regression Imputation (PLR) is a method that addresses missing values by identifying “breakpoints” within the data [24]. The method begins by provisionally filling the missing values with the mean. It then automatically determines the optimal breakpoints, partitioning the data into multiple linear sub-intervals, using the “segmented” package in R [25]. Subsequently, a separate linear regression model is fitted within each sub-interval to predict the values. The expression for the model is as follows:

Where, is the predicted egg production for the individual in the week ; is the fixed intercept for segment ; is the fixed effect coefficient for the batch effect within segment ; represents the deviation of the baseline egg production value of the individual from the overall mean; is the residual error, and is assumed to be normally distributed as , is the residual variance.

Natural Splines Imputation (NS), a method based on mixed-effects models, is suitable for handling data characterized by hierarchical structures or repeated measures [26]. This method fits a linear mixed-effects model that incorporates both fixed effects (e.g., batch effect) and random effects (e.g., individual effect). The model employs natural spline functions to capture non-linear temporal trends. Parameters are estimated using Restricted Maximum Likelihood (REML) in conjunction with the Bobyqa optimization algorithm [27]. Furthermore, a boundary constraint is applied to ensure that predicted egg production values are greater than zero. The entire modeling process is implemented using the “lme4” package in R [27]. The model is expressed as follows:

Where, and are the global intercept and the fixed batch effect, respectively; is the natural spline component that describes the non-linear temporal trend; and are the random intercept and random slope for individual , respectively; is the random residual.

Machine learning methods

K-Nearest Neighbors Imputation (KNN) is an imputation technique based on a non-parametric statistical framework [28]. Its core principle involves identifying the K nearest neighbors for an observation with a missing value, using a defined distance metric within the feature space. The attributes of these neighbors are then used to estimate the missing value, for example, through a weighted average. In this study, KNN was implemented using the KNN function from the “VIM” package in R [29]. Gower’s distance was employed to handle mixed data types, and an inverse-distance weighted average was used for the imputation. The expression is as follows:

Where, is the estimate for the missing value; is the known value of the i-th nearest neighbor; is the selected number of neighbors; The weight is defined as , where is the distance to the -th neighbor and is a small constant to prevent division by zero. For the KNN model, the optimal K-value was selected according to the elbow method, choosing the value that corresponded to the minimum MSE (Fig. 2).

Fig. 2.

Effect of the number of neighbors (K) on Mean Squared Error (MSE)

Random Forest Imputation (RF) is a non-parametric machine-learning approach for handling missing data [30]. In this study, RF imputation was implemented using the “missForest” package in R [31]. The input matrix consisted of batch information and weekly egg production records, while the individual identifier was excluded from model fitting. The batch variable was treated as a categorical predictor, and all weekly records were jointly used as predictors during iterative imputation. For each iteration, one variable containing missing values was treated as the response and predicted from the remaining variables using a random forest model. The imputed matrix was updated iteratively until convergence or until the maximum number of iterations was reached.

To maintain consistency across all simulated datasets, fixed RF parameter settings were used throughout the study. Specifically, the settings were ntree = 200, maxiter = 10, and parallelize = “forests”. In the random forest framework, bootstrap sampling and random feature selection at each tree node collectively help mitigate overfitting risk. Complementarily, the Out-of-Bag (OOB) error, calculated from observations excluded from each bootstrap sample, serves as an internal validation metric. The principle of this method is illustrated in Fig. 3.

Fig. 3.

Flowchart of the Random Forest Imputation process

In the model expression, let denote the incomplete data matrix, where indexes individuals and indexes variables. For a variable containing missing entries, the observed part of was used to train a random forest predictor based on the remaining variables The missing values were then updated as , , where denotes the set of individuals with missing values in variable , and is the iteration index. This procedure was repeated over variables until convergence. No separate validation partition was used for RF imputation, because the “missForest” procedure performs iterative imputation internally under fixed parameter settings.

Long Short-Term Memory (LSTM), proposed by Hochreiter and Schmidhuber [5], is a recurrent neural network architecture designed to model sequential dependence in time-series data. In this study, each individual’s egg production trajectory from week 20 to week 43 was represented as a sequence. Missing values were first temporarily interpolated to construct an initial complete sequence, after which the data were normalized using Min–Max scaling. The input at each time step included three components: the egg production value, the encoded batch information, and a missingness indicator. Sliding windows of length 7 were generated as model inputs, and the target output was the subsequent egg production value.

The neural network consisted of one LSTM layer with 64 hidden units, followed by a dense layer with 32 neurons and a final output layer with one neuron. Model training used the Adam optimizer with a learning rate of 0.001 and mean squared error as the loss function. Early stopping based on validation loss was applied to reduce the risk of overfitting. To maintain consistency across all simulated datasets, fixed parameter settings were used throughout the study, including seq length = 7, iterations = 3, train batch size = 1024, and predict batch size = 4096. The generated sequence samples were randomly divided into training and validation subsets at a ratio of 80:20, and the validation subset was used only for early stopping rather than for hyperparameter optimization. Predicted values were iteratively fed back to replace only the originally masked missing positions. Furthermore, a non-negativity constraint (≥ 0) was imposed on the final imputed egg production values. The principle of this method is illustrated in Fig. 4.

Fig. 4.

Structure of a Long Short-Term Memory (LSTM) unit. is the egg production at time step , is the cell state at time step , is the cell state at time step , represents the sigmoid function; is the output at time step , and is an activation function used in the neural network

In the model expression, for individual , let the input sequence within a window of length L be , where each contains the scaled egg production value, batch encoding, and a missingness indicator. The LSTM model learns a mapping , where denotes the neural network parameterized by , and is the predicted egg production value at the next time step. The predicted values were iteratively fed back to replace only the originally missing positions.

Implementation details of machine-learning methods

In the present study, the machine-learning models were implemented using fixed parameter settings rather than a formal grid-search or Bayesian optimization framework. This design was intended to maintain methodological consistency across all simulated datasets and to reflect a practically deployable imputation setting. Under this design, the same predefined parameter settings were applied to all missing-rate scenarios, which allowed a fair comparison of model behavior while keeping the computational burden manageable. The parameter settings for the machine learning models are detailed in Table 2.

Table 2.

Parameter settings used for the machine-learning imputation models in this study^a

Imputation Method	Computational Environment	Hyper parameters
KNN	R 4.4.2	K: 8; Distance Metric: Gower’s distance; Weighting: inverse-distance
RF	R 4.4.2	Ntree: 200; maxiter: 10; parallelize = forests
LSTM	Python 3.10.18	Hidden units: 64; Dense units: 32 → 1; Dropout: 0.2; Optimizer: Adam; Learning rate: 0.001; Loss: MSE; Epochs: 100; Early stopping patience: 10; Sequence length: 7; Iterations: 3; Batch size: 1024; predict batch size = 4096

^aFixed parameter settings were used across all simulated datasets and missing-rate scenarios to ensure methodological consistency and comparability among methods while keeping the computational burden manageable

Heritability estimation

Heritability was estimated for the complete dataset, the datasets with simulated missing values, and the six imputed datasets using the same single-trait animal model framework described above. Variance components were estimated by AI-REML in DMU (v5.2) [21], and heritability was calculated as , where and denote the additive genetic variance and residual variance, respectively. Standard deviations (SD) of heritability estimates were calculated across replicate datasets and were used to compare the extent to which different imputation strategies preserved genetic information.

Statistical analysis of data

To evaluate the impact of phenotypic data imputation on the accuracy and unbiasedness of estimated breeding value, this study was conducted in three steps. First, the performance of each imputation method was evaluated by comparing the imputed values against the original true values at the positions where data were missing. The metrics for this comparison were the Root Mean Squared Error (RMSE), Mean Squared Error (MSE), and the coefficient of determination (R²) [32, 33]. Second, the accuracy of the EBV was assessed. Using the corrected phenotypes (EBV + Residual) from the complete dataset as the benchmark, we calculated the Pearson and Spearman correlations between this benchmark and the EBV obtained from both the datasets with missing values and the imputed datasets. To further evaluate the effects of imputation method and missing rate on prediction accuracy, a two-way analysis of variance (ANOVA) was performed with imputation method and missing rate as fixed effects, followed by Tukey’s HSD post-hoc test for multiple comparisons (α = 0.05). Third, the unbiasedness of the predictions was quantified through a linear regression analysis. The EBV from the complete dataset were regressed onto the EBV from the imputed datasets and the datasets with missing values, respectively. The resulting regression coefficient was used as the indicator of unbiasedness to measure systematic bias [34].

Hardware configuration

All computational tasks in this study, including missing value imputation and genetic evaluation, were performed on a single computer. Its core specifications are as follows: processor: AMD Ryzen 9 9900X; memory: 64 GB of DDR5 RAM running at 6000 MHz.

Results

Performance of imputation methods at different missing rates

After the egg production time-series data were imputed, an ideal method is expected to yield low RMSE and MSE values, alongside a high R². The results (Table 3) indicated that, with the exception of PLR, which performed consistently across different missing rates, the performance of all other methods declined as the percentage of missing data increased. Among all methods, RF consistently demonstrated the best performance. Compared to FBM, PLR, NS, KNN, and LSTM, RF reduced the RMSE by 15.50% – 15.73%, 27.61% – 31.22%, 22.09% – 25.46%, 17.50% – 21.47%, and 17.14% – 18.56%, respectively. The MSE was reduced by 28.31% – 28.94%, 47.63% – 52.69%, 39.27% – 44.43%, 31.98% – 38.32%, and 31.40% – 33.75%, respectively. Correspondingly, the R² was improved by 7.42% – 8.54%, 21.24% – 23.41%, 14.31% – 15.88%, 8.79% – 13.53%, and 8.51% – 10.68% when compared to the same set of methods. For the datasets with a 20% artificial missing rate, one imputed dataset was randomly selected for each method to generate example scatter plots, which are presented in Fig. 5.

Table 3.

Evaluation metrics for different imputation methods under various missing rates^a

Imputation Method	Missing rate(%)	RMSE	MSE	R2
FBM	5	1.116 ± 0.003	1.246 ± 0.008	0.795 ± 0.002
	10	1.129 ± 0.002	1.275 ± 0.005	0.790 ± 0.001
	15	1.154 ± 0.002	1.331 ± 0.004	0.780 ± 0.001
	20	1.176 ± 0.002	1.382 ± 0.004	0.773 ± 0.001
PLR	5	1.371 ± 0.003	1.879 ± 0.009	0.692 ± 0.002
	10	1.365 ± 0.003	1.864 ± 0.009	0.692 ± 0.001
	15	1.367 ± 0.002	1.869 ± 0.006	0.692 ± 0.001
	20	1.369 ± 0.002	1.875 ± 0.005	0.692 ± 0.001
NS	5	1.265 ± 0.004	1.600 ± 0.009	0.737 ± 0.002
	10	1.266 ± 0.003	1.604 ± 0.008	0.735 ± 0.001
	15	1.268 ± 0.002	1.607 ± 0.005	0.735 ± 0.001
	20	1.272 ± 0.002	1.617 ± 0.004	0.734 ± 0.001
KNN	5	1.143 ± 0.004	1.307 ± 0.009	0.785 ± 0.002
	10	1.183 ± 0.004	1.400 ± 0.009	0.769 ± 0.001
	15	1.223 ± 0.003	1.495 ± 0.007	0.754 ± 0.001
	20	1.262 ± 0.003	1.592 ± 0.008	0.739 ± 0.001
RF	5	0.943 ± 0.003	0.889 ± 0.005	0.854 ± 0.001
	10	0.956 ± 0.003	0.914 ± 0.005	0.849 ± 0.001
	15	0.974 ± 0.002	0.948 ± 0.004	0.844 ± 0.001
	20	0.991 ± 0.002	0.982 ± 0.004	0.839 ± 0.001
LSTM	5	1.138 ± 0.005	1.296 ± 0.012	0.787 ± 0.002
	10	1.173 ± 0.007	1.376 ± 0.017	0.773 ± 0.003
	15	1.196 ± 0.004	1.431 ± 0.011	0.764 ± 0.002
	20	1.214 ± 0.006	1.476 ± 0.015	0.758 ± 0.003

Abbreviations: FBM Forward–Backward Mean Imputation, PLR Piecewise Linear Regression Imputation, NS Natural Splines Imputation, KNN K-Nearest Neighbors Imputation, RF Random Forest Imputation, LSTM Long Short-Term Memory

^aValues in the table are presented as mean ± se

Fig. 5.

Comparison of egg production curves between the complete dataset and the imputed dataset

Effect of different imputation methods on the accuracy of breeding value estimation

In this study, the accuracy of the different imputation methods was evaluated by calculating the correlation coefficients between the predicted EBV from each dataset and the corrected phenotypes from the complete dataset. For both Pearson and Spearman correlations, a value closer to one indicates higher EBV prediction accuracy. The results (Table 4) revealed that, for both evaluation models, prediction accuracy declined as the missing rate increased. However, compared with the datasets containing missing values, all imputation methods improved prediction accuracy to varying degrees. When genomic information was incorporated, ssGBLUP consistently produced higher prediction accuracy than PBLUP across the complete, missing, and imputed datasets. For the datasets with missing values, ssGBLUP improved accuracy by 28.49% to 31.28%, whereas for the imputed datasets, the improvement ranged from 32.57% to 33.80%. Across all missing rates, imputed datasets consistently achieved higher prediction accuracy than the unimputed dataset. Statistical analysis confirmed that these improvements were significant (p < 0.05). Among the imputation methods, RF generally produced the highest accuracy, followed closely by KNN, NS, and PLR. However, the differences among these methods were relatively small and not statistically significant (p > 0.05).

Table 4.

Comparison of prediction accuracy for different phenotypic datasets^a

Dataset	Missing rate(%)	PBLUP		ssGBLUP
		Pearson	Spearman	Pearson	Spearman
Unimputed	5	0.211 ± 0.003b	0.207 ± 0.002b	0.277 ± 0.002b	0.270 ± 0.003b
	10	0.204 ± 0.002b	0.201 ± 0.002b	0.265 ± 0.002b	0.258 ± 0.003b
	15	0.194 ± 0.003b	0.192 ± 0.003b	0.249 ± 0.003b	0.244 ± 0.003b
	20	0.186 ± 0.004b	0.185 ± 0.004b	0.239 ± 0.004b	0.235 ± 0.003b
FBM	5	0.220 ± 0.002a	0.215 ± 0.002a	0.293 ± 0.002a	0.286 ± 0.002a
	10	0.219 ± 0.002a	0.214 ± 0.002a	0.291 ± 0.002a	0.285 ± 0.002a
	15	0.218 ± 0.002a	0.214 ± 0.002a	0.289 ± 0.001a	0.283 ± 0.002a
	20	0.217 ± 0.002a	0.213 ± 0.003a	0.289 ± 0.002a	0.283 ± 0.002a
PLR	5	0.219 ± 0.002a	0.214 ± 0.002a	0.291 ± 0.002a	0.285 ± 0.002a
	10	0.219 ± 0.002a	0.215 ± 0.002a	0.291 ± 0.002a	0.284 ± 0.002a
	15	0.218 ± 0.003a	0.214 ± 0.003a	0.289 ± 0.001a	0.283 ± 0.002a
	20	0.216 ± 0.002a	0.212 ± 0.002a	0.288 ± 0.002a	0.282 ± 0.002a
NS	5	0.219 ± 0.002a	0.215 ± 0.002a	0.292 ± 0.002a	0.286 ± 0.002a
	10	0.219 ± 0.002a	0.215 ± 0.002a	0.291 ± 0.002a	0.285 ± 0.002a
	15	0.218 ± 0.003a	0.214 ± 0.003a	0.289 ± 0.002a	0.283 ± 0.002a
	20	0.216 ± 0.002a	0.212 ± 0.002a	0.289 ± 0.002a	0.283 ± 0.002a
KNN	5	0.219 ± 0.002a	0.214 ± 0.002a	0.293 ± 0.002a	0.286 ± 0.002a
	10	0.219 ± 0.002a	0.215 ± 0.002a	0.292 ± 0.002a	0.285 ± 0.002a
	15	0.219 ± 0.003a	0.214 ± 0.003a	0.291 ± 0.001a	0.284 ± 0.002a
	20	0.217 ± 0.002a	0.213 ± 0.002a	0.289 ± 0.002a	0.283 ± 0.002a
RF	5	0.220 ± 0.002a	0.215 ± 0.002a	0.293 ± 0.002a	0.286 ± 0.002a
	10	0.219 ± 0.002a	0.215 ± 0.002a	0.292 ± 0.002a	0.285 ± 0.002a
	15	0.219 ± 0.003a	0.215 ± 0.003a	0.291 ± 0.002a	0.285 ± 0.002a
	20	0.218 ± 0.002a	0.213 ± 0.002a	0.291 ± 0.002a	0.284 ± 0.002a
LSTM	5	0.219 ± 0.002a	0.214 ± 0.002a	0.292 ± 0.002a	0.285 ± 0.002a
	10	0.219 ± 0.002a	0.215 ± 0.002a	0.291 ± 0.002a	0.284 ± 0.002a
	15	0.218 ± 0.002a	0.214 ± 0.003a	0.290 ± 0.002a	0.283 ± 0.002a
	20	0.216 ± 0.002a	0.212 ± 0.003a	0.288 ± 0.002a	0.281 ± 0.002a

Values with different superscript letters (a, b) indicate statistically significant differences between groups (p < 0.05)

^aValues in the table are presented as mean ± se. Different superscript letters within the same column indicate significant differences (p < 0.05), based on Tukey’s HSD test

Based on Pearson correlations at missing rates from 5% to 20%, the EBV accuracy for the missing datasets using only pedigree information (PBLUP) ranged from 0.186 to 0.211. After applying the six imputation methods, the accuracies ranged from 0.216 to 0.220. Among them, RF consistently achieved the highest accuracy (0.218 – 0.220) across all missing rates, representing an improvement of 4.27% to 17.20% over the non-imputed scenarios. When genomic information was incorporated (ssGBLUP), the EBV accuracy for the missing datasets ranged from 0.239 to 0.277. The six imputation methods yielded accuracies between 0.288 and 0.293. RF again demonstrated the best and most robust performance (0.291 – 0.293), representing a substantial improvement of 5.78% to 21.76% over the non-imputed scenarios (p < 0.05 compared with unimputed data).

In terms of rank correlation (Spearman correlation), data missingness severely disrupted the ranking consistency. The accuracy for the non-imputed data was 0.185 – 0.207 with pedigree information only, and 0.235 – 0.270 with genomic data. As shown in Table 4, RF generally showed the highest accuracy among the imputation methods for both evaluation models, achieving accuracies of 0.213 – 0.215 (for PBLUP) and 0.284 – 0.286 (for ssGBLUP). Compared to the non-imputed scenarios, these represented improvements of 3.86% – 15.14% and 5.93% to 20.85%, respectively.

Effect of different imputation methods on the unbiasedness of breeding values

In this study, unbiasedness was measured by the coefficient of the regression of EBV from the complete dataset (benchmark EBV) on the predicted EBV. A regression coefficient closer to one indicates that less systematic bias was introduced by the imputation method, which in turn helps ensure the reliability of the genomic selection.

The evaluation results (Fig. 6) indicate that all imputation methods led to an underestimation of EBV variance to varying degrees. FBM achieved the best results in both genetic evaluation models, with regression coefficients ranging from 1.00 to 1.02 for PBLUP and 1.03 to 1.05 for ssGBLUP. Compared to the non-imputed scenarios, this represented a reduction in prediction bias of 4.59% – 20.16% and 5.51% – 21.64% for PBLUP and ssGBLUP, respectively. Notably, RF performed second best to FBM, with coefficients of 1.02 – 1.06 (PBLUP) and 1.04 – 1.09 (ssGBLUP). NS, KNN, and LSTM also demonstrated favorable performance in terms of unbiasedness, whereas PLR yielded the poorest results.

Fig. 6.

Comparison of unbiasedness for different phenotypic datasets. “Missing dataset refers to a dataset artificially generated with missing values from a complete dataset before applying any imputation methods”

Effect of imputation on heritability estimation

Missing phenotypic records led to a marked downward bias in heritability estimates relative to the complete dataset (h² = 0.246). In the datasets with missing values, heritability ranged from 0.109 to 0.192. After imputation, the estimates recovered to 0.215 – 0.233, depending on the method. Among all approaches, RF produced the estimates closest to the complete-data benchmark (0.222 – 0.233), indicating that it preserved additive genetic signal more effectively than the alternative methods. Detailed heritability estimates and their standard deviations are presented in Table 5.

Table 5.

Heritability estimates (h² ± se) across imputation methods and missing rates^a

Dataset	Missing rate(%)	VA	VE	VP	h2
Complete	/	36.332	111.120	147.452	0.246
Unimputed	5	30.396 ± 0.439	127.854 ± 0.463	158.249 ± 0.311	0.192 ± 0.003
	10	27.735 ± 0.501	140.678 ± 0.676	168.413 ± 0.423	0.165 ± 0.003
	15	22.825 ± 0.501	153.990 ± 0.592	176.815 ± 0.365	0.129 ± 0.003
	20	19.966 ± 0.590	162.610 ± 0.571	182.576 ± 0.555	0.109 ± 0.003
FBM	5	30.965 ± 0.359	103.649 ± 0.332	134.614 ± 0.208	0.230 ± 0.003
	10	28.270 ± 0.356	94.902 ± 0.342	123.173 ± 0.193	0.230 ± 0.003
	15	24.920 ± 0.318	87.182 ± 0.330	112.101 ± 0.204	0.222 ± 0.003
	20	21.677 ± 0.307	79.311 ± 0.293	100.988 ± 0.192	0.215 ± 0.003
PLR	5	34.673 ± 0.381	113.792 ± 0.367	148.465 ± 0.209	0.234 ± 0.003
	10	34.621 ± 0.418	116.024 ± 0.385	150.645 ± 0.220	0.230 ± 0.003
	15	33.690 ± 0.417	118.994 ± 0.395	152.684 ± 0.273	0.221 ± 0.003
	20	33.760 ± 0.432	122.315 ± 0.409	156.075 ± 0.223	0.216 ± 0.003
NS	5	33.327 ± 0.375	110.366 ± 0.349	143.693 ± 0.211	0.232 ± 0.003
	10	32.386 ± 0.405	108.330 ± 0.377	140.716 ± 0.219	0.230 ± 0.003
	15	30.302 ± 0.373	106.370 ± 0.376	136.672 ± 0.253	0.222 ± 0.003
	20	28.579 ± 0.382	104.408 ± 0.389	132.987 ± 0.267	0.215 ± 0.003
KNN	5	33.155 ± 0.376	109.554 ± 0.360	142.709 ± 0.215	0.232 ± 0.003
	10	31.535 ± 0.388	105.223 ± 0.354	136.758 ± 0.211	0.231 ± 0.003
	15	29.019 ± 0.363	100.249 ± 0.363	129.268 ± 0.216	0.224 ± 0.003
	20	26.062 ± 0.346	94.174 ± 0.366	120.236 ± 0.246	0.217 ± 0.003
RF	5	33.886 ± 0.383	111.334 ± 0.357	145.220 ± 0.211	0.233 ± 0.003
	10	33.376 ± 0.386	110.353 ± 0.357	143.729 ± 0.208	0.232 ± 0.003
	15	32.203 ± 0.390	109.901 ± 0.371	142.104 ± 0.228	0.227 ± 0.003
	20	31.308 ± 0.403	109.534 ± 0.399	140.842 ± 0.203	0.222 ± 0.003
LSTM	5	32.939 ± 0.376	109.111 ± 0.346	142.050 ± 0.207	0.232 ± 0.003
	10	31.323 ± 0.376	105.344 ± 0.343	136.667 ± 0.219	0.229 ± 0.003
	15	29.288 ± 0.371	102.549 ± 0.384	131.838 ± 0.244	0.222 ± 0.003
	20	27.319 ± 0.361	99.533 ± 0.340	126.852 ± 0.277	0.215 ± 0.003

V_A Additive variance. V_E Residual variance. V_P Phenotypic variance. h² Heritability

^aValues in the table are presented as mean ± se

Computational time

Computational time is explicitly defined as the wall-clock time required to process one missing dataset under the fixed parameter settings listed in Table 2, from model initialization to completed imputation output. All imputation procedures were executed on the same CPU-based hardware platform to ensure a fair comparison. Importantly, no separate hyperparameter optimization stage was performed in this study; consequently, the reported times solely reflect the imputation process itself, without any additional tuning overhead.

Among the evaluated methods, traditional and statistical model-based approaches required substantially less time. Specifically, FBM exhibited the lowest average time consumption (0.18 – 0.35 s), while PLR and NS both completed imputation within 2 s. In contrast, machine-learning-based methods generally required longer runtimes. LSTM showed the highest computational cost, ranging from 192.1 to 300.4 s across missing-rate scenarios. Compared with LSTM, RF required considerably less time under high missing rates, consuming approximately 30% of the time needed by LSTM, while KNN required 12.3 – 42.4 s. A comparison of the computational time required for the different imputation methods is shown in Fig. 7.

Fig. 7.

Comparison of computational time required by different imputation methods

Discussion

This study provides a systematic evaluation of imputation strategies for longitudinal egg production data and highlights that improvements in imputation accuracy do not necessarily translate proportionally into gains in genomic prediction accuracy. Our study establishes RF as the optimal methodology for handling missing values in longitudinal egg production data, demonstrating superior performance metrics (R² reduction of merely 1.76% at 20% missing rate) and substantial EBV accuracy improvement (up to 21.75%) compared to conventional approaches. The algorithm’s robustness spans across practical missing-data scenarios (5% – 20% missing rates), outperforming five alternative imputation methods through its unique capacity to model complex biological interactions. These findings are consistent with those of You et al. [7], who also concluded that Random Forest was the optimal model for imputing phenotypic data in dairy cattle. Similarly, Bumanis et al. [35] achieved excellent predictive performance for egg yield using a Random Forest model, demonstrating that machine learning models leveraging multiple information sources outperform non-linear compartmental models that rely on a single source. Our approach builds upon and extends previous genomic evaluations of this population [36]. While those studies provided foundational insights using multi-trait models for static phenotypic records, the present work pivots toward capturing the dynamic, longitudinal complexity of egg production. By systematically investigating how missing data patterns in time-series datasets influence the trajectory of genomic merit, we address a critical dimension of breeding evaluation that remains inaccessible to static modeling frameworks.

RF consistently demonstrated superior performance in modeling egg-laying dynamics due to its ability to capture complex, nonlinear interactions among genotype, environment, and management factors that influence laying performance, as evidenced by its high accuracy in predicting daily oviposition events and identifying key predictors in both physiological and environmental domains [37–41]. Unlike methods such as LSTM, RF excels at feature selection and variable importance analysis, enabling the identification of critical biological and environmental determinants of egg production, including age, temperature, and gut microbiota composition [39–41]. Furthermore, RF’s iterative modeling approach aligns with the biological progression of laying birds, effectively mirroring the sequential and cumulative nature of physiological changes during key production phases such as sexual maturation and peak laying, and providing robust insights into genotype–phenotype associations and regulatory mechanisms underlying traits like eggshell strength [37, 38]. This biological alignment makes RF a powerful and physiologically coherent tool for imputation and prediction in poultry science. We demonstrate that high imputation accuracy at the phenotypic level does not linearly translate to the reliability of genomic predictions. This suggests that some imputation methods may recover numerical values but fail to preserve the underlying covariance structures and genetic variance necessary for accurate genomic evaluation. Importantly, this also explains why RF is more effective at preserving the underlying genetic signal, rather than merely improving numerical imputation accuracy.

In contrast, the other methods had their respective strengths and weaknesses. FBM was simple and fast, providing a local estimate using the mean of adjacent points. While suitable for sequences with strong auto-correlation, it tends to underestimate variability and ignore inter-variable relationships; as the missing rate rose from 5% to 20%, its R2 dropped by 2.77%. PLR [24] fits different stages of the egg production cycle with multiple linear segments, making it more adaptive than a single linear model, and its imputation accuracy did not fluctuate with increasing missing rates. NS [26] was more refined, combining spline fitting to account for both the overall non-linear trend and individual heterogeneity; its performance fluctuated by only 0.41% as the missing rate increased to 20%. KNN [28] fills missing values with a weighted average of the K most similar neighbors, leveraging local similarity. However, its distance metric can become less effective in high-dimensional spaces, and it exhibited the largest drop in R2, falling by 5.86%. Finally, the core ‘memory cells’ of LSTM [5] are adept at capturing long-term dependencies in time series, but it was less flexible and effective than RF in modeling the complex interactions of synchronous, heterogeneous features, resulting in a 3.69% decrease in R2.

The missingness levels evaluated in this study (5%, 10%, 15%, and 20%) were selected to represent a range of mild to moderately severe missing-data scenarios, enabling systematic evaluation of the robustness of different imputation methods under increasing levels of information loss. This range reflects standard commercial conditions in poultry production (typically below 15%) while including a 20% threshold as a stress test for algorithm robustness. In commercial settings, egg production data are recorded as longitudinal time-series and are often subject to missing observations due to mortality, recording errors, and equipment limitations. Our results demonstrate that the performance differences among imputation methods become more pronounced as the missing rate increases, highlighting the importance of selecting robust methods for practical breeding applications where moderate missing rates are not uncommon.

Although Transformer-based models have shown strong performance in time series and industrial data imputation tasks, they typically require large datasets for effective training and may be prone to overfitting when applied to small-scale datasets such as those found in egg production scenarios [42–44]. The challenge of varying levels of missing data further complicates the imputation process, and highly flexible models such as Transformers may struggle to achieve stable convergence and reliable results when ground-truth data is sparse [42, 44]. In contrast, tree-based methods such as random forest imputation are generally less demanding in terms of data volume and computational resources, and their inductive bias is often better suited for structured tabular data, making them a pragmatic choice for small-scale, structured datasets [45]. In this study, machine learning models were implemented using fixed parameter settings rather than extensive hyperparameter optimization. This choice was made to ensure a fair comparison across imputation methods under identical conditions. Given the repeated simulation design and dataset scale, exhaustive parameter tuning would substantially increase computational cost while introducing additional variability. Therefore, fixed parameters were adopted to provide a stable and reproducible evaluation framework.

Beyond the primary findings, our results reveal a critical relationship between phenotypic time-series completeness and heritability estimation. Missing records introduced a noticeable downward bias in heritability estimates, indicating that incomplete phenotypes weakened the recoverable additive genetic signal. After imputation, heritability partially recovered, though the extent of recovery varied among methods. Notably, Random Forest yielded both the highest prediction accuracy and heritability estimates closest to the complete-data benchmark, suggesting that it better preserves the covariance structure underlying genetic evaluation. These findings underscore that, for breeding applications, the evaluation of imputation methods should extend beyond numerical accuracy to explicitly consider their impact on genetic parameter estimation and the preservation of the additive genetic signal.

Conclusion

This study evaluated the efficacy of six imputation methods for handling missing values in egg production time-series data. The results showed that Random Forest provided the most robust overall performance, achieving the best balance between accurate phenotypic reconstruction and improved downstream breeding-value prediction. These findings support the use of RF imputation as an effective preprocessing strategy for longitudinal phenotypes in poultry breeding.

Abbreviations

AI-REML

Average Information-Restricted Maximum Likelihood

BLUP

Best Linear Unbiased Prediction

EBV

Estimated Breeding Value

FBM

Forward–Backward Mean Imputation

GEBV

Genomic Estimated Breeding Value

HWE

Hardy–Weinberg Equilibrium

KNN

K-Nearest Neighbors Imputation

LSTM

Long Short-Term Memory

MAF

Minor Allele Frequency

MCAR

Missing Completely At Random

MSE

Mean Squared Error

NS

Natural Splines Imputation

OOB

Out-of-Bag

PBLUP

Pedigree-based Best Linear Unbiased Prediction

PLR

Piecewise Linear Regression Imputation

QC

Quality Control

R²

Coefficient of Determination

RF

Random Forest Imputation

RMSE

Root Mean Squared Error

RNN

Recurrent Neural Network

SNP

Single Nucleotide Polymorphism

ssGBLUP

Single-step Genomic Best Linear Unbiased Prediction

Authors’ contributions

T.L. and H.Q. conceived and supervised the study. J.L., S.J., J.M., J.L. and P.W. conducted experiments and analyzed data. J.L. and T.L. wrote the manuscript. J.L., D.S., and H.Q. revised the content. All authors read and approved the final version.

Funding

This work was supported by the STI2030-Major Projects (2023ZD04064), the project of State Key Laboratory of Swine and Poultry Breeding Industry (2023QZ-NK07, GDNKY-ZQQZ-K10), and the earmarked fund for CARS-41.

Data availability

The code and phenotypic data supporting the findings of this study are openly accessible at https://github.com/JOYCTUI/Egg-Production-Time-Series-Interpolation, while the corresponding genotype data can be obtained from the corresponding author upon reasonable request.

Declarations

Ethics approval and consent to participate

This study utilized historical commercial breeding records with no animal handling, intervention, or experimental procedures; thus, formal ethical review was not required per established standards.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.