Fast3VmrMLM: A fast algorithm that integrates genome-wide scanning with machine learning to accelerate gene mining and breeding by design for polygenic traits in large-scale GWAS datasets

Abstract

Genetic dissection and breeding by design for polygenic traits remain substantial challenges. To address these challenges, it is important to identify as many genes as possible, including key regulatory genes. Here, we developed a genome-wide scanning plus machine learning framework, integrated with advanced computational techniques, to propose a novel algorithm named Fast3VmrMLM. This algorithm aims to enhance the identification of abundant and key genes for polygenic traits in the era of big data and artificial intelligence. The algorithm was extended to identify haplotype (Fast3VmrMLM-Hap) and molecular (Fast3VmrMLM-mQTL) variants. In simulation studies, Fast3VmrMLM outperformed existing methods in detecting dominant, small, and rare variants, requiring only 3.30 and 5.43 h (20 threads) to analyze the 18K rice and UK Biobank-scale datasets, respectively. Fast3VmrMLM identified more known (211) and candidate (384) genes for 14 traits in the 18K rice dataset than FarmCPU (100 known genes). Additionally, it identified 26 known and 24 candidate genes for seven yield-related traits in a maize NC II design; Fast3VmrMLM-mQTL identified two known soybean genes near structural variants. We demonstrated that this novel two-step framework outperformed genome-wide scanning alone. In breeding by design, a genetic network constructed via machine learning using all known and candidate genes identified in this study revealed 21 key genes associated with rice yield-related traits. All associated markers yielded high prediction accuracies in rice (0.7443) and maize (0.8492), enabling the development of superior hybrid combinations. A new breeding-by-design strategy based on the identified key genes was also proposed. This study provides an effective method for gene mining and breeding by design.

This study presents Fast3VmrMLM, a novel algorithm that integrates genome-wide scanning with machine learning and computational techniques to accelerate gene mining and breeding by design for polygenic traits using large-scale SNP-, haplotype-, and molecular-based GWAS datasets. The software runs efficiently on a Linux server with 30 threads and 1 TB of memory. Applying this approach to rice, the study identifies 21 key genes associated with yield-related traits through a genetic network constructed from both known and candidate genes, thereby proposing a new strategy for breeding by design based on these key genetic targets.

Introduction

Polygenic traits contribute to many critical breeding and production traits in crops, including resistance to rice blast, wheat Fusarium head blight, Sclerotinia in oilseed rape, and cotton Fusarium wilt. For these traits, gene identification remains difficult in genetic studies (Purugganan and Jackson, 2021), and phenotype improvement through the alteration of only a few genes is often ineffective because these traits are controlled by numerous polygenes (Yi and Xu, 1999; Kroymann and Mitchell-Olds, 2005). In particular, during the post-Green Revolution era, many key genes have already been utilized(Kumar et al., 2023), but effective resistant and high-yielding varieties have not been developed for most highly destructive diseases.

The genetic dissection and breeding of polygenic traits remain major challenges (Purugganan and Jackson, 2021; Lappalainen et al., 2024). In early efforts to mine genes, marker analysis was proposed; genome scanning is now widely used in quantitative trait locus (QTL) mapping (Lander and Botstein, 1989), genome-wide association studies (GWAS) (Risch and Merikangas, 1996), and bulked segregant analysis (Schneeberger et al., 2009). These methods often focus on strong signals from genes with large effects, substantially reducing the power to detect genes with smaller effects (Wang et al., 2016; Wen et al., 2018). In GWAS, commonly used approaches mainly consider the allelic substitution effect rather than incorporating both additive and dominant effects. Additionally, they control for the polygenic background of the allelic substitution effect but not for the complete additive and dominant polygenic backgrounds, further reducing gene detection power (Li et al., 2022a). To address this issue, Li et al. (2022a) established a compressed variance component mixed model, termed 3VmrMLM. However, the increasing availability of large genomic and phenomic datasets poses additional challenges for statistical computation. In crop breeding by design for polygenic traits, more than a century of recurrent selection has been validated in maize and soybean (Werner and Wilcox, 1990; Lambert, 1994; Dudley and Lambert, 2010). Nevertheless, recurrent selection requires long time, and the absence of known genes contributes to uncertainty in crop improvement efforts. Recently, Wang et al. (2019) sequenced the genome of Tetep, a rice cultivar with broad-spectrum resistance to blast, cloned and tested a set of NLR genes, and designed an extensive set of molecular markers to rapidly introduce clustered and paired NLRs from the Tetep genome into new resistant cultivars. This work underscores the genetic foundation for gene identification in the improvement of polygenic traits.

To support gene mining and crop breeding by design for polygenic traits, we proposed a fast, efficient, and large-scale GWAS method, Fast3VmrMLM, to identify more genes through genetic analysis, then constructed a gene network for rice yield-related traits to determine key genes for breeding. In Fast3VmrMLM, the conventional genome-wide scanning step for identifying significantly associated markers (SAMs) is replaced by a two-step approach. The first step involves genome-wide scanning combined with fast and large-scale data algorithms (Loh et al., 2015; Zhou et al., 2018) to select potentially associated markers. The second step uses machine learning to identify SAMs. Additionally, Fast3VmrMLM has been extended to detect haplotype (Fast3VmrMLM-Hap) and molecular (Fast3VmrMLM-mQTL) variants. In the construction of the genetic network, machine learning was utilized to capture complex genetic relationships by identifying quantitative trait nucleotide (QTN)-by-QTN interactions. This new framework was validated through a series of simulation studies and analyses of real datasets in rice and maize.

Results

Strategies for genetic dissection and breeding by design of polygenic traits

Polygenic traits are crucial in crop breeding and production. However, challenges persist in identifying their underlying genes and enhancing phenotypic values. In gene mining, commonly used methods rely on genome-wide scanning and focus on only the strongest signals, which may overlook important loci and fail to consider the complexity of polygenic traits. To address these limitations, we developed a novel framework and proposed a fast, efficient, and large-scale algorithm named Fast3VmrMLM. In this method, genome-wide scanning is first used to select potentially associated markers using a relatively loose threshold, rather than to identify SAMs. Machine learning is subsequently applied to select markers with non-zero effects, and likelihood ratio testing is used to identify SAMs (Figure 1). Specifically, a full model based on 3VmrMLM (Li et al., 2022a)—which incorporates additive and dominant effects, along with their respective polygenic backgrounds—is combined with the new genome-wide scanning and machine learning framework to enhance the power of gene detection. Nonetheless, the algorithm continues to face computational challenges in the era of big data and artificial intelligence (AI).

Figure 1.

Strategies for genetic dissection and crop breeding of polygenic traits.

In genetic dissection, genome-wide scanning is used to select potentially associated markers rather than to identify SAMs. These selected markers are subsequently evaluated to identify SAMs for polygenic traits using machine learning and a likelihood ratio test. In crop breeding, all QTNs with known or candidate genes in one or multiple datasets are used to identify QTN-by-QTN interactions (QQIs) for each polygenic trait via machine learning. These QQIs for a given polygenic trait are used to construct a genetic network to identify key genes. All significant loci are also used to perform genomic selection and to predict superior hybrid combinations. Figures depicting rice and maize plants were created using BioGDP.com (Jiang et al., 2025).

To achieve fast computing on large genomic datasets, the five variance components in the full model described above were compressed into three (Li et al., 2022a ). The preconditioned conjugate gradient (PCG) method (Kaasschieter, 1988) was used to replace the decomposition and inversion of large matrices in the mixed linear model solution and machine learning process. Marker effect estimation and the corresponding Wald test in 3VmrMLM were replaced by a vectorized Wald test (Li et al., 2024), based on the conditional expectation method (Henderson, 1975; Xu et al., 2014) and the GRAMMAR-gamma approximation (Svishcheva et al., 2012), enabling dimensionality reduction of computational complexity in variance-covariance estimation.

To accommodate large-scale datasets with sample sizes in the tens of thousands, the kinship matrix was first integrated into the PCG iterations (Kang et al., 2008; Loh et al., 2015) to avoid storing an n × n matrix in memory. The Woodbury matrix equation (Golub and van Loan, 1996) was then used to reduce the dimensionality of the large matrix in the machine learning process. Additionally, a binary data structure was utilized to store the genotypic data.

To identify key genes for polygenic traits, all QTNs closely linked to known or candidate genes (Wei et al., 2021) were used to detect QTN-by-QTN interactions via a machine learning approach. These interactions, obtained across multiple datasets, were used to construct a genetic network. Hub genes in the resulting network were considered key genes for polygenic traits.

The above strategy is summarized in Figure 1.

Monte Carlo simulation studies

Comparison of four mixed models using the Fast3VmrMLM method

To evaluate the advantage of the compressed variance component mixed model used in Fast3VmrMLM, four models (Figure 2A) were compared in Monte Carlo simulation studies under four scenarios: no polygenic background (I-1), additive-dominant background (I-2), epistatic background (I-3), and additive-dominant-epistatic background (I-4) (Supplemental Table 1). Statistical power and false-positive rates (FPRs) across different p value thresholds (from 1.0e−39 to 9.0e−3) were used to generate receiver operating characteristic curves. The novel model implemented in Fast3VmrMLM exhibited the largest area under the curve (Figure 2A), demonstrating superior performance compared with the other models.

Figure 2.

Comparison of different models and methods in Monte Carlo simulation studies.

(A) Comparison of the average power (y axis) of four mixed linear models at different observed FPRs (x axis) in simulation datasets I-1 to I-4 (1000 replicates).

(B) Comparison of the average power (y axis) of the Fast3VmrMLM with existing methods (3VmrMLM, FarmCPU, and EMMAX) at different p value thresholds (x axis) (100 replicates).

(C) Effect of genetic modes and minor allele frequencies (x axis) on average power (y axis) for new and existing methods (100 replicates).

(D) Running times (y axis) of Fast3VmrMLM, FarmCPU, and EMMAX with different sample sizes (x axis).

Performance comparison of Fast3VmrMLM, FarmCPU, and EMMAX

To confirm the advantage of the new method (Fast3VmrMLM) over existing methods (3VmrMLM, FarmCPU, and EMMAX), simulation datasets I-1 to I-4 were used to evaluate the four approaches. The average power under the Bonferroni correction threshold (p < 5.00e−7) for all 10 simulated QTNs was 92.12%, 97.00%, 46.20%, and 36.00% for Fast3VmrMLM, 3VmrMLM, FarmCPU, and EMMAX, respectively (Supplemental Table 2; Figure 2B). The first two methods exhibited significantly higher power than the latter two, primarily due to the lower power of FarmCPU and EMMAX in identifying the fifth to eighth QTNs with α = 0 (Supplemental Tables 1 and 2). The average FPRs for the above four methods were 0.02‱, 0.20‱, 0.04‱, and 0.00‱, respectively, indicating that Fast3VmrMLM offers a superior balance between high power and low FPR (Supplemental Table 3). Additionally, Fast3VmrMLM yielded lower mean squared error (MSE) and mean absolute deviation (MAD) for QTN effect estimates compared with existing methods (Supplemental Tables 4 and 5).

To further validate these findings using real marker genotypic datasets, simulated phenotypic datasets II-1 to II-4, based on the true parameters listed in Supplemental Table 6, were analyzed with the same four methods. The average power under the Bonferroni correction threshold was 78.94%, 76.81%, 60.79%, and 31.08%, respectively (Supplemental Table 7). The average FPRs were 0.53‱, 1.34‱, 0.54‱, and 2.98‱, respectively, with similar trends observed for false discovery rate (FDR), false negative rate (FNR), and F₁ score (Supplemental Table 8). Fast3VmrMLM also showed lower MSE and MAD for both QTN effect estimates and positional estimates compared with the other methods (Supplemental Tables 9 and 12). These results support a similar conclusion.

Comparison of Fast3VmrMLM with existing methods under different genetic modes and allelic frequency combinations

To investigate the effect of genetic mode and allelic frequency combinations on the performance of the new and existing methods, nine scenarios were considered in simulation dataset III, comprising three genetic modes (additive, dominant, and additive-dominant) and three allelic frequencies (0.50, 0.25, 0.05) (Supplemental Text 1). Fast3VmrMLM showed consistent average power (64.89%–66.58%) across the three genetic models, while FarmCPU and EMMAX exhibited varying average powers in the additive (67.06% and 54.47%), dominant (25.56% and 19.28%), and additive-dominant (48.58% and 35.53%) models (Supplemental Table 13; Figure 2C). The reduced power of FarmCPU and EMMAX resulted from their poor ability to detect dominant QTNs with allelic frequencies of 0.25 (≤8.50%) and 0.50 (≈0.00%). This is theoretically consistent because α = a + d (q–p), where a and d represent the additive and dominant effects of a QTN, respectively, and q and p denote the allelic frequencies of alleles A and a, respectively. Additionally, all three methods exhibited low false positive rates (FPRs) and MSEs (Supplemental Tables 14 and 15).

To further assess type I error rates for the new and existing methods, simulation dataset IV, which contained no QTNs, was analyzed. Association tests were conducted for 10⁸ genetic variants (Supplemental Text 1; Kang et al., 2008). Type I error rates under the Bonferroni correction threshold (p < 5.00e−7) were 4.80e−7, 1.90e−7, and 1.0e−8 for Fast3VmrMLM, FarmCPU, and EMMAX, respectively, indicating no inflation of type I error in Fast3VmrMLM or the existing methods.

Comparison of running times and memory consumption for Fast3VmrMLM and existing methods across different large-scale sample sizes

To evaluate the performance of these GWAS methods on large-scale datasets, the new and existing methods were compared across five sample size levels: 4000 (V-1), 8000 (V-2), 12 000 (V-3), 16 000 (V-4), and 20 000 (V-5), each with 100 000 SNPs. The average running time over five replicates ranged from 44.57 s to 289.19 s for Fast3VmrMLM, from 39.76 s to 546.88 s for FarmCPU, and from 322.73 s to 14 370.00 s for EMMAX in the five simulated datasets (Supplemental Table 16; Figure 2D). These results indicate that Fast3VmrMLM is faster than FarmCPU, and both are considerably faster than EMMAX. Memory consumption ranged from 0.70 to 1.28 GB for Fast3VmrMLM, from 1.73 to 25.76 GB for FarmCPU, and from 1.22 to 26.16 GB for EMMAX across the five datasets, indicating lower memory requirements for Fast3VmrMLM compared with the other two methods. Furthermore, Fast3VmrMLM (6.53 min) ran significantly faster than FarmCPU (75.67 min) on a dataset comprising 40 000 individuals and 100 000 SNPs (20 replicates).

To assess the performance of Fast3VmrMLM on a UK Biobank-scale dataset, the method was applied to a simulated dataset of 500 000 individuals, each with one million SNPs. The runtime and memory consumption were 5.43 h and 120.29 GB, respectively (Supplemental Table 16). Notably, Fast3VmrMLM successfully identified QTNs with frequencies as low as 0.03% (Supplemental Table 17), highlighting its potential utility for Biobank-scale analyses. All simulation studies were conducted on a server with 1.0 TB of memory and 20 CPUs. We observed that both memory usage and runtime linearly increased with the number of markers, rather than the number of individuals.

Comparison of Fast3VmrMLM-Hap with existing methods in identifying rare haplotypic variants

In simulated dataset VI, haplotypes were used to simulate functional elements in real rice genomics (Supplemental Text 1). Among the 10 simulated causal variants, four were rare (minor allele frequency <5%; Supplemental Table 18). The power to detect all 10 variants was highest for 3VmrMLM (81.30%), followed by Fast3VmrMLM-Hap (76.20%), FarmCPU (74.60%), Fast3VmrMLM (66.10%), and EMMAX (56.70%) (Supplemental Table 18). When specifically focusing on the four rare variants, 3VmrMLM-Hap exhibited the highest power (92.25%), followed by 3VmrMLM (83.00%), FarmCPU (69.50%), Fast3VmrMLM (56.50%), and EMMAX (50.50%) (Supplemental Table 18), indicating the superior performance of Fast3VmrMLM-Hap in detecting rare variants.

Performance of Fast3VmrMLM-mQTL in identifying mQTLs

To evaluate the performance of Fast3VmrMLM-mQTL in detecting mQTLs, the Monte Carlo simulation dataset VII (1000 individuals, each with 33 341 structural variants [SVs]; 10 comprised causal SVs with heritabilities ranging from 5% to 10%; 100 replicates) was analyzed (Supplemental Table 19). The results showed a power of 74.8% and a false positive rate of 2.0281‱ (Supplemental Table 19), demonstrating the method’s effectiveness in mQTL detection.

Re-analysis of the 18K rice dataset

Gene mining for 14 traits in the 18K rice population

Fourteen traits across three environments in the large-scale 18K rice dataset of Wei et al. (2024) were re-analyzed using Fas3VmrMLM, and the results were compared with those obtained via FarmCPU (Wei et al., 2024). Fast3VmrMLM identified 1555 SAMs at the Bonferroni correction threshold (1.71e−8), compared with 1117 identified by FarmCPU (Supplemental Tables 20–33) (Wei et al., 2024). Of the 1555 SAMs identified by Fast3VmrMLM, 1222 overlapped with those detected by FarmCPU SAMs; 333 were uniquely identified by Fast3VmrMLM.

Within ±200-kb windows surrounding all SAMs identified above, we mined known genes supported by transgenic experimental evidence from http://www.ricedata.cn/. The results indicated that 100, 211, and 215 known genes were identified by FarmCPU, Fast3VmrMLM, and Fast3VmrMLM-Hap, respectively (Supplemental Tables 34–47; Figures 3 and 4A; Supplemental Figures 1–5). Among the 359 known genes identified in this study, 62 were detected jointly by the new methods (Fast3VmrMLM and Fast3VmrMLM-Hap) and FarmCPU; 259 were detected exclusively by the new methods (Figure 4A). Of the 70 known genes identified in at least two environments, Fast3VmrMLM and Fast3VmrMLM-Hap identified 39 and 41, respectively, compared with 17 identified by FarmCPU. Eleven were jointly detected by the new and FarmCPU methods, whereas 53 were identified only by the two new methods (Supplemental Figure 6A). In summary, the new methods identified a greater number of known genes, with higher repeatability, than FarmCPU.

Figure 3.

Manhattan plots of heading date in the 18K rice dataset.

The x axis represents the chromosomal and physical coordinates of the markers; the left y axis shows the negative logarithmic transformation of p values obtained from genome-wide scanning, and the right y axis displays LOD scores obtained from the likelihood ratio test using the new methods. Known genes located within ±200-kb windows of significant QTNs (p value <1.71e−8, indicated by dashed line) identified by new and FarmCPU methods, are marked in dark red; those identified by the new method in more than two environments are marked in blue, and those identified by FarmCPU alone are marked in gray. Shanghai, Hangzhou, and Hainan represent the three environments. SNP, Fast3VmrMLM; Hap, Fast3VmrMLM-Hap. If $-{\log }_{10}p\ge 40$, the transformation $-\log p^{\prime }=\left(-\log p-40\right)/1000+40$ was performed. Variants with LOD scores greater than 20 were transformed using $\text{LO}{\mathrm{D}}^{\prime }=\left(\text{LOD}-20\right)/50+20$.

Figure 4.

Known, candidate, and key genes for yield-related traits in rice.

(A) Number of known genes identified by Fast3VmrMLM (light green), Fast3VmrMLM-Hap (blue), and FarmCPU (light pink) in the 18K rice dataset.

(B) Frequency distributions of the allele substitution effect (α, axis) of SAMs identified by Fast3VmrMLM (light green) and FarmCPU (light pink) in the 18K rice dataset. Annular charts indicate the proportion of SAMs detected exclusively by Fast3VmrMLM, and those detected by both Fast3VmrMLM and FarmCPU, for the first (left) and second (right) groups.

(C) Subset network showing genes with degree values greater than five in the 18K and 1439 rice datasets.

(D) Evidence for hub gene pathways reported in previous studies.

To identify candidate genes, all genes within ±200-kb windows of the SAMs were subjected to gene differential expression analysis, Gene Ontology (GO) annotation analysis (http://systemsbiology.cau.edu.cn/-agriGOv2/), and haplotype analyses (Supplemental Text 3). In total, 384 and 367 candidate genes were identified by Fast3VmrMLM (Supplemental Table 48) and Fast3VmrMLM-Hap, respectively (Supplemental Table 49). Among them, 184 were shared by both methods, while 200 and 183 were uniquely identified by Fast3VmrMLM and Fast3VmrMLM-Hap, respectively (Supplemental Figure 6B).

Comparison of the new and FarmCPU methods

Among the 333 SAMs exclusively detected by Fast3VmrMLM, 35 had large absolute dominance ratios (|d/a| ≥ 5.0), and 28 exhibited dominant variance ratios greater than 50%. Of the 212 SAMs associated with known genes that were identified by Fast3VmrMLM alone but not by FarmCPU (Supplemental Table 50), 17 had |d/a| ≥ 5.0 and 13 had dominant variance ratios ≥50% (Supplemental Table 51). These findings demonstrate the advantage of Fast3VmrMLM in detecting SAMs with dominant effects.

In total, 71 of the 333 SAMs had small standardized allelic substitution effects (α) (|α| ≤ 0.05), and two (r² > 1%) exhibited large genetic effects (Supplemental Tables 20–33; Figure 4B). Among the 181 SAMs with known genes in Supplemental Table 50 that had small α (|α| < 0.5), 32 showed large genetic effects (r² > 0.5%; Supplemental Table 52). These results highlight the advantage of Fast3VmrMLM in identifying SAMs with small allelic substitution effects.

Of the 1692 variants identified by Fast3VmrMLM-Hap (Supplemental Tables 53–66), 123 were jointly identified by Fast3VmrMLM and/or FarmCPU, whereas 1569 were detected exclusively by Fast3VmrMLM-Hap. Among the 1569 variants, 685 were rare (minor allele frequency <2%). Fast3VmrMLM-Hap identified more rare variants (785/1692; 46.39%) than Fast3VmrMLM (61/1555; 3.92%) and FarmCPU (2/908; 0.22%) (Supplemental Tables 34–47). Of the 310 variants associated with known genes and detected only by Fast3VmrMLM-Hap, 86 were rare (Supplemental Table 67). These results demonstrate the advantage of Fast3VmrMLM-Hap in identifying rare variants.

Key genes of yield-related traits for rice breeding by design

To identify key genes associated with yield-related traits for rice breeding by design, a QTN-by-QTN interaction (QQI) network was constructed for nine yield-related traits using the 18K and 1439 (Huang et al., 2015) rice datasets, as described in Wei et al. (2024) (Figure 4C; Supplemental Figure 7). In the 18K rice dataset, 399 known or candidate genes were found by Fast3VmrMLM to be located near 371 QTNs (Wei et al., 2021) associated with heading date (HD), plant height (PH), panicle number (PN), grain length (GL), grain yield (GY), and grain width (GW) (Supplemental Tables 34, 35, 38, and 42–44). In the 1439 rice dataset (Huang et al., 2015), 297 known genes were found near 271 QTNs associated with HD, PH, PN, GY, grain number (GN), seed setting rate (SSR), and 1000-grain weight (TGW) (Supplemental Tables 68–74; Supplemental Text 2). Significant QQIs for each trait in each dataset were identified via empirical Bayes at the 0.01 probability level (Wei et al., 2024), and all significant QQIs were used to construct the QQI network. Overall, 527 QQIs involving 388 gene pairs were identified as significant (Supplemental Table 75) and used to build the QQI network. Twenty-one hub genes, each involved in at least seven epistatic pairs, were identified. These hub genes were involved in 261 (49.53%) of all epistatic interactions. Notably, six hub genes—Ghd8, OsSOC1, Ghd7.1, sd1, OsFTL1, and OsDEP1—were consistent with those reported in Wei et al. (2024).

All 21 hub genes—Ghd8 (Yan et al., 2011), OsSOC1 (Lee et al., 2004), Ghd7.1 (Yan et al., 2013), sd1 (Sasaki et al., 2002), TAD1 (Xu et al., 2012), LYL1 (Li et al., 2019), and others (Supplemental Table 75)—have been reported as key genes in yield-related pathways. For example, our network identified epistatic interactions between Ghd8 and Ghd7.1, as well as Hd1, where these genes co-regulate heading date (HD) and play a major role in determining the yield potential of rice cultivars (Zhang et al., 2019) (Figure 4D). Additional supporting evidence from our network is presented in Supplemental Table 76 and Supplemental Text 2, demonstrating the effectiveness of Fast3VmrMLM in identifying key genes associated with polygenic traits.

In addition, the prediction accuracy of phenotypes in genomic selection, based on all SAMs identified by Fast3VmrMLM, was evaluated using five-fold cross-validation. The results showed prediction accuracies for the nine traits ranging from 0.6900 to 0.9185 (Supplemental Figure 8).

Analysis of maize NC II mating design datasets

Gene mining for seven traits in two maize NC II mating design populations

Three best linear unbiased prediction (BLUP) phenotypic datasets for seven yield-related traits from 2019, 2022, and the combined dataset were analyzed in association with 13 528 478 SNPs using Fast3VmrMLM, Fast3VmrMLM-Hap, and 3VmrMLM. The analyzed traits included PH, ear height (EH), seed weight (SW), ear thickness (ET), kernel row number (KRN), kernel number per row (KNR), and yield per plot (Yield). On average, Fast3VmrMLM identified 59.81 SAMs and 2.81 suggested associated markers per trait per year (Supplemental Tables 77–83), Fast3VmrMLM-Hap identified 59.52 SAMs and 10.43 suggested associated markers (Supplemental Tables 84–90), and 3VmrMLM identified 53.10 SAMs and 4.69 suggested associated markers (Supplemental Tables 91–97).

Within ±200-kb windows of all significantly and suggested associated markers (S²AMs), 26, 26, and 10 known genes (Supplemental Tables 98–100), along with 24, 23, and 33 candidate genes identified through gene differential expression analysis (Lawrence et al., 2004), GO annotation analysis (Lawrence et al., 2004, Love et al., 2014), rice homologous genes (Emms and Kelly, 2015), and haplotype analyses (Supplemental Figure 9; Supplemental Table 101; Supplemental Text 3), were detected by Fast3VmrMLM, 3VmrMLM, and Fast3VmrMLM-Hap, respectively. Among the known genes, four were commonly identified by both the new method and 3VmrMLM, whereas 41 and six were identified exclusively by the new method and 3VmrMLM, respectively. Among the candidate genes, seven were commonly identified by both approaches; 41 and 16 were uniquely identified by the new method and 3VmrMLM, respectively. These results demonstrate the effectiveness of the new methods and the complementarity between the new and 3VmrMLM approaches (Supplemental Figures 6C and 6D).

Comparison of the new and 3VmrMLM methods

Among all S²AMs, 133 were jointly identified by Fast3VmrMLM and 3VmrMLM, demonstrating consistency between the two methods. The proportions of S²AMs with dominant variance ratios greater than 50% were 52.73% for Fast3VmrMLM and 45.49% for 3VmrMLM (Supplemental Figures 10A and 10B), indicating the effectiveness of both methods in detecting S²AMs with dominant effects.

Prediction accuracy of genomic selection for maize yield-related traits

All S²AMs identified by Fast3VmrMLM and 3VmrMLM were used to perform genomic selection with the rrBLUP package (Endelman, 2011). Five-fold cross-validation with 20 replicates was conducted to obtain average prediction accuracies. The results showed prediction accuracies for the seven traits ranging from 0.6900 to 0.9185; notably, the accuracies for PH and Yield were 0.9013 and 0.9185, respectively (Supplemental Figure 11; Supplemental Table 102). Superior hybrid combinations were then selected based on the predicted phenotypes. Accordingly, 400 hybrid combinations with strong overall performance were listed in Supplemental Table 103 for further breeding validation.

Analysis of soybean structural variant dataset

To improve the universality of Fast3VmrMLM, we developed Fast3VmrMLM-mQTL for the detection of mQTLs, such as SVs. Fast3VmrMLM-mQTL was implemented to analyze soybean seed oil content datasets from Wuhan 2014 and Nanjing 2016 (Zuo et al., 2022). Two known genes—Glyma.08G213100 (Li et al., 2023) and Glyma.03G130900 (Kanai et al., 2019) (Supplemental Table 104)—were identified within 300-kb physical distance of significantly associated SVs, indicating the method’s effectiveness.

Comparison of genome-wide scanning plus machine learning framework with genome-wide scanning alone

To demonstrate the advantage of the genome-wide scanning plus machine learning framework (Fast3VmrMLM) over genome-wide scanning alone (MLM-scan), we compared the two methods using simulation datasets I-1 to I-4. Fast3VmrMLM outperformed MLM-scan in statistical power (92.12% versus 91.67%), FPR (0.02‱ versus 0.43‱), FDR (2.25% versus 24.96%), FNR (7.88% versus 8.33%), and F₁ score (0.95 versus 0.81) (Figure 2B; Supplemental Tables 2 and 3). These results indicate that Fast3VmrMLM achieves a better balance between power and FPR than MLM-scan. Similar trends were observed in simulation datasets II-1 to II-4 (Supplemental Tables 7 and 8).

To further validate this conclusion, we observed that MLM-scan failed to identify 716 of the 1555 SAMs detected by Fast3VmrMLM in the 18K rice datasets. Among these missed loci, 126 were located near known genes (Supplemental Table 105).

Discussion

Two important advances were achieved in this study. First, genome-wide scanning for gene identification in current statistical genetics methods was replaced by a genome-wide scanning plus machine learning framework, adapted for the era of large-scale data and AI. This framework aims to identify more genes to address the challenges of genetic dissection in polygenic traits (Purugganan and Jackson, 2021; Lappalainen et al., 2024). For example, Fast3VmrMLM (211 genes) and Fast3VmrMLM-Hap (215 genes) identified significantly more known genes than FarmCPU (100 genes) for 14 polygenic traits in the 18K rice dataset (Figure 4A). To handle large-scale datasets, several fast, efficient, and large-scale algorithms were integrated with our compressed variance component mixed model to improve computational and storage efficiency (Supplemental Table 16). Fast3VmrMLM successfully processed both the 18K rice and UK Biobank-scale datasets with 3.30 and 5.43 h, respectively, using a server with 20 threads and 1 TB of memory (Supplemental Tables 16 and 17). Second, all QTNs associated with known or candidate genes for nine rice yield-related traits were used to detect QQIs for each trait using a machine learning approach (empirical Bayes). These QQIs, identified across multiple datasets, were used to construct a genetic network to identify key genes for a specific type of polygenic trait. In the 1439 and 18K rice datasets, 21 hub genes associated with rice yield-related traits were identified within a network comprising 527 QQIs across 388 gene pairs. Evidence presented in Supplemental Table 76 supports the relevance of these hub genes. These genes are expected to play a critical role in improving rice yield-related traits in future breeding efforts.

The genome-wide scanning plus machine learning framework for gene mining in the era of large-scale data and AI

Since the introduction of the genome-wide scanning strategy for gene identification in linkage analysis (Lander and Botstein, 1989), GWAS (Risch and Merikangas, 1996), and bulked segregant analysis (Schneeberger et al., 2009), a substantial number of genes have been identified for polygenic traits across animal, plant, and human genetics. However, the genetic dissection of polygenic traits remains challenging (Purugganan and Jackson, 2021; Lappalainen et al., 2024) for two main reasons: first, existing methods focus primarily on a limited number of loci with strong signals and large effects; second, single-locus models exhibit low power for gene detection. To address these challenges, it is essential to integrate genome-wide scanning with machine learning in the era of big data and AI. The Fast3VmrMLM algorithm was developed with this goal. In the first stage, genome-wide scanning is used to select as many potentially associated markers as possible with a relaxed significance threshold. In the second stage, all potentially associated markers are analyzed in a multi-locus model using an empirical Bayes-based machine learning method, to identify S²AMs. Simulation studies (Supplemental Tables 7 and 8) and real-data analyses (Supplemental Table 105) demonstrated that this new framework outperforms genome-wide scanning alone. Compared with FarmCPU, Fast3VmrMLM considers both additive and dominant effects for each potentially associated marker and controls for additive and dominant polygenic backgrounds, thus addressing the limitations of existing GWAS methods in detecting small α and dominant effects (Supplemental Tables 2–15 and 50–52; Figure 2). To handle large-scale datasets, all Fast3VmrMLM procedures were optimized using several computational techniques, including the PCG method (Kaasschieter, 1988), GRAMMAR-gamma approximation (Svishcheva et al., 2012), the Woodbury matrix identity (Golub and van Loan, 1996), and the conditional expectation method (Henderson, 1975; Xu et al., 2014) (Supplemental Texts 4–7). To enable the identification of additional rare variants using bin markers, Fast3VmrMLM-Hap was developed (Supplemental Tables 18 and 67). To further extend its applicability, Fast3VmrMLM-mQTL was developed for the detection of mQTLs, such as SVs. In summary, the genome-wide scanning plus machine learning framework enables comprehensive gene mining, particularly suited for the demands of large-scale data and AI.

Conventional gene-mining methods for polygenic traits typically select a few strong signals from genome-wide scans and rely on multi-omics evidence for candidate genes near those loci—especially those repeatedly identified across multiple environments—to support gene-trait associations. In contrast, this study utilizes Fast3VmrMLM, which integrates differential expression analysis, gene annotation analysis, and haplotype analysis for candidate genes located around all S²AMs to support gene-trait associations (Supplemental Tables 48, 49, and 101). This approach provides unprecedented insights into the genetic architecture of polygenic crop traits and offers valuable references for experimental validation after GWAS. The goal is to identify as many genes related to polygenic traits as possible, addressing the persistent challenge of gene mining in this context. In the real-data analysis, 233 and 51 known genes associated with complex traits in rice and maize, respectively, were identified based on previous studies (Supplemental Tables 34–47 and 98–100). In addition, this study identified 381 and 64 candidate genes for complex traits in rice and maize (Supplemental Tables 48, 49, and 101), respectively, which have not yet been confirmed by transgenic experiments. For example, LOC_Os02g57660, which encodes a key enzyme (PIP5K) involved in the phosphatidylinositol signaling pathway—known to regulate plant growth, development, and stress responses (Zhang et al., 2020)—is a homolog of the known gene OsPIP5K1, which has been linked to yield-related traits (Fang et al., 2020).

Mining key genes of polygenic traits for breeding by design

Many polygenic traits, such as yield, are important production traits and major breeding objectives. However, these traits are typically controlled by numerous polygenes. In crop breeding, identifying which genes are critical remains a fundamental challenge. This study aims to address that question. We propose that three criteria can be used to determine whether a gene is key for yield-related traits: (1) pleiotropic genes involved in the regulation of multiple yield-related component traits (Huang et al., 2016), (2) hub genes with central roles in interaction networks; and (3) genes previously used in breeding programs. Using these criteria, 21 hub genes were identified in this study (Supplemental Tables 75 and 106). Specifically, Ghd8, OsSOC1, Ghd7.1, sd1, TAD1, and OsDEP1 satisfied all three criteria. LYL1 and WG7 met the first two criteria; OsPIN5b and TSCD11 met the first criterion; and SDG711 and OsFTL1 met the second criterion. The remaining nine hub genes were classified as candidate genes based on multi-omics evidence, requiring further validation of their function and breeding potential. Among the 21 hub genes, Ghd8, OsSOC1, Ghd7.1, sd1, OsFTL1, and OsDEP1 were consistent with key genes in the yield trait network identified by Wei et al. (2024). Additionally, Ghd8, Ghd7.1, sd1, and OsDEP1 were reported as key genes for rice yield improvement in breeding by Wu et al. (2018); Ghd8, Ghd7.1, OsSOC1, sd1, and OsDEP1 were recognized as representative key functional genes in rice molecular design breeding (Guo et al., 2019); Ghd8, Ghd7.1, and OsDEP1 were identified as key genes in green super rice breeding (Zhang et al., 2024); and Ghd8, Ghd7.1, TAD1, and OsDEP1 were documented as key genes involved in complex agronomic traits in rice (Zuo and Li, 2014). These findings highlight the utility of machine learning in identifying key genes for polygenic traits and demonstrate its potential to facilitate breeding by design. In human genetics, many complex diseases, such as type 2 diabetes (DeForest and Majithia, 2022), are also polygenic, and key genes have similarly been identified in disorders such as rheumatoid arthritis (Holmdahl, 2000). Therefore, this method is also applicable to human genetic studies.

Once key genes for a specific type of polygenic trait have been identified, they are first introduced into an elite cultivar through backcrossing and accelerated breeding in pure-line breeding, or into elite male and female inbred lines in hybrid breeding. Subsequently, multi-way crossing and doubled haploid technology are employed to pyramid and fix these key genes into a superior accession. This results in an elite cultivar in pure-line breeding, or elite male and female inbred lines that, when crossed, produce a superior hybrid in hybrid breeding. Such an approach represents a new strategy for the improvement of polygenic traits through breeding.

Although genomic selection has proven highly successful in animal breeding (Meuwissen et al., 2016), its high cost has limited broader application in crop breeding. It is therefore essential to reduce its cost. In this study, genomic selection based on all SAMs identified by Fast3VmrMLM achieved high prediction accuracies, ranging from 0.59 to 0.91 for rice and from 0.69 to 0.92 for maize. These results highlight the substantial value of the identified S²AMs in improving phenotypic trait prediction and accelerating crop breeding. Notably, the cost of genotyping has been significantly reduced by developing a chip based on these ^S2AMs. More importantly, genomic selection can also be applied to predict superior hybrid combinations in the partial maize NC II design experiment (Supplemental Table 103). Thus, the proposed framework supports both genomic selection and molecular breeding by design.

Comparison of Fast3VmrMLM with existing methods

Fast3VmrMLM differs fundamentally from existing GWAS methods. It is based on a genome-wide scanning plus machine learning framework, whereas most existing methods rely solely on genome-wide scanning. As a result, Fast3VmrMLM demonstrates significantly higher statistical power for multiple reasons. First, Fast3VmrMLM incorporates both additive and dominant effects and controls for their polygenic backgrounds, whereas most existing methods consider only the allelic substitution effect (α) and its corresponding polygenic background. This distinction underscores the efficiency of Fast3VmrMLM in identifying S²AMs with dominant effects and those with α ≈ 0.0, as well as in estimating comprehensive genetic effects (Supplemental Tables 2–15 and 50–52; Figure 2). Second, the statistical model of Fast3VmrMLM relies on a multi-locus model that simultaneously includes all potentially associated markers, in contrast to the single-locus model commonly used in genome-wide scanning by most existing methods. The two-step approach of Fast3VmrMLM first applies a loose significance threshold during genome-wide scanning to reduce the likelihood of missing S²AMs due to proximal contamination (Jiang et al., 2021), followed by a stringent threshold (Bonferroni correction) in the likelihood ratio test after the machine learning stage to control the FPR. Most existing methods, in contrast, often use a one-step strategy with a strict threshold to identify SAMs (Segura et al., 2012; Liu et al., 2016). This design gives Fast3VmrMLM an advantage in achieving both high power and low FPR, while mitigating the dimensionality challenge of multi-locus modeling. Third, the unified three-component (compressed) variance component mixed model in Fast3VmrMLM can be readily extended to detect gene-by-environment and gene-by-gene interactions without introducing additional terms into the mixed model. This avoids the complexity associated with algorithm redesign required by many existing methods when adapting to new model structures.

To meet the challenges posed by large-scale data and AI, algorithmic capabilities for large-scale datasets, fast computation, and low memory consumption were incorporated into the compressed variance component mixed model of Fast3VmrMLM, as described above. Fast3VmrMLM, developed based on the comprehensive genetic effect framework of 3VmrMLM, retains its advantages in identifying loci with small α and dominant effects (Supplemental Tables 2, 7, and 13; Figure 2C), while offering faster computation, and improved suitability for large-scale data. In real-data analysis, we observed that the gene-mining results of Fast3VmrMLM complemented those of 3VmrMLM (Supplemental Tables 98–101; Supplemental Figures 6C and 6D). Therefore, this study presents Fast3VmrMLM as a fast, efficient, and large-scale GWAS algorithm that complements and accelerates the performance of 3VmrMLM.

Finally, the superior performance of Fast3VmrMLM over FarmCPU is attributed to its novel framework (Figure 2B; Supplemental Table 105), as well as its high power in identifying dominant (Figure 2C; Supplemental Table 51), small-effect (Supplemental Tables 2 and 52), and rare QTNs (Supplemental Tables 18 and 67).

Detection and estimation of the dominant effect of SAMs

Dominance represents a key genetic basis for complex traits and heterosis and plays a significant role in genomic selection. Although the 18K rice dataset contains a low proportion of heterozygous genotypes (Supplemental Figure 10C), Fast3VmrMLM identified 100 SAMs with dominant variance ratios exceeding 50% (Supplemental Figure 10D), underscoring the importance of dominance in GWAS for this dataset. Additionally, SAMs with both additive and dominant effects identified by Fast3VmrMLM achieved high prediction accuracies in rice (0.7443) and maize (0.8492). These findings reinforce the importance of incorporating dominance effects in GWAS and genomic selection.

Fast3VmrMLM software is designed to run on a Linux server equipped with 60 CPUs and 1 TB of memory. For optimal results, users may adjust the “svrad” parameter according to specific species and traits. The sparse matrix technique was not employed in this study because most kinship coefficients exceeded 0.3 in the 1,439 and 18K rice datasets (Supplemental Figure 12). In future work, we plan to extend Fast3VmrMLM to detect QTN-by-QTN and QTN-by-environment interactions.

Methods

Compressed variance component mixed linear model

The phenotype $y_i$ $\left(i=1,\cdots ,n\right)$ of the ith individual for a complex trait in an association mapping population is modeled as:

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+{\mathbf{Z}}_k{a}_k+{\mathbf{W}}_k{d}_k+{\mathbf{U}}_a+{\mathbf{U}}_d+\mathbf{\epsilon }$$ (Equation 1)

where $\mathbf{X}$ is the $n\times q$ incident matrix for fixed effect ${\mathbf{\beta }}_{q\times 1}$; $a_k$ and $d_k$ are additive and dominant effects of the kth locus, respectively, and ${\mathbf{Z}}_k$ and ${\mathbf{W}}_k$ are corresponding dummy variables that indicate genotypes of the kth locus, defined as ${\mathbf{Z}}_k={\left(-1,0,1\right)}^{\mathrm{T}}$ and ${\mathbf{W}}_k={\left(0,1,0\right)}^{\mathrm{T}}$ for qq, Qq and QQ, respectively (Xu, 2013). The additive and dominant polygenic effects are modeled as ${\mathbf{U}}_a\sim \text{MVN}\left(0,{\tau }_a{\mathbf{K}}_a\right)$ and ${\mathbf{U}}_d\sim \text{MVN}\left(0,{\tau }_d{\mathbf{K}}_d\right)$, where ${\mathbf{K}}_a=\frac{1}{m}\sum _{k=1}^m{\mathbf{Z}}_k{\mathbf{Z}}_k^{\mathrm{T}}=\frac{1}{m}{\mathbf{Z}}_{n\times m}{\mathbf{Z}}^{\mathrm{T}}$ and ${\mathbf{K}}_d=\frac{1}{m}\sum _{k=1}^m{\mathbf{W}}_k{\mathbf{W}}_k^{\mathrm{T}}=\frac{1}{m}{\mathbf{W}}_{n\times m}{\mathbf{W}}^{\mathrm{T}}$ are additive and dominant kinship matrices, respectively; m represents the number of markers. The parameters ${\tau }_a$ and ${\tau }_d$ are variances of the additive and dominant polygenic effects. The residual error $\mathbf{\epsilon }\sim \text{MVN}\left(0,{\mathbf{I}}_{n\times n}{\tau }_e\right)$, with residual variance ${\tau }_e$ and unit matrix ${\mathbf{I}}_{n\times n}$.

To reduce the number of variance components in mixed model (1), as described in Li et al. (2022a), the model (1) is reformulated as:

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+\left({\mathbf{Z}}_k{\mathbf{W}}_k\right)\left(\begin{array}{l}{a}_k\\ d_k\end{array}\right)+{\mathbf{U}}_{{\left(ad\right)}^{\mathrm{T}}}+\mathbf{\epsilon }$$ (Equation 2)

where vector ${\left(\begin{array}{cc}{a}_k& d_k\end{array}\right)}^{\mathrm{T}}$ replaces $a_k$ and $d_k$ in model (1), or the allelic substitution effect α in existing methods (Lippert et al., 2011; Yang et al., 2011; Zhou et al., 2018; Jiang et al., 2019, 2021). The polygenic background term ${\mathbf{U}}_{{\left(\begin{array}{cc}a& d\end{array}\right)}^T}\sim \text{MVN}\left(0,\mathbf{K}\tau \right)$ follows a normal distribution with mean 0 and variance $\tau$, where $\mathbf{K}=\frac{1}{2m}\left(\mathbf{Z}{\mathbf{Z}}^{\mathrm{T}}+\mathbf{W}{\mathbf{W}}^{\mathrm{T}}\right)$; $\tau$ is the generalized genetic variance, which replaces the separate additive and dominant polygenic background components (Xu, 2013) in model (1) or the allelic substitution polygenic background used in SAIGE (Zhou et al., 2018) and fastGWA (Jiang et al., 2019, 2021). Thus, the five variance components in model (1) are reduced to three in model (2).

To reduce computational burden in the estimation of variance for each marker, we first considered the null model derived from model (2):

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+{\mathbf{U}}_{{\left(ad\right)}^{\mathrm{T}}}+\mathbf{\epsilon }$$ (Equation 3)

This null model was used to estimate the parameters $\tau$ and ${\tau }_e$, where ${\mathbf{U}}_{{\left(\begin{array}{cc}a& d\end{array}\right)}^{\mathrm{T}}}=\sum _{k=1}^m\left(\begin{array}{cc}{\mathbf{Z}}_k& {\mathbf{W}}_k\end{array}\right)\left(\begin{array}{l}{a}_k\\ d_k\end{array}\right)={\left(\begin{array}{cc}\mathbf{Z}& \mathbf{W}\end{array}\right)}_{n\times 2m}\left(\begin{array}{l}\mathbf{a}\\ \mathbf{d}\end{array}\right)$ represents the polygenic effects, and $\left(\begin{array}{l}\mathbf{a}\\ \mathbf{d}\end{array}\right)\sim \text{MVN}\left(\mathbf{0},\frac{\tau }{2m}{\mathbf{I}}_{2m}\right)$ is the random effect vector for all markers, in conjunction with unit matrix ${\mathbf{I}}_{2m}$. Subsequently, all marker effects were predicted using the conditional expectation method (Henderson, 1975; Xu et al., 2014).

Restricted maximum log-likelihood function and its solution

The restricted maximum log-likelihood (REML) function of model (3), given $\tau$ and ${\tau }_e$, is defined as:

$$l_R\left(\tau ,{\tau }_e\right)\propto -\frac{1}{2}\left[\left(n-q\right)\log \left({\tau }_e\right)+\log \left|\mathbf{\Sigma }\right|+\log \left|{\mathbf{X}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{X}\right|+\frac{1}{{\tau }_e}{\mathbf{y}}^{\mathrm{T}}\mathbf{\text{Py}}\right]$$ (Equation 4)

where $\mathbf{P}={\mathbf{\Sigma }}^{-1}-{\mathbf{\Sigma }}^{-1}\mathbf{X}{\left({\mathbf{X}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\mathrm{T}}{\mathbf{\Sigma }}^{-1}$ and $\mathbf{\Sigma }=\frac{\tau }{2m}\left(\mathbf{Z}{\mathbf{Z}}^{\mathrm{T}}+\mathbf{W}{\mathbf{W}}^{\mathrm{T}}\right)+{\mathbf{I}}_{n\times n}{\tau }_e$ is the variance-covariance matrix of $\mathbf{y}$. The average information REML (AI-REML) method (Arthur et al., 1995) is used to iteratively estimate the variance components $\tau$ and ${\tau }_e$ (Supplemental Text 4). To reduce the computational burden of inverting the variance-covariance matrix $\mathbf{\Sigma }$ during each iteration, particularly in large-scale datasets, the PCG method (Kaasschieter, 1988) is utilized (Supplemental Text 4).

Fast3VmrMLM algorithm

Fast3VmrMLM consists of two steps: (1) single-locus genome-wide scanning to select potentially associated markers and (2) machine learning to identify S²AMs. In the first step, the conditional expectation method (Henderson, 1975) is applied to model (3) to predict the conditional expectations and corresponding variances of marker effects ${\left(\begin{array}{cc}\mathbf{a}& \mathbf{d}\end{array}\right)}_{2m\times 1}^{\mathrm{T}}$, given the estimated parameters $\tau$ and ${\tau }_e$ (Supplemental Text 5). The computational complexity of variance and covariance estimation for each marker in this step is $O\left(m{n}^2\right)$, which becomes inefficient when the sample size n is large. To address this issue, the GRAMMAR-gamma approximation method (Svishcheva et al., 2012) is used to estimate the gamma ratio by randomly selecting a subset of markers and approximating the variance-covariance matrices for all markers (Supplemental Text 6).

Based on the estimated effects and variance-covariance matrices of all markers, a vectorized Wald test strategy (Li et al., 2024) is used to calculate genome-wide Wald statistics via $W_k=\left(\begin{array}{cc}{\hat{a}}_k& {\hat{d}}_k\end{array}\right){\left[\begin{array}{cc}\text{var}\left({\hat{a}}_k\right)& \text{cov}\left({\hat{a}}_k,{\hat{d}}_k\right)\\ \text{cov}\left({\hat{d}}_k,{\hat{a}}_k\right)& \text{var}\left({\hat{d}}_k\right)\end{array}\right]}^{-1}\left(\begin{array}{l}{\hat{a}}_k\\ {\hat{d}}_k\end{array}\right)\sim {\chi }_{\text{df}=2}^2$, where $W_k$ is the Wald statistic for the kth marker. Using a relatively loose p value threshold (Wellcome Trust Case Control, 2007; Wang et al., 2016; Li et al., 2022a), variants with p values ≤1.00e−5 were regarded as potentially associated variants. After initial selection, we removed some potentially associated variants around the strongest variant. The window size for exclusion depends on the species (Li et al., 2022a, Li et al., 2022b); for example, 20 kb for PH, EH, and SW in maize, and 200 kb for ET, KRN, KNR, and Yield in maize, and all traits in rice.

In the second step, all selected potentially associated markers were entered into Equation 5, and a machine learning method—expectation-maximization (EM) empirical Bayes (Xu, 2010; Li et al., 2022a)—was used to estimate all genetic effects.

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+{\sum }_{i=1}^q{\mathbf{Z}}_i{\gamma }_i+\mathbf{\epsilon }$$ (Equation 5)

where $\mathbf{y}$, $\mathbf{X}$, $\mathbf{\beta }$, $\mathbf{\epsilon }$ are as defined in Equation 1; $q$ is the number of potentially associated variants selected from the Wald test; and ${\gamma }_i$ represents the genetic effect associated with the indicator vector ${\mathbf{Z}}_i$. Based on a normal prior for ${\gamma }_i$, $\mathrm{P}\left({\gamma }_i|{\tau }_i\right)$, with a scaled inverse ${\chi }^2$ prior for ${\tau }_i$, all effects can be estimated using the EM empirical Bayes framework (Supplemental Text 7). Matrix inversion poses a major computational challenge in large-scale datasets. Therefore, this study introduces two strategies to improve computational efficiency and reduce memory usage within the EM empirical Bayes procedure (Supplemental Text 7). When $q$ is smaller than the sample size ($n$), the Woodbury matrix identity (Golub and van Loan, 1996) is used to calculate ${\mathbf{\Sigma }}^{-1}$ via ${\mathbf{\Sigma }}^{-1}={\mathbf{I}}_{q\times q}{\tau }_e^{-1}+{\tau }_e^{-2}\mathbf{Z}{\left({\mathbf{C}}^{-1}+{\mathbf{Z}}^{\mathrm{T}}\mathbf{Z}{\tau }_e^{-1}\right)}^{-1}{\mathbf{Z}}^{\mathrm{T}}$, where $\mathbf{C}$ is a diagonal matrix with diagonal elements ${\mathbf{C}}_{ii}={\tau }_i$. When both $q$ and $n$ exceed 4000, the PCG method (Kaasschieter, 1988) is used.

All markers with non-zero effects from the machine learning method are subsequently evaluated using a likelihood ratio test. The critical p value of 0.05/m, based on Bonferroni correction, and the threshold of LOD score ≥3.0 (Wang et al., 2016) are used to identify S²AMs.

Fast3VmrMLM-Hap and Fast3VmrMLM-mQTL algorithms

To identify rare variants and additional trait-associated genes, the Fast3VmrMLM-Hap algorithm was developed. In this approach, adjacent SNP markers are first grouped into bin markers—also referred to as dosage markers (Zhou et al., 2022)—based on linkage disequilibrium, as described by An et al. (2020). Haplotypes are then derived based on these dosage values. To minimize the number of rare haplotype variables, samples with similar dosages are grouped into a single haplotype. These genome-wide haplotypes across chromosomal regions are subsequently used in trait association analysis with Fast3VmrMLM, constituting the Fast3VmrMLM-Hap algorithm. To identify associations between molecular markers and traits, Fast3VmrMLM was further extended to Fast3VmrMLM-mQTL, which accommodates markers with differing numbers of genotypes across loci.

Monte Carlo simulation studies

In Monte Carlo simulation studies, PLINK software (Chang et al., 2015) was used to simulate datasets I, III, V, and VII, whereas the real genotypic dataset of Simmental Beef (Zhu et al., 2016) was used to simulate dataset II. The rice genotypic dataset (Huang et al., 2015) was used to simulate datasets IV and VI. The evaluation indicators followed those described in Li et al. (2022a). Technical details are provided in Supplemental Text 1.

Real-data re-analyses of the 18K rice dataset

The 18K rice dataset consisted of 15 recombinant inbred line (RIL) populations and one four-way MAGIC population (Wei et al., 2024). The RIL populations were derived from crosses between Huanghuazhan and 15 other accessions, whereas the MAGIC population originated from a four-way cross involving Huanghuazhan, Kasalath, Nipponbare, and IAC25. In this study, 14 traits from 18 421 lines in the 18K rice dataset were analyzed in association with 2 929 530 SNPs for GWAS using Fast3VmrMLM and Fast3VmrMLM-Hap. The top two principal components were used to correct for population structure (Wei et al., 2024).

Genes located within a ±200-kb window of SAMs were examined. Genes with at least one piece of evidence from overexpression, CRISPR, RNAi experiments, mutant versus wild type comparisons, or molecular mechanism studies were considered known genes (https://www.ricedata.cn/ontology/default.aspx). Candidate genes were identified through differential expression analysis (Wang et al., 2010; Sato et al., 2011; Jathar et al., 2022), GO annotation (http://systemsbiology.cau.edu.cn/-agriGOv2/), and haplotype analyses. All physical positions of markers and genes are based on the rice MSU Reference Genome Release 7.0 (https://rice.uga.edu/).

Genomic selection for rice and maize was conducted using the R package rrBLUP (Endelman, 2011), based on BLUP values and all identified S²AMs. Five-fold cross-validation was used to assess prediction accuracy, implemented via the createFolds function in the R package caret (Kuhn, 2008).

Genetic interaction detection and network construction

To identify key genes associated with nine rice yield-related traits—HD, PH, PN, GL, GY, and GW in the 18K rice dataset (Wei et al., 2024), and HD, PH, PN, GY, GN, SSR, and TGW in the 1439 rice dataset (Huang et al., 2015)—all QTNs linked to known or candidate genes in the 18K rice dataset (Supplemental Tables 34, 35, 38, and 42–44) and all known genes identified in the 1439 rice dataset (Supplemental Tables 68–74) were used to detect epistatic interactions via the EM empirical Bayes method. The epistatic effects of all QTN pairs were estimated using the following multi-locus model

$$\mathbf{y}=\mathbf{X}\mathbf{\beta }+{\sum }_{i=1}^q{\mathbf{Z}}_i{a}_i+\sum _{i=1}^{q-1}\sum _{j=i+1}^{j=q}{\mathbf{Z}}_i\#{\mathbf{Z}}_j{\left(aa\right)}_k+\mathbf{\epsilon }$$ (Equation 6)

where ${\left(aa\right)}_k$ denotes the kth additive-by-additive interaction effect between the ith and jth SNPs, with $k=\left(i-1\right)\left(q-0.5i\right)+j-i$, $\mathbf{y}$ represents the BLUP values calculated from multi-environment phenotypes for each trait. Other terms are the same as in model (1). All effects and variances in model (6) were estimated using the fast version of the EM empirical Bayes method described in Supplemental Text 7. A Wald test was then performed on each SNP pair based on the estimated effects and variances to identify significant epistatic interactions (p < 0.01) (Wei et al., 2024). Considering the low SNP heterozygosity rate in the 18K rice dataset (97.52% of SNPs had heterozygosity with rates below 4.0%; Supplemental Figure 10C), only additive and additive-by-additive epistatic effects were considered in model (6).

All known and candidate genes associated with the nine traits, along with the significant epistatic pairs, were used to construct a genetic network using Cytoscape software (Shannon et al., 2003). The degree of each gene in the network was calculated and visually represented by node size and label font size, both proportional to the degree value. Connecting lines, representing epistatic interactions specific to each trait, were color-coded to distinguish genes involved in multiple yield-related traits. Genes with high connectivity were identified as hub genes.

Construction, planting, and phenotyping of the maize population

The North Carolina Design II (NC II) maize population, planted in 2019 and 2022, consisted of 145 parental lines (56 male and 89 female) and 1166 hybrids. In 2019, 97 inbred lines, including eight male and 89 female, along with 712 hybrids, were planted in Baixiang, Luohe, and Zhengzhou, China. In 2022, 65 inbred lines, including 55 male and 10 female, along with 521 hybrids, were planted in Shijiazhuang and Zhengzhou. Seventeen inbred lines and 67 hybrids were shared across both years. Pedigree information is presented in Supplemental Table 107. The NC II maize population was phenotyped for seven yield-related traits including PH, EH, KRN, KNR, SW, ET, and yield per plant (Yield). Control varieties were planted at each site to correct for environmental variation in phenotypes.

Whole-genome resequencing and genotyping

Fifty seeds were collected from each inbred line, and young seedlings were harvested 4 days after sowing. The seedlings were thoroughly ground in liquid nitrogen, and DNA was extracted using the CTAB method. DNA samples were submitted to BGI for 10× whole-genome resequencing on the BGI platform. Raw sequencing data were quality-controlled and filtered using SOAPnuke (version: v2.2.6) (Chen et al., 2018). Sequencing reads were aligned to the B73V5 (https://download.maizegdb.org/Zm-B73-REFERENCE-NAM-5.0/) (Lawrence et al., 2004) reference genome using the MEM algorithm of BWA (version V0.7.17) (Li and Durbin, 2009), generating alignment files in SAM format. These SAM files were converted into sorted BAM files using the view and sort commands of Samtools (version: V1.16.1) (Li et al., 2009). The MarkDuplicates tool in Picard (v2.22.8) (https://broadinstitute.github.io/picard/) was used to mark PCR duplicates. The Qualimap2 (version v.2.3) (Okonechnikov et al., 2016) bamqc tool was used to perform quality control analyses of the BAM files. Based on the BAM files aligned to the reference genome, GATK (version v4.1.8.1) (McKenna et al., 2010) was used to call SNPs and InDels across all samples.

Real-data analyses for the maize dataset

The R package lme4 (version 1.1.31) (Bates et al., 2015) was used to calculate BLUP values for all seven traits across multiple locations in 2019, 2022, and the combined dataset. PLINK (Chang et al., 2015) was used for genotype quality control by filtering variants with a missing rate >0.3. Beagle was then used for imputation, and PLINK was subsequently used to filter variants with a minor allele frequency <0.01 to obtain the final genotype dataset. This process yielded 13 528 478 markers for all 145 parental lines. All ungenotyped hybrids were inferred from their corresponding parents using Fast3VmrMLM (Supplemental Text 8; Supplemental Table 107).

The top three principal components, computed using the flashpcaR package (version 2.1) (Abraham et al., 2017), were used to correct for population structure. A ±200-kb window around S²AMs was used to identify known and candidate genes. Candidate gene confirmation (Supplemental Text 3) and genomic selection procedures were consistent with those used for the rice dataset.

Data and code availability

The Fast3VmrMLM, Fast3VmrMLM-Hap, and Fast3VmrMLM-mQTL algorithms proposed in this study have been integrated into the R software package Fast3VmrMLM v1.0, which is freely available at https://github.com/YuanmingZhang65/.

The 18K (Wei et al., 2024) and 1439 (Huang et al., 2015) rice datasets were obtained from https://figshare.com/s/12978737918eecb74903 and http://www.ncgr.ac.cn/RiceHap4, respectively. The simulation datasets are available from the corresponding author upon request.

Funding

This study was supported by the National Natural Science Foundation of China, China (32470657 and 32270673).

Acknowledgments

We thank Prof. Xuehui Huang from the College of Life Sciences of Shanghai Normal University for reading and commenting on the manuscript draft and for providing the 1439 and 18K rice datasets used in this study. No conflict of interest is declared.

Author contributions

Y.-M.Z. conceived and supervised the study and revised this manuscript. Y.W. and G.S. assisted in the supervision of the study and conducted the maize experiment. Y.-M.Z. and J.W. developed the methods and designed the study. Y.-M.Z., J.W., Y.C., and M.Z. wrote the draft. J.W. developed the software tool. J.W. and G.L. performed the simulation studies. J.W., Y.C., M.Z., A.Z., and X.C. contributed to GWAS and genetics analysis of the maize, rice, and soybean datasets. All the authors reviewed and approved the final manuscript.

Published: May 22, 2025