Synthetic-augmented multimodal deep learning fuses dual-angle RGB images and phenology to unlock genotype-informative canopy structural trait in wheat

Abstract

The wheat canopy genome harbors abundant yet untapped genetic variation that could be harnessed to enhance yield potential. The green area index (GAI) is a structural metric that reflects the photosynthetically active canopy surface and is closely linked to final grain yield. Current image-based GAI retrieval methods often suffer from signal saturation and coarse structural depiction, constraining downstream genetic analyses. To address this limitation, we constructed a comprehensive image dataset spanning eight field experiments across China and France, encompassing approximately 600 genotypes under six distinct management regimes. Leveraging this diverse data, we developed a multimodal deep-learning framework augmented by simulated-to-realistic (sim2real) synthetic data transfer. This framework fuses nadir and oblique RGB images with accumulated thermal time to produce high-precision, time-series GAI estimates. Validated on independent testing datasets from both China and France, the multimodal approach demonstrated robust performance with an accuracy of R² = 0.88 and an RMSE of 0.49 m² m⁻², representing an improvement of about 22% over the traditional gap fraction method. In three site-year field experiments involving 565 genotypes, the GAI dynamics derived from the multimodal approach showed higher broad-sense heritability (0.20-0.48) than those from the gap fraction approach (0.02-0.13) and stronger genotypic correlations with yield (0.19-0.40 versus 0.09-0.31). Furthermore, genetic analysis confirmed the biological fidelity of the estimated traits, identifying loci that co-localize with known architectural regulators such as Rht-D1, TaTB1-4D, and TaBGC1-4D. Consistently, the multimodal-derived phenotypes were specifically enriched in cell-wall remodeling and hormonal signaling pathways (e.g., brassinosteroid) that directly regulate canopy expansion. Overall, the proposed method offers a powerful tool for unlocking genetic gain in canopy architecture and accelerating canopy-targeted wheat improvement.

1. Introduction

Wheat breeders have long sought to convert the rich standing genetic variation in canopy architecture into higher, more resilient yields [1,2]. However, modern wheat cultivars still lack many of the alleles that regulate key morphological traits, underscoring the untapped genetic potential locked in the canopy genome [3]. Although the release of a high-quality reference genome has made this opportunity increasingly tractable [4], exploiting it depends on a phenotypic descriptor that links canopy structure directly to yield. The green area index (GAI), defined as the green plant area per unit ground area, captures the photosynthetic source that drives radiation use efficiency [5], biomass [6] and ultimately grain set [7]. Early-season GAI correlates strongly with grain yield, implying that genotypes with vigorous canopy development can be selected to enhance yield potential [8]. Consequently, GAI is a trait of major interest for breeders aiming to improve both productivity and stress resilience in wheat.

Recent advances in high-throughput phenotyping have greatly enhanced the ability to measure canopy traits such as GAI rapidly and non-destructively in the field [9,10]. Ground-based platforms can acquire repeated Red-Green-Blue (RGB) images of wheat plots throughout the season, enabling time-resolved monitoring of canopy development [11,12]. From these cost-effective RGB images, the gap fraction (GF)—a direct proxy for canopy light transmission governed by the Beer-Lambert law—remains the standard physical metric for estimating canopy structural traits in high-throughput phenotyping pipelines [13,14]. These image-derived metrics, including simple canopy cover percentage, have successfully approximated canopy size and greenness across multiple studies [[15], [16], [17]].

Despite this progress, current phenotyping approaches for canopy GAI have several limitations. Most imaging studies rely on single-angle (typically nadir, top-down) views of the canopy, which can miss structural details and lead to signal saturation in dense stands. For instance, once the wheat canopy fully covers the ground (GAI ≈ 3-4 m² m⁻²), further increases in leaf area produce diminishing changes in RGB greenness, obscuring true genetic differences at high vigor [18,19]. As a result, the heritability of image-derived traits such as canopy cover is often lower than that of directly measured traits, blunting the power to detect genetic effects [20]. Data scarcity compounds the problem: obtaining ground-truth GAI through destructive sampling or specialized instrumentation is both time-consuming and costly, yielding relatively small labeled datasets for model development [21]. These constraints—including single-view observations, sensor saturation, and a lack of abundant training data—have collectively restricted the effectiveness of high-throughput GAI phenotyping in breeding programs.

One promising yet underutilized solution to overcome data scarcity is synthetic canopy rendering and in silico data augmentation. Advances in computer graphics and plant modeling now allow researchers to generate photorealistic images of virtual crop canopies under controlled parameters [22]. Annotated synthetic images have been proposed as an efficient means to expand training datasets for deep-learning models[23]. These virtual datasets can greatly broaden the range of canopy structures, lighting conditions, and growth stages encountered during model training, thereby improving robustness. Generative modeling approaches (e.g., Generative Adversarial Networks) have been explored to create synthetic images that mimic real field conditions [24,25]. By augmenting scarce real images with abundant synthetic renderings, one can mitigate data shortages and expose learning models to canopy examples beyond what is easily captured in field experiments. However, technical precision is only half the battle; ensuring biological interpretability is equally critical [26]. To date, the application of photorealistic simulation in crop phenotyping remains in its infancy [27], and a critical challenge persists: verifying the biological fidelity of deep-learning-derived traits [28]. While image-based models can achieve high statistical accuracy, they are often treated as black boxes. This opacity leaves it unclear whether they capture genuine physiological signals or merely exploit background correlations unrelated to plant biology to minimize error.

In this study, we propose a synthetic-augmented multimodal deep-learning framework that fuses dual-angle RGB imaging, cumulative thermal time (TT), and photorealistic canopy renderings to estimate wheat canopy GAI and evaluate the genotypic informativeness of the resulting dynamic traits (Fig. 1A). We collected paired nadir and oblique RGB images for field-grown wheat plots, providing complementary perspectives that capture 3D canopy structure and mitigate saturation. TT was incorporated as an additional input to encode phenological stage. The framework is built on a Vision Transformer (ViT) backbone pre-trained on ImageNet-21 K, which enables efficient capture of global image context [29]. During training, the network receives both images and the corresponding TT for each plot and date, learning GAI as a joint function of canopy appearance and developmental progress (Fig. 1B). Synthetic wheat-canopy images spanning a broad range of GAI values further expand the training domain and enhance model robustness. We applied the framework to track GAI from emergence to flowering across three site-year experiments involving 565 genotypes. Crucially, we moved beyond standard accuracy metrics to rigorously validate the biological fidelity of the derived traits. We benchmarked the multimodal method against a conventional GF model in terms of broad-sense heritability, genotypic correlation with yield, and alignment with known genetic pathways via genome-wide association studies and gene ontology enrichment. To our knowledge, this is the first canopy-trait estimation pipeline that integrates dual-angle RGB phenotyping, phenology-aware time series and in silico synthetic augmentation, providing breeders with a more interpretable and genetically informative target for crop improvement.

Fig. 1.

Multimodal pipeline and model architecture for green area index (GAI) estimation and downstream genetic discovery. (A) End-to-end workflow: dual-angle RGB images from field phenotyping are fused with style-translated synthetic images generated by in silico phenotyping to augment model training. Together with accumulated thermal time, they feed a multimodal Vision Transformer (ViT) that outputs phenology-specific GAI. The resulting GAI supports phenotypic, genotypic, and genetic analyses. (B) Structure of the multimodal ViT: cropped image patches from both viewing angles are embedded and passed through a multi-layer Transformer encoder; thermal time recorded at imaging is concatenated in the multilayer perceptron head to regress the corresponding GAI.

2. Materials and methods

2.1. Field experiments

We conducted eight site-year experiments from 2017 to 2024 across five locations: Baima, Jurong, Xuzhou, and Yangling in China, and Gréoux in France. These experiments encompassed approximately 600 wheat genotypes under six distinct management regimes, including varying nitrogen levels (low, medium, and high), irrigation versus rainfed conditions, and standard local practices.

The French experiment was conducted at Gréoux (43°45′N, 5°52′E) during the 2017-2018 season (GR18), involving 14 cultivars under irrigated and rainfed conditions. The field followed a randomized block design with two sections based on the water regime. Plots measured 2 × 10 m with a sowing density of 250 plants m⁻², and destructive measurements were conducted on three replicates of six representative cultivars at the stages of stem elongation and flag leaf ligule visible. In China, three agronomic experiments were conducted during the 2020-2021 season in Jiangsu Province. The experiment at Baima (BM21; 31°36′N, 119°11′E) involved 120 cultivars grown under low (0 kg N ha⁻¹) and high (360 kg N ha⁻¹) nitrogen levels in 1 × 1 m plots (240 seeds m⁻²). Concurrently, experiments in Jurong (JR21; 32°10′N, 119°07′E) and Xuzhou (XZ21; 34°16′N, 117°17′E) tested five cultivars under three nitrogen levels (0, 180, and 300 kg N ha⁻¹) in 9 × 10 m plots (300 seeds m⁻²). Destructive sampling was performed at emergence, stem elongation, flag leaf sheath extension, and flowering by harvesting either 10 stems (BM21) or 1 m row segments (JR21 and XZ21) per plot.

Four experiments were conducted using a population of 565 genotypes under standard management practices: three at Yangling (34°18′N, 108°05′E) from 2021 to 2024 (YL22, YL23 and YL24), and one at Baima from 2023 to 2024 (BM24). These experiments followed an augmented randomized block design [30] to rigorously control for spatial heterogeneity. Each field was divided into 16 blocks, with each block containing 40 plots. Within each block, five check genotypes were replicated once, while a random subset of 35 test genotypes (from the total 560 test lines) were planted without replication, resulting in a total of 640 plots per experiment. Each plot measured 5 × 1.2 m with a sowing density of 270 seeds m⁻², managed under standard local fertilizer and irrigation practices. For YL22 and YL23, 32 representative cultivars were selected for destructive sampling from emergence to flowering by harvesting two diagonal rows (0.4 m in length each).

The sampling and manual measurement procedures were similar across experiments. Plants within defined quadrats were harvested, separated into organs (leaves, stems, and spikes), and their projected areas were measured using a LI-3100C Area Meter (LI-COR, USA). The total projected area was then summed and scaled to the plot level to derive the ground-truth GAI. Further details are available in our previous work [31].

2.2. Images acquisition and simulation

2.2.1. Field images acquisition

In the GR18 experiment, phenotypic monitoring was conducted using the autonomous Phenomobile robot (Fig. 1A) [32]. Guided by centimeter-level RTK-GPS, the platform moved along the row direction at a speed of 0.3 m s⁻¹, carrying two sensor gimbals oriented perpendicular to the row direction at zenith angles of 0° and 45°. Each gimbal was equipped with an RGB camera (VLG-40C, Baumer, USA) mounted 1.5 m above the canopy. These cameras captured flash-illuminated images with a resolution of 2000 × 2000 pixels, covering a 0.4 × 0.4 m field of view with a spatial resolution of 0.2 mm.

For the experiments in China, image acquisition was performed using portable ground platforms: a handheld system (JR21, XZ21, YL24, BM24) and a push-trolley system (BM21, YL22, YL23). Despite the difference in carrier, both platforms shared an identical sensor configuration: two high-resolution RGB cameras (RX0, Sony, Japan) mounted at 0° and 45° zenith angles, approximately 1.5 m above the canopy. These cameras captured synchronized [0°, 45°] images with a resolution of 4800 × 3200 pixels, covering a ground footprint of 2.4 × 1.6 m with a spatial resolution of 0.5 mm.

2.2.2. Synthetic images simulation

To address the scarcity of labeled data, we followed the pipeline proposed by Gao et al. [25] to generate a synthetic dataset. The ADEL-Wheat functional-structural model [33] embedded in the digital plant phenotyping platform (D3P) [22] was employed to construct diverse 3D wheat canopy scenes by varying structural parameters. Crucially, the ground-truth GAI for each scene was directly computed from the total surface area of green organs in the 3D mesh. This approach ensured precise labeling accuracy, with each GAI value perfectly corresponding to the subsequently rendered synthetic image. These 3D scenes were rendered into 2D RGB images via ray-tracing (512 × 512 pixels, Fig. S1). To bridge the domain gap between real and simulated images, we applied a CycleGAN-based domain adaptation technique [34]. CycleGAN learns a mapping between the synthetic and real image domains using a cycle-consistency loss, thereby enhancing the photorealism of the renders while preserving their structural geometry. The model was trained on an NVIDIA RTX 4090 GPU using the Adam optimizer with a momentum parameter of 0.5, a batch size of 1, and an initial learning rate of 0.0002. The final simulated-to-realistic (sim2real) dataset comprised 2000 images spanning vegetative stages, which were used to augment the training of the GAI estimation model (Supplementary Note 1 and Fig. S1).

2.3. Training, testing and application datasets

Based on the diversity of experimental sites, genotypes, and management practices, the collected field images and synthetic data were categorized into distinct training, testing, and application sets (Table 1). The training set was compiled from the BM21, JR21, YL22, and YL23 experiments in China. It encompassed approximately 50 representative genotypes grown under four management regimes (low, medium, and high nitrogen levels, as well as standard management) and covered growth stages from emergence to flowering. In total, this set contained 594 paired samples of images and destructive GAI measurements, which were used to optimize the parameters of the multimodal model. To further enhance model performance and generalization, the synthetic images and their corresponding ground-truth GAI were fused with this field training data.

Table 1.

Summarized dataset information from destructive measurement and phenotyping image collection for several experiments used in this study.

Dataset	Experiment	Year	Country	Site	Lat, Long	Cultivars	Treatment	Sampling stages	Samples number	Phenotyping stages	Images number
Training	BM21	2021	China	Baima	31°36′N, 119°11′E	15	Low, high N	Z10, Z30, Z40, Z60	201	Z10, Z30, Z40, Z60	201
	JR21	2021		Jurong	32°10′N, 119°07′E	5	Low, median, high N	Z10, Z30, Z40	45	Z10, Z30, Z40	45
	YL22	2022		Yangling	34°18′N, 108°05′E	32	Standard	Z10, Z30, Z40, Z60, Z70	133	Z10, Z30, Z40, Z60, Z70	133
	YL23	2023		Yangling	34°18′N, 108°05′E	32	Standard	Z10, Z20, Z30, Z40, Z50, Z60, Z70	215	Z10, Z30, Z40, Z60, Z70	215
Testing	GR18	2018	France	Gréoux	43°45′N, 5°52′E	6	Irrigated, rainfed	Z30	55	Z30	55
	XZ21	2021	China	Xuzhou	34°16′N, 117°17′E	5	Low, median, high N	Z10, Z30, Z40	45	Z10, Z30, Z40	45
Application	YL23	2023	China	Yangling	34°18′N, 108°05′E	565	Standard	-	-	Z10, Z20, Z30, Z40, Z50, Z60	7680
	YL24	2024		Yangling	34°18′N, 108°05′E	565	Standard	-	-	Z10, Z20, Z30, Z40, Z50, Z60	6400
	BM24	2024		Baima	31°36′N, 119°11′E	565	Standard	-	-	Z10, Z20, Z30, Z40, Z60	4480

Note: Experiments are labeled by site and harvest year. “Lat, Long” indicates the latitude and longitude of each site. Treatments include low, medium, and high nitrogen levels, as well as irrigated and rainfed conditions, and a single standard management modality. Sampling and phenotyping stages are expressed in Zadoks scale: Z10: seeding growth; Z20: tillering; Z30: stem elongation; Z40: booting; Z50: Inflorescence emergence; Z60: anthesis; Z70: milk development [35]. Samples number indicates the number of plots subjected to manual sampling, and images number indicates the number of plots captured with RGB phenotyping.

The testing set and application set were reserved for model evaluation and downstream analysis, respectively. The testing set consisted of data from the GR18 (France) and XZ21 (China) experiments, comprising 100 samples with destructive GAI ground truth. This set included 11 genotypes under five distinct management conditions, including irrigated versus rainfed regimes and varying nitrogen levels. Notably, the GR18 experiment represents a completely independent domain with cultivars and environmental conditions distinct from the Chinese experiments, ensuring a rigorous evaluation of model transferability. This set was used exclusively to evaluate the estimation accuracy of the multimodal and gap fraction models. Finally, the application set included the large-scale population experiments (YL23, YL24, BM24), involving 565 genotypes under standard management. This extensive dataset served as the basis for heritability estimation and genotype-phenotype association analysis, validating the biological relevance and breeding utility of the traits derived from both the multimodal and gap fraction models.

2.4. Green area index estimation from images

2.4.1. Multimodal model

The multimodal framework estimates GAI by fusing dual-angle RGB images with cumulative thermal time (TT) (Fig. 1B). In this workflow, nadir (0°) and oblique (45°) images were resized to 384 × 384 pixels and concatenated channel-wise to form a 6-channel input tensor. This input tensor was then divided into non-overlapping patches of 16 × 16 pixels, resulting in a sequence of 576 patches (24 × 24) per image. This sequence is processed by a Vision Transformer (ViT) backbone, which extracts a 1024-dimensional global feature vector from these images. To integrate phenological context, the normalized TT value was concatenated with the image features, and a multilayer perceptron (MLP) head regressed the final GAI. This single-time-step design enables robust prediction without relying on continuous historical data. The cumulative TT was calculated as the sum of daily mean temperatures (>0 °C) from emergence to the imaging date [36]. The daily temperature (at 2 m above ground) used in this study was sourced from the NASA POWER (https://power.larc.nasa.gov/data-access-viewer), which provides meteorological variables derived from the NASA GMAO MERRA-2 reanalysis dataset [37].

To identify the optimal visual encoder for this framework, we conducted a preliminary comparison of the ViT [38] against three other state-of-the-art architectures: ConvNeXt [39], EVA [40] and DINOv2 [41]. To ensure a rigorous comparison, all candidate models were trained on the training dataset using an identical protocol and evaluated on the independent testing set (Table 1). As detailed in Supplementary Note 2 and Fig. S2, ViT achieved the highest accuracy (R² = 0.868, RMSE = 0.498 m² m⁻², MAE = 0.365 m² m⁻²) and was therefore selected as the backbone. Its superior performance is attributed to the self-attention mechanism, which effectively captures long-range spatial dependencies critical for delineating canopy structure.

The final model was implemented in PyTorch and trained on an NVIDIA RTX 4090 GPU. We employed the mean absolute error (MAE) loss function and the AdamW optimizer with an initial learning rate of 5 × 10⁻⁶ and a weight decay of 1 × 10⁻². The learning rate was modulated using a cosine annealing strategy. Training was conducted with a batch size of 4 for 200 epochs, incorporating early stopping based on validation loss to prevent overfitting.

2.4.2. Gap fraction model

For comparison, we implemented a standard gap fraction inversion model rooted in radiative transfer theory. First, the non-vegetation (gap) pixels in nadir (0°) and oblique (45°) images were identified using a SegFormer-based semantic segmentation model [42]. The gap fraction (GF) was calculated as the ratio of gap pixels to total pixels. To relate GF to GAI, we adopted the turbid-medium model based on the ellipsoidal leaf inclination distribution [43,44], expressed as:

$$\text{GF}\left(\theta \right)=e^{-K\left(\alpha ,\theta \right)\text{GAI}}$$ (1)

where K is the extinction coefficient, which depends on the viewing zenith angle θ and the average leaf angle α. Specifically, K was calculated by integrating the projection function over the leaf inclination density function following the numerical integration method described by Verhoef [45].

To train the inversion model, we generated a theoretical dataset containing 30000 combinations of GAI (ranging from 0.05 to 7 m² m⁻², log-scaled) and α (30°-80°) via Latin hypercube sampling. This dataset was randomly partitioned into a training set (80%) and an evaluation set (20%). A Gaussian process regression (GPR) model was trained on the training subset, using the simulated GF values at 0° and 45° as inputs to regress the corresponding GAI. To simulate realistic field noise and prevent overfitting, 5% white Gaussian noise was injected into the input GF values during training. The theoretical performance of this GPR-based inversion was evaluated on the held-out evaluation subset (Supplementary Note 3 and Fig. S3), demonstrating high retrieval accuracy (R² = 0.94, RMSE = 0.48 m² m⁻²) but exhibiting expected signal saturation at high canopy densities (GAI >5 m² m⁻²) due to the exponential nature of light attenuation.

2.5. Broad-sense heritability and genotypic correlations

Both the multimodal approach and the GF model were applied to RGB image series collected in the three wheat experiments (YL23, YL24 and BM24; Table 1) to estimate GAI from emergence to flowering. From each per-plot GAI trajectory we derived two dynamic traits: (i) the GAI growth rate, obtained as the slope of a linear regression of GAI against cumulative TT during stem elongation (three to five observation samples per plot), and (ii) the maximum GAI, defined as the highest GAI estimated within the anthesis window.

To compare the estimation performance of the two methods we calculated broad-sense heritability (H²), which quantifies the proportion of phenotypic variance attributable to genetic variance and, under identical environments and genotypes, directly reflects the extent to which measurement noise dilutes genotypic differentiation. H² was computed as [46]:

$$H^2=\frac{{\sigma }_G^2}{{\sigma }_G^2+\frac{{\sigma }_{G\times E}^2}{n_E}+\frac{{\sigma }_{\epsilon }^2}{n_E{n}_R}}$$ (2)

Where ${\sigma }_G^2$, ${\sigma }_{G\times E}^2$, and ${\sigma }_{\epsilon }^2$ represent the genotypic, genotype-by-environment interaction, residual variances, respectively. $n_E$ is the number of environments, and $n_R$ is the number of replicates. Specifically, the ${\sigma }_{\epsilon }^2$ was estimated from the replicated check genotypes (described in Section 2.1), while the ${\sigma }_G^2$ and ${\sigma }_{G\times E}^2$ were estimated from the test genotypes. Here, $n_E$ = 3 is the number of environments. Since the test genotypes were unreplicated within each environment, $n_R$ was set to 1 in the denominator to reflect the expected genetic gain for the selection of test lines. In addition, we assessed the genotypic relevance of the two GAI traits by correlating their genotype best linear unbiased predictors (BLUPs) with BLUPs for grain yield and grain number; higher genotypic correlations indicate that a phenotyping method captures phenotypic differences sharing an underlying genetic basis with these yield components. This part of statistical analysis were conducted by lme4 package [47] in R studio [48].

2.6. Genome-wide association studies and gene ontology enrichment

To validate the biological relevance of the proposed phenotyping method, we conducted genome-wide association studies (GWAS) on the populations from the three experiments (YL23, YL24, and BM24; Table 1) using the 660 K SNP array to test if the estimated GAI traits are regulated by biologically plausible pathways. The genetic structure and diversity of these populations have been well documented in previous studies [49,50]. GWAS for GAI growth rate and maximum GAI was performed using a mixed linear model implemented in the rMVP package [51]. A uniform genome-wide significance threshold of p = 2.60 × 10⁻⁴ was applied (-log₁₀(p) = 3.59). This threshold was derived based on the effective number of independent markers specific to this population, as calculated in our previous study [50], to rigorously correct for multiple testing. QTLs were defined as clusters of significant markers within a 5 Mb genomic window. This window size was selected based on the genome-wide linkage disequilibrium decay (average 3.8 Mb) characterizing this specific population [50]. Genes located within each 5 Mb QTL interval were retrieved from the reference genome (IWGSC RefSeq v1.0) and defined as candidate genes. Information regarding known genes and co-localized traits within these intervals was obtained from the WheatOmics database [52] (http://wheatomics.sdau.edu.cn/). Subsequently, these candidate genes identified within these significant QTL intervals were subjected to gene ontology (GO) enrichment. Functional annotation was obtained via Triticeae-Gene Tribe [53] (http://wheat.cau.edu.cn/TGT/), and enrichment significance was assessed by the hypergeometric test implemented in clusterProfiler package [54], followed by Benjamini-Hochberg correction (FDR <0.05). Redundant GO terms were collapsed to yield a non-overlapping result set.

3. Results

3.1. Gap fraction input constrains GAI estimation accuracy

The comparative evaluation of eight deep-learning configurations on the independent test sets (XZ21 and GR18) (Fig. 2) shows that using only one-dimensional gap fraction (GF) inputs constrains the accuracy of green area index (GAI) retrieval, with the best GF-based model (45° view) reaching an R² of 0.84 and an RMSE of 0.56 m² m⁻², while the nadir GF model (0° view) suffers from saturation at high GAI (>4 m² m⁻²; R² = 0.77, RMSE = 0.67 m² m⁻²). Adding a second viewing angle or thermal time (TT) provides little additional benefit for GF models, indicating information redundancy. By contrast, models that ingest the raw RGB image markedly improve performance: a single nadir or 45° image already matches the best GF result (R² ≈ 0.84, RMSE ≈ 0.55-0.56 m² m⁻²), dual-angle images further reduce error (R² = 0.85, RMSE = 0.54 m² m⁻²), and the full multimodal scheme combining both images with TT achieves the highest precision (R² = 0.87, RMSE = 0.50 m² m⁻²), lowering the RMSE by 25% relative to the nadir GF baseline. Residuals are evenly distributed around the 1:1 line for both test experiments, demonstrating good cross-site generality. These results confirm that the richer spatial-spectral cues embedded in dual-angle RGB data, especially when complemented by phenological information such as cumulative TT, effectively overcome the dimensional limitations of a purely structural GF input and yield a more robust and accurate estimation of wheat canopy GAI.

Fig. 2.

Scatterplot evaluation of eight deep-learning configurations for wheat canopy green area index (GAI) estimation across the independent test sets XZ21 and GR18 (Table 1). Upper row: models driven by gap fraction (GF) inputs acquired at nadir (0°), oblique (45°), their combination, and their combination plus cumulative thermal time (TT). Lower row: corresponding models driven by raw RGB images. Symbols denote measured GAI, solid lines indicate the 1:1 reference line, and panel annotations report the coefficient of determination (R²) and root mean square error (RMSE).

3.2. Balanced fusion of synthetic images further sharpens multimodal accuracy

The performance of GAI estimation for the multimodal model is further improved by fusing real field images with synthetic canopy samples (Fig. 3). The horizontal axis denotes the number of real RGB images (0, 100, 300, 500) and the vertical axis denotes the number of simulated canopy images (0, 300, 500, 2000) used for training; each cell reports the resulting test set RMSE (on XZ21 and GR18). When trained solely on real images (zero synthetic), RMSE declines from 0.63 (100 RGB images) to 0.58 (300 RGB images) and 0.50 (500 RGB images). Synthetic data alone yield RMSEs of 0.81 (300 synthetic images), 0.64 (500 synthetic images) and 0.70 (2000 synthetic images), indicating moderate standalone performance. Crucially, introducing a moderate number of synthetic images substantially enhances accuracy: for example, adding 300 simulated samples to 100 real images reduces RMSE to 0.55, and combining 500 real images with 300 synthetic images achieves the lowest RMSE of 0.49. In contrast, excessive synthetic data (2000) produce diminishing or negative returns when real samples are scarce. These findings demonstrate that a balanced mixture of simulated and real images can compensate for limited field data and yield the most robust GAI predictions, whereas an overreliance on synthetic samples leads to performance saturation.

Fig. 3.

Heatmap of test set (XZ21 and GR18) root mean square error (RMSE) for wheat canopy green area index (GAI) estimation by a multimodal model trained on varying combinations of real RGB images (0-500) and synthetic canopy images (0-2000). The synthetic images are sim-to-real images obtained by “realizing” the simulated image generated by digital plant phenotyping platform (D3P) using CycleGAN.

3.3. The multimodal framework consistently surpasses the gap fraction model

3.3.1. Higher point-wise accuracy across sites

The direct comparison on the independent test set demonstrates that the multimodal (MM) model substantially outperforms the GF approach in wheat canopy GAI estimation (Fig. 4A). MM achieves an R² of 0.88 and an RMSE of 0.49 m² m⁻², whereas GF yields an R² of 0.72 and an RMSE of 0.74 m² m⁻². Data from both XZ21 (circles) and GR18 (triangles) align tightly around the 1:1 line under MM, indicating minimal bias and tighter error distribution, while GF predictions systematically underpredict at higher GAI levels and exhibit greater scatter. These findings confirm that integrating dual-angle RGB images with TT information delivers significantly more accurate and robust GAI estimates across contrasting field environments. This robustness is further corroborated by a stratified performance analysis (Supplementary Note 4 and Table S1), which shows that the multimodal model reduces RMSE by 47% (0.309 vs. 0.584 m² m⁻²) in the low-GAI regime (GAI <3 m² m⁻²) and effectively overcomes saturation in the medium-to-high range (GAI ≥3 m² m⁻², 0.723 vs. 0.964 m² m⁻²).

Fig. 4.

Comparison of the multimodal (MM, blue) and gap fraction (GF, yellow) models for estimating green area index (GAI). (A) Scatter plot of measured versus predicted GAI for the independent test set. (B) Genotype-specific GAI curves (thin lines) and cross-genotype median trajectories (thick lines) with phenotyping observations (dots) and phenological stages (jointing, flowering). Note that the highlighted intervals for jointing and flowering are schematic representations of general developmental phases and do not strictly correspond to the exact timing across the three experiments. (C). Bar plots of GAI growth rate during jointing and maximum GAI at flowering for each experiment, annotated with broad-sense heritability (H²) values for MM and GF models. (D) Genotypic correlations among wheat canopy traits of GAI growth rate during jointing (left column) and maximum GAI at flowering (right column), and two yield components of grain number (top row) and grain yield (bottom row) across 565 genotypes. Solid lines depict fitted regressions, and significance is indicated as ∗∗∗p < 0.001, ∗p < 0.05.

3.3.2. Greater genetic signal in temporal trajectories

Across all three field experiments (YL23, YL24, BM24), the multimodal (MM) model more faithfully reconstructs the temporal dynamics of wheat canopy GAI from emergence to flowering than the GF approach (Fig. 4B). In the top row, MM closely tracks the median and genotype-specific trajectories with a gradual sigmoidal increase through jointing and plateauing before flowering, whereas GF produces elevated GAI estimates during early growth and shows greater variability across genotypes. Fig. 4C shows that MM-derived growth rates during jointing are significantly higher than GF estimates in all experiments (e.g., 0.011 vs 0.009 m² m⁻² °Cd⁻¹ in YL24; p < 0.001), and maximum GAI at flowering likewise exceeds GF outputs (e.g., 6.1 vs 5.0 m² m⁻² in YL23; p < 0.001). Crucially, the broad-sense heritability (H²) of these dynamic traits is markedly greater under MM (growth rate H² = 0.20 vs 0.02; maximum GAI H² = 0.48 vs 0.13), demonstrating that integrating dual-angle RGB images with TT not only enhances estimation accuracy but also strengthens the genetic signal in derived phenotypes.

3.3.3. Stronger genotypic correlations with grain number and yield

Genotypic relationships between canopy dynamics and yield components, quantified as best linear unbiased predictors (BLUPs) from a joint mixed-model analysis of all 565 wheat genotypes, reveal that GAI traits estimated by the multimodal model (MM) are more tightly linked to agronomic performance than the corresponding GF estimates (Fig. 4D). Specifically, the BLUP for GAI growth rate during jointing shows a stronger negative correlation with grain number (MM: r = −0.30, p < 0.001; GF: r = −0.23, p < 0.001) and with grain yield (MM: r = −0.34, p < 0.001; GF: r = −0.31, p < 0.001). For maximum GAI at flowering, the MM-derived BLUP correlates more distinctly with grain number (r = −0.19, p < 0.001) and grain yield (r = −0.25, p < 0.001) than the GF counterpart (r = −0.09, p < 0.05 and r = −0.16, p < 0.001, respectively). The wider variation ranges and reduced scatter observed for MM traits are consistent with the higher heritability values reported in Fig. 4D, indicating that the multimodal framework captures genotype-specific canopy dynamics more faithfully and yields phenotypes with greater predictive power for yield determination.

3.4. Genome-wide association studies and gene ontology enrichment confirm the biological relevance of multimodal-derived traits

The Genome-wide association studies (GWAS) and subsequent gene ontology (GO) enrichment analyses based on the multimodal method reveal distinct yet complementary genetic architectures underlying the two key canopy traits: GAI growth rate during jointing and maximum GAI at flowering (Fig. 5). GWAS reveal distinct genetic architectures for GAI dynamics: loci for GAI growth rate during jointing exceeded the genome-wide significance threshold on chromosomes 12 and 16, whereas loci for maximum GAI at flowering clustered prominently on chromosomes 1, 7, 8 and 12 (Fig. 5A). Notably, the genetic signals on chromosome 12 align with well-characterized regulators of wheat plant architecture: the peak for growth rate co-localized with Rht-D1/Rht1 and TaTB1-4D, while the significant locus for maximum GAI coincided with TaBGC1-4D/TaFLO6-4D. Detailed information on these loci, including chromosomal positions, QTL intervals, and co-localizations with previously reported traits, is summarized in Table S2. The growth rate at jointing exhibits multiple dispersed QTL signals, consistent with complex polygenic regulation, and was functionally enriched for genes associated with ethylene signaling, nuclear transport, and cell-wall modifying enzymes such as β-glucosidases (Fig. 5B). In contrast, maximum GAI at flowering shows more and stronger genetic signals, particularly involving genes related to brassinosteroid signaling, microtubule dynamics, and cell-wall remodeling. Crucially, both traits converge on pathways directly linked to leaf expansion and physical cell-wall modification, particularly via β-glucosidase activity, indicating a shared biochemical and structural foundation for leaf area determination. This clear association of identified genetic loci and biological processes with well-established physiological mechanisms of plant cell elongation indirectly validates the robustness, accuracy, and biological relevance of the multimodal derived GAI estimates, thus affirming the performance and efficacy of this phenotyping method.

Fig. 5.

Genome-wide association studies (GWAS) and gene ontology (GO) enrichment of best linear unbiased predictors (BLUPs) for wheat canopy green area index (GAI) dynamics derived from the multimodal model. (A) Manhattan plots for GAI growth rate during jointing (left) and maximum GAI at flowering (right). The red dashed line indicates the genome-wide significance threshold (-log₁₀(p) = 3.59). Known genes co-localizing with significant peaks (Rht-D1/Rht1; TaTB1-4D; TaBGC-4D/TaFLO6-4D) are annotated. (B) GO enrichment bar charts for biological process (BP), cellular component (CC) and molecular function (MF) categories, plotting gene ratio (bar length) and -log₁₀(p) (color) for each trait.

4. Discussion

4.1. Synergistic fusion of dual-angle vision, temporal priors, and targeted simulation drives robust, high-precision GAI estimation

The precision of our multimodal framework derives from fusing three complementary information streams—dual-angle RGB images, cumulative thermal time (TT), and physics-based synthetic canopies—within a Vision Transformer (ViT) backbone that is indifferent to camera geometry yet sensitive to phenological context. Nadir and 45° oblique views deliver orthogonal structural cues: top-down pixels quantify ground cover, whereas slanted views reveal vertical layering and within canopy gaps. Classical gap fraction (GF) or single angle inversions treat these perspectives separately and therefore miss the 3D texture that distinguishes a dense, erect canopy from an equally green planophile one. By ingesting both images as joint tokens, the ViT learns long-range spatial dependencies, leaf texture, shadow geometry, and subtle color gradients (Fig. S4), without explicit radiative transfer equations, allowing direct GAI inference from raw appearance. Crucially, the deliberate choice of a 16 × 16 pixels patch size balances computational efficiency with the fine-grained spatial resolution required to accurately resolve the narrow leaves and highly fragmented gaps typical of wheat canopies.

TT provides a compact, biologically powerful signal on development, and its weight in the model grows as the crop nears anthesis. Sensitivity tests with synthetic images show that even a ±20 % error in TT (≈2-3 days) keeps GAI bias below 10 % across development (Fig. S5). Thus, TT makes the network more informative late in the vegetative phase while remaining tolerant of typical phenology noise. A parallel multilayer perceptron (MLP) fuses this temporal cue with the image embeddings, enabling the model to separate canopies that look identical in RGB yet stand at different stages. Combining TT with dual-angle images raises the information ceiling well beyond single angle or GF models, explaining the 15-35 % RMSE reduction we observed relative to a calibrated GF baseline (Fig. 2).

Robust performance across space, season and germplasm stems from targeted synthetic augmentation, our third pillar. Rather than millions of renders, we introduced only a few hundred radiometrically realistic canopies that spanned extremes of leaf angle, density and illumination. Although we used a different synthetic-to-real ratio from Gao et al. [25] and our renders lacked spike-growth scenes, the added virtual diversity still broadened the training distribution. However, we also observed that excessive synthetic data yielded diminishing returns, suggesting that a balanced mix is optimal for regularizing attention maps without overwhelming the realistic signal. This capability preserved accuracy (Fig. 4A and Table S1) when the model was transferred from Chinese to French experiments—a transfer representing a substantial change in soil background, solar elevation, and management (Table 1). This result echoes broader trends in agricultural computer vision [23,55].

The ViT-plus-MLP architecture is inherently modular: extra sensors or management variables could be slotted in as new branches without retraining the vision core, offering breeders a practical upgrade path. In sum, the innovation lies not in any single component but in the synergistic fusion of complementary views, temporal priors and simulation-based regularization inside a transformer that treats field heterogeneity as signal rather than noise. This design simultaneously boosts accuracy, cushions domain shifts and keeps the method scalable with off-the-shelf cameras, positioning it as a template for next-generation multimodal phenotyping [28].

4.2. High heritability translates to genetic gain and informed breeding decisions

The ultimate utility of a high-throughput phenotype in breeding depends not merely on its estimation accuracy against ground truth, but on its heritability—the extent to which phenotypic variance reflects genetic potential rather than environmental noise or measurement error (Fig. 4C). While the gap fraction (GF) method has long served as a standard proxy, its signal saturation in dense canopies artificially compresses genotypic variance, limiting the power to select superior lines. In contrast, our results demonstrate that the multimodal framework not only reduces estimation error but, more critically, amplifies the genetic signal available for selection. Specifically, the broad-sense heritability (H²) for multimodal-derived traits was consistently higher than that for GF estimates across all environments (e.g., rising from 0.13 to 0.48 for maximum GAI). This substantial increase demonstrates that the multimodal approach effectively recovers the genotypic variance often masked by saturation in optical metrics. In the context of the breeder's equation (R = H²S) [56], the elevation in heritability directly translates to higher selection accuracy and greater expected genetic gain (R).

Beyond statistical precision, the multimodal traits offer critical physiological insights for optimizing canopy architecture. The negative genetic correlations we observed between maximum GAI at flowering or GAI growth rate during jointing and both grain number and grain yield are biologically plausible and offer useful insight for breeders (Fig. 4D). Large or rapidly expanding canopies enhance early light capture, yet they also increase respiratory costs and intra-canopy shading, potentially limiting photoassimilate allocation to developing spikes [57]. Previous studies have shown that wheat genotypes with excessive mid-season leaf area often exhibit poorer grain set because lower florets abort under self-shading and nitrogen dilution [[58], [59], [60]]. The fact that our multimodal approach captured this inverse source-sink balance more strongly than the GF estimates reinforces its value in revealing trade-offs that conventional metrics might overlook. For breeding, these results suggest that optimizing, not maximizing, canopy size (e.g. selecting for moderate, well-timed leaf area) may increase grain set and yield stability [61,62].

Furthermore, these dynamic traits can serve as early phenotypic markers for stress resilience. For instance, a moderate GAI growth rate during the vegetative stage, accurately captured by our framework, acts as a proxy for a conservative water-use strategy. This is particularly valuable in rainfed environments, where restraining early canopy expansion preserves soil moisture for the critical grain-filling phase, thereby mitigating terminal drought stress. By identifying genotypes that balance source-sink dynamics rather than pursuing luxury growth, the framework facilitates the selection of varieties with greater yield stability under fluctuating climatic conditions.

4.3. Genetic architecture and biological relevance of multimodal traits

The biological robustness of the multimodal method is further validated by the specific genetic architecture underpinning the estimated traits. Beyond statistical association, the specific genetic loci identified by the multimodal model offer pinpoint validation of its phenotypic accuracy (Fig. 5A). The genomic peaks for GAI growth rate co-localized with Rht-D1/Rht1 and TaTB1-4D, master regulators of plant height [63] and tiller number [64], respectively. Their association with the growth rate trait confirms that the model successfully captures the dynamic canopy expansion driven by vertical growth and lateral branching. Conversely, the significant locus for maximum GAI coincided with TaBGC1-4D/TaFLO6-4D, a key regulator of flowering time and spike development [65]. This aligns with the physiological principle that the duration of the vegetative phase directly determines biomass accumulation and the final canopy size at flowering.

At the pathway level, gene ontology (GO) analysis of leading SNPs strengthens the interpretation that the multimodal model captures genuine structural signals rather than visual artifacts (Fig. 5B). Loci linked to maximum GAI were enriched for brassinosteroid (BR) signaling components and for microtubule- or cell-wall-polysaccharide hydrolases, consistent with BR-driven cell enlargement and wall remodeling that set the upper limit of leaf area [66]. By contrast, loci governing GAI growth rate were enriched for β-glucosidases/β-fucosidases and ethylene-responsive genes, which encode wall-loosening enzymes and their regulators that control instantaneous leaf expansion [67,68].

Compared with the multimodal method, the biological specificity of the GF baseline is less apparent. The GF method identified loci such as TaDIR-A1 and TaPIN1-6D associated with maximum GAI (Fig. S6). These genes are involved in stress defense [69] and auxin transport [70], they do not exhibit the same direct causality on macroscopic canopy architecture as the architectural genes recovered by the multimodal approach. Furthermore, the GF-derived traits showed enrichment primarily for general metabolic and housekeeping functions (e.g., mitochondrial transport, chromatin assembly; Fig. S7). This stark contrast confirms that the signal saturation inherent in the GF method masks the specific genetic variation governing canopy architecture, whereas the multimodal approach effectively recovers these mechanistic links.

This correspondence between trait definition and biological function demonstrates that the multimodal estimator is not a black box but a biologically interpretable tool that pinpoints genetic control points. More broadly, these findings illustrate the power of functional phenomics, where high-throughput traits can be connected back to gene functions and biological pathways [20,71]. By linking a complex phenotype like time-resolved GAI to genes involved in sugar metabolism and hormone signaling, we gain a more mechanistic understanding of the trait. This integrative insight, from field images to genes, exemplifies how phenomics can complement genomics in crop improvement.

4.4. Future directions: integrating G × E analysis into scalable, image-to-decision breeding pipelines

The multimodal model developed in this study does not cover post-anthesis GAI estimation. One major limitation is that reference GAI values derived from manual measurements become unreliable once leaf senescence begins, as yellow-green pixels representing senescing tissue are difficult to distinguish visually. Another challenge is that current synthetic image generation does not realistically simulate the 3D morphology of wheat spikes [72]. These issues make it difficult to quantify GAI estimation errors during the post-anthesis phase.

Although our results demonstrate that the multimodal GAI estimation model offers improved biological interpretability, fully realizing the value of canopy imaging for breeding applications requires a complete pipeline that spans field imaging, genotype-by-environment (G × E) analysis, and decision making for cultivar selection [73,74]. Since plant structure is the primary interface for environmental perception and interaction, structural traits such as GAI typically exhibit high environmental plasticity [75]. Through G × E analysis, one can identify functional traits reflected through structural traits that show higher heritability and carry more genetic information. Transforming the high environmental sensitivity of structural traits into environmentally robust genetic targets holds greater significance for breeding [76]. Recent studies have shown that multi-environment time series GWAS frameworks and tensor kernel reaction norm genomic-prediction models can process thousands of dynamic phenotypes and generate allele × environment sensitivity surfaces within hours [77,78]. Therefore, future directions should focus on embedding G × E effects into breeding decisions, forming a closed loop pipeline from image to gene. This would enable large scale, automated workflows. The multimodal model developed in this study will be made open-source to facilitate further development, such as lightweight adaptations for edge deployment. With declining computational costs, increasing data availability, and the maturation of G × E modeling tools, the connectivity between image, gene, and breeding decision will drive multi-environment field trials to evolve from static summaries produced at the end of the season to adaptive iteration during the growing season. This shift will shorten the cycle from discovery to validation for favorable alleles and empower breeders to make timely, environment-specific decisions on optimal canopy strategies during critical growth windows.

Author contributions

S.L. and F.B. conceived the original idea of the multimodal framework and the GF baseline models. C.Z. conceptualized the manuscript structure and the biological validation strategy. Z.S. developed the deep learning algorithms, performed sim2real training, and conducted model evaluations. C.Z. led the statistical analysis, trained the GF baseline models, and directed the biological validation interpretation. P.J. executed the GWAS and GO enrichment analyses. Y.G. assisted in synthetic data generation. R.Y., P.J., and Z.S. conducted field experiments and data collection. B.d.S. provided the external validation dataset. J.W. and D.H. provided the wheat germplasm and experimental sites. C.Z. and Z.S. drafted the manuscript. C.Z. led the revision process and the response to reviewers. S.L., R.H., and Y.D. supervised the project and acquired funding.

Data and code availability

The pre-trained model weights, inference code, and usage instructions are publicly available in the Hugging Face repository at https://huggingface.co/PheniX-Lab/GAI-Estimation/tree/main. The raw image and phenology datasets analyzed during the current study are not publicly available due to ongoing collaborative projects but are available from the corresponding author on reasonable request.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Given their respective roles as Editor-in-Chief and Associate Editor, Frédéric Baret and Shouyang Liu had no involvement in the peer review of this article and had no access to information regarding its peer review. Full responsibility for the editorial process for this article was delegated to another journal editor. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article: