Thermal image segmentation in weedy fields via synthetic RGB-trained models and GAN-based cross-modality alignment

Abstract

Accurate plant segmentation in thermal imagery remains a significant challenge for high throughput field phenotyping, particularly in outdoor environments where low contrast between plants and weeds and frequent occlusions hinder performance. To address this, we present a framework that leverages synthetic RGB imagery, a limited set of real annotations, and GAN-based cross-modality alignment to enhance semantic segmentation in thermal images. We trained models on 1128 synthetic images containing complex mixtures of crop and weed plants in order to generate image segmentation masks for crop and weed plants. We additionally evaluated the benefit of integrating as few as 20 real, manually segmented field images within the training process using various sampling strategies. When combining all the synthetic images with a few labeled real images, we observed a maximum relative improvement of the mean IoU score of 25% compared to the synthetic-only baseline. Cross-modal alignment was enabled by translating RGB to thermal using CycleGAN-Turbo, allowing robust template matching without calibration. Results demonstrated that combining synthetic data with limited manual annotations and cross-domain translation via generative models can significantly boost segmentation performance in complex field environments for multi-model imagery.

1. Introduction

Open-field plant phenotyping faces significant challenges in separating target crops from complex backgrounds. In weedy field environments, plants often overlap with weeds and other clutter, causing occlusions and visual confusion in imagery [1]. Crops and weeds frequently share similar colors, shapes, and textures, making it difficult for vision algorithms to distinguish between them [[2], [3], [4]]. These issues lead to missed detections or false positives when using conventional segmentation approaches in the field. This difficulty is further apparent in non-RGB imaging modalities, such as thermal imagery. Thermal imagery is commonly used to evaluate plant physiological status by capturing thermal radiation emitted by plants [5]. The resulting images can provide valuable insights into plant water relations and vascular functions [6,7]. Plant, weed, and soil temperatures can be similar under certain conditions, yielding low contrast between vegetation and background [8]. Traditional methods for thermal segmentation often involve thresholding temperature histograms or watershed algorithms to isolate canopy pixels [9]. In practice, researchers have found it advantageous to leverage co-registered RGB images to guide thermal image segmentation (e.g., masking the thermal image with an RGB-derived plant mask) [[10], [11], [12]]. Overall, accurately segmenting individual plants in weedy, outdoor conditions, especially in modalities beyond visible light, remains an open problem [[13], [14], [15]].

Environmental factors further complicate the differentiation of weeds and crops of interest in thermal images. Sunlight exposure, wind, and moisture can alter plant temperatures in the field. A prior study observed that weeds just a few feet apart can register significantly different temperatures if one is sunlit and another is shaded [16]. Another limitation is that different plant species under the same management conditions often have similar thermal signatures. For example, if a crop and a weed are both well-watered and not under stress, their canopy temperatures may be similar, making it difficult to distinguish crops from weeds [17]. In mixed vegetation, the overlapping thermal signals can confuse algorithms, leading to a higher rate of false detections. Zamani et al. [18] created a dataset of paired visible and thermal UAV images in rice paddies to classify rice plants versus weeds. Their system achieved improved accuracy when combining both modalities in distinguishing weeds from crops by using a late-fusion neural network that combined features from both the visible and thermal images.

Robust plant segmentation is a critical prerequisite for downstream phenotyping analyses. Masks of each plant allow computation of plant metrics such as canopy temperature, which in turn enable physiological traits to be inferred. For example, leaf and canopy temperatures are well-known proxies for transpiration and water status. Thermal imaging has been used to monitor drought response, as plant temperature correlates with transpiration rate and stomatal conductance [19]. Additionally, thermal measurements are used to monitor stomatal closure of grapevines [20] and also wheat [21]. However, any background pixels mistakenly included in a plant's mask can skew these measurements. Even a small fraction of soil or residue mixed into the canopy temperature calculation can drastically alter indices like the crop water stress index (CWSI), sometimes producing non-physical values [9].

Although modern machine learning tools have allowed for high-throughput prediction of plant traits from imagery data, the performance of these tools is limited by the availability of labeled data for model training [22]. There is an emphasis on expanding public image datasets for agricultural tasks, but this alone may not adequately address the complexities of specific scenarios [23]. For instance, models trained in one domain may not generalize well to another due to differences in lighting, camera angle, or plant species [2,24]. The general approach is to label new data tailored to specific domains, but this process is costly and time-consuming [25]. Manual labeling of image segmentation masks can be exceptionally time-consuming, as it requires careful identification and annotation of regions of interest in the images. Furthermore, accurate manual segmentation can be subject to human error, as it can be difficult to visually discern between object classes in images. However, by leveraging underlying representational knowledge encoded in a model that is trained on many data points, we can reduce the volume of real data required while still achieving competitive performance.

Our study aims to improve the segmentation accuracy of thermal images for cowpea plants grown in weedy, open-field conditions with a mixture of synthetic and real RGB images, as seen in Fig. 2. The proposed approach, summarized in Fig. 1, combines synthetic image generation, domain adaptation, and cross-modality alignment to reduce manual annotation burdens and improve segmentation quality in multi-modal datasets. By tackling segmentation in these challenging conditions, we seek to enable more reliable extraction of plant temperature, ultimately advancing high-throughput field phenotyping in real-world, weedy fields. Although this work focuses on plant and weed segmentation in thermal imagery, the approach is generalizable to other imaging modalities with low object class contrast. Our contributions include:

• 1.Utilized synthetic images to minimize the need for manual segmentation labels, a process that is particularly time-consuming and challenging in agriculture due to the complexities of plant structures.
• 2.Demonstrated that incorporating target-domain images into the training of a semantic segmentation model, originally developed with a synthetic dataset, can lead to decreased performance as model complexity increases. We highlight the trade-offs between model capacity and domain adaptation.
• 3.Showed that applying strategic data sampling methods when mixing a small set of real images with a larger synthetic dataset improves generalization and reduces overfitting to the synthetic domain.
• 4.Established a RGB-to-thermal image translation pipeline that enables robust cross-modal image alignment, ensuring accurate transfer of segmentation masks between modalities.

Fig. 2.

Visual examples of real and synthetic imagery dataset from different phenological stages (emergence, vegetative, flowering) and modalities (real RGB, thermal, synthetic RGB).

Fig. 1.

Overall framework to extract thermal values from the plant, excluding weeds and background. Part A) The training of the semantic segmentation model using the synthetic images (including any real images if any). Part B) The template matching of Real RGB and Real Thermal via CycleGAN-turbo. Part C) Transfer prediction masks using selected region.

1.1. Utilizing synthetic data for model training

Addressing plant segmentation in thermal imagery for weedy field environments requires overcoming two interconnected challenges. First, training accurate segmentation models demands large amounts of labeled data, yet manual annotation of complex plant-weed scenes is prohibitively expensive. Second, even with a well-trained RGB segmentation model, transferring predictions to thermal images is non-trivial due to differences in sensor characteristics and the lack of direct pixel correspondence between modalities. Our framework addresses both challenges in a unified pipeline: we leverage synthetic RGB imagery to train segmentation models with minimal manual annotation, then employ GAN-based image translation to align RGB predictions with thermal images without requiring geometric calibration.

Beyond synthetic data, alternative strategies exist for training segmentation models with limited annotation resources. Weakly supervised learning offers a promising direction by replacing expensive pixel-level masks with cheaper forms of supervision such as image-level labels or bounding boxes. For plant disease segmentation, class activation maps (CAMs) derived from classification networks can serve as pseudo-labels for training segmentation models, achieving reasonable accuracy with only weak annotations [26]. Similarly, combining CAMs with few-shot pretrained classifiers has demonstrated stronger generalization than fully supervised methods when applied to disease types not seen during training [27]. Instance-level augmentation techniques can also extend small datasets by using weak segmentation masks to crop and paste objects onto new backgrounds, yielding substantial performance gains even without ground-truth annotations [28].

Foundation models present another compelling avenue for reducing annotation requirements. Large-scale vision models such as the Segment Anything Model (SAM) enable zero-shot segmentation guided by simple prompts like bounding boxes or points, eliminating the need for task-specific training data altogether [29]. In plant phenotyping, researchers have combined SAM with vision-language models to achieve hierarchical plant segmentation without any additional training, outperforming supervised methods in cross-domain generalization [30]. PlantCaFo integrates prior knowledge from multiple pretrained models including CLIP and DINO, achieving over 93% accuracy with only 16 labeled samples per class on plant disease datasets [31].

Recent work has demonstrated that synthetic data can provide virtually unlimited labeled examples for training models, which significantly reduces the need for costly manual annotation [32]. Additionally, it can reduce or even completely prevent incorrect labels. This strategy has been applied in domains like plant image analysis. For example, a recent leaf segmentation study showed that a model trained largely on synthetic images achieved accuracy comparable to using real images, especially when only limited real data was available [33]. However, purely synthetic datasets often suffer from limited diversity and a persistent domain gap, meaning models trained on them may not generalize well to real-world inputs [34]. If the synthetic data lacks sufficient variety, models risk overfitting to synthetic-specific patterns and may fail to capture the complexity of real imagery [[35], [36], [37]]. To mitigate these issues, researchers have developed techniques to make synthetic data more realistic and to bridge the synthetic-to-real gap [38,39]. For instance, unsupervised domain adaptation via adversarial style transfer or recent diffusion models can render more photorealistic synthetic images while preserving labels, yielding improved performance over training on unadapted synthetic data [40,41]. Another effective approach is to mix synthetic and real data during training. By pretraining a model on abundant synthetic data and then fine-tuning it with a small set of real samples, real-world variability can be injected and significantly alleviate domain biases [42].

While synthetic data and domain adaptation strategies address the challenge of training segmentation models with limited annotations, a separate challenge arises when applying these RGB-trained models to thermal imagery for phenotyping applications.

1.2. Cross-modality segmentation

Cross-modality segmentation with RGB and thermal imagery is challenging because each sensor captures different scene cues. This means that gradients and textures present in a visible image may not appear in the thermal image of the same scene (and vice versa), yielding few direct correspondences for feature matching. In practice, even if one finds common structures (e.g., object edges), aligning RGB and thermal pixels is problematic unless the sensors are carefully calibrated. Separate RGB and thermal cameras usually have different intrinsic parameters (focal length, distortion, resolution) and a relative pose offset, resulting in parallax and scale differences that break pixel-wise alignment [43]. Without calibration, a hot object might project to different image locations in the two modalities, compounding the difficulty of cross-domain segmentation.

One practical strategy to bridge this gap is to translate RGB images into the thermal domain using image-to-image translation. Several approaches exist for this task. Paired methods such as Pix2Pix [44] require aligned image pairs during training, which are often difficult to obtain for RGB-thermal datasets. Diffusion-based models have recently emerged as powerful alternatives, offering improved image quality and diversity [45], though they typically require longer inference times. Variational autoencoders (VAEs) provide another option but often produce blurrier outputs compared to adversarial methods [46]. Among these, CycleGAN [47] and its variants have become popular for RGB-to-thermal translation because they can learn mappings from unpaired data, eliminating the need for pixel-aligned training pairs. By converting an RGB frame into a generated thermal image, one can create pseudo “paired” inputs of the same modality. For example, Wang et al. [48] generated cross-modal paired images for person re-identification, using a generative adversarial network (GAN) to produce a thermal version of each visible image so that features can be matched instance-wise across modalities. Recent works add explicit structural constraints or advanced architectures to preserve thermal characteristics, for instance, an edge-guided multi-domain translator (EMRT) and CycleGAN-Turbo to achieve more faithful thermal renditions of RGB content [[49], [50], [51]]. On the other hand, researchers are also fusing modalities at the feature level using transformers to implicitly align RGB-thermal information. Helvig et al. [52] introduced a cross-attention transformer for IR-visible object detection that learns correspondences between the two modalities’ feature maps. Alignment via this method may not be necessary in our case since the scenes between the RGB and thermal imagery are not vastly different.

2. Materials and methods

2.1. Overview

First, we leverage a simulation framework to generate synthetic RGB imagery and pixel-perfect semantic masks for crops and weeds. In addition, we complement it with a smaller set of manually labeled real images collected in a cowpea breeding trial using a ground-based rover equipped with visible and thermal cameras. To bridge the synthetic-to-real domain gap, we investigate multiple strategies for integrating real images into the synthetic training set, including methods called: direct injection, balanced sampling, and fine-tuning. A semantic segmentation model is trained using various encoder-decoder configurations and optimized with multi-class Dice loss [53]. Predicted RGB segmentation masks are then transferred to corresponding thermal images using a GAN-based template matching pipeline, which aligns RGB and thermal views. This enables extraction of plant temperature data while excluding weeds and background.

2.2. Dataset

2.2.1. Real images

Images were collected within a cowpea crop (Vigna unguiculata) [54] breeding trial using a ground-based rover (T4-146, Mineral, Mountain View, CA, USA) across multiple time points. The imaging system on the rover includes both visible and thermal spectrum sensors. The RGB images were acquired using a Basler acA2500-20gc (Basler AG, Ahrensburg, Germany) color area scan camera. This camera captures images at a resolution of 2592x1944 pixels. The rover utilized a closed canopy to block ambient sunlight, and illuminated plants under the canopy using a XLamp XHP70.2 6500K RGB LED lights by Cree (Cree LED, Durham, NC, USA), ensuring consistent lighting conditions during image capture. The long-wave infrared thermal imagery was collected using a FLIR Boson 640 (Teledyne FLIR, Wilsonville, Oregon) thermal camera, configured with an 8.7 mm lens with a 50°horizontal field of view (HFOV). This sensor captures images with a resolution of 640x512 pixels. More details for each camera are summarized in Table 1 and a sample image is shown in Fig. 3.

Table 1.

Camera specifications for RGB and thermal imaging.

Parameter	Basler acA2500-20gc (RGB)	FLIR Boson 640 (Thermal)
Resolution (pixels)	2592 × 2048	640 × 512
Pixel Size (μm)	4.8 × 4.8	12 x 12
Focal Length (mm)	Lens-dependent	8.7
Field of View (HFOV)	Lens-dependent	50°
Shutter Type	Global shutter	Rolling (thermal)
Spectral Band	Visible (RGB)	8–14 μm (LWIR)
Principal Point (cx, cy)	Image center (approx.)	Image center (approx.)
Frame Rate (Hz)	Up to 21	Up to 60

Fig. 3.

Top row: Synthetic instance segmentation masks generated from Helios [55,56]. For semantic segmentation, each instance is collated into a single mask per class (plant and weed). Bottom row: Example real, manually labeled semantic segmentation masks for plant (blue) and weed (red) classes.

Image data were collected at eight different time points, each spaced approximately five days apart. At each time point, image acquisition was conducted at the same time of day, yielding an average of approximately 11, 000 images per camera. For this study, a representative subset of 159 images spanning multiple phenological stages was manually annotated. The 159 labeled real images were split into training (up to 80 images), validation (22 images), and test (57 images) sets using stratified sampling across all eight acquisition time points. This ensures that each split contains images spanning the full range of phenological stages present in the dataset, allowing evaluation across diverse growth conditions rather than biasing toward a particular stage.

2.2.2. Synthetic images

A synthetic RGB imagery dataset was generated using Helios [55,56], an open source 3D plant and environment biophysical modeling framework. The current version of Helios can be downloaded at https://www.github.com/PlantSimulationLab/Helios.

Helios uses a unique, radiation modeling approach to generate synthetic images and corresponding segmentation masks. Cowpea and weed model plants were generated at growth stages matching the field imagery using the “plant architecture” plug-in of Helios, which is a parametric procedural plant architectural model. Weed species models included in the synthetic images were cheeseweed (Malva neglecta), puncturevine (Tribulus terrestris), bindweed (Convolvulus arvensis), and ground cherry (Physalis philadelphica), all of which are openly available in the Helios plant library. Each architectural parameter in the models can be specified based on a random distribution in order to create variability in the images.

The camera model requires specification of the spectral reflectivity and transmissivity of all surfaces in the simulated scene, the spectral intensity of the light source, and the camera spectral response and intrinsic parameters. Various measured surface reflectivity and transmissivity spectra are available in the Helios spectral library, which can be used to introduce variability in the image set. The light source and camera properties were specified based on the manufacturer data sheets. Segmentation masks were generated by assigning a unique integer ID to each element comprising each plant or weed in the image, which is then used by Helios to automatically generate pixel ID maps for each image, as shown in Fig. 3. A more detailed description and evaluation of the synthetic imagery generation methodology can be found in Ref. [56].

Typically, synthetic images are treated as training data based on their capability to generate a virtually limitless amount of labeled images. Additionally, plant modeling parameters can be tweaked to close the domain gap, in exchange for resources and compute time, to allow for improved model performance. The synthetic images were curated to closely match the real images, as they also include rover parts, as seen in Fig. 3. A timeseries instance segmentation dataset of 1128 images was created with the following classes: plant and weed.

Helios can simulate radiometric properties for thermal modeling, potentially generating synthetic thermal imagery and removing the need for cross-modality alignment. However, the generated synthetic thermal imagery may not capture the complex temperature distributions that could be observed in real field conditions.

2.3. Semantic segmentation

2.3.1. Injecting real images into a synthetic dataset

Training with the synthetic images enables efficient model development and virtually unlimited dataset scaling, since simulation frameworks can generate thousands of labeled samples on demand. But, at the same time, model performance remains tightly coupled to the quality and diversity of its training data [57,58]. Although our synthetic images have been curated to resemble the real images, models trained exclusively on them tend to under-perform on actual imagery due to a persistent “synthetic-to-real” domain gap [59]. Nevertheless, synthetic data still encapsulates structural and contextual features that, when paired with appropriate augmentations, can drive strong learning outcomes. For instance, Cakić et al. [1] showed that hybrid datasets with roughly 30–50% synthetic data content delivered higher detection accuracy and cross-domain robustness than purely real or purely synthetic sets.

Our training set consisted of 1128 synthetic images, while the validation and test sets each included 10 real images with manual annotations. Although Helios can produce an essentially limitless stream of synthetic data, our experiments (see Figure S1) show that generating beyond approximately 1000 images offers no measurable benefit for this task. Moreover, because the synthetic samples were designed to capture the same key visual characteristics present in the real images, the feature distributions between the two domains remain closely aligned.

An additional 15 real images were reserved to assess how model performance evolved as real data was progressively integrated into the synthetic training pool. We investigated three strategies for incorporating real data, as seen in Fig. 4: (1) direct injection, where real images are added to the synthetic dataset without sampling constraints; (2) balanced sampling, where each training batch includes at least one real image; and (3) fine-tuning, where the model pretrained on synthetic data is subsequently fine-tuned using the selected real images. For all sampling strategies, real images were augmented using a diverse set of transformations, including horizontal flipping, shift-scale cropping, Gaussian noise, perspective distortion, and various color perturbations, to enhance variability and reduce overfitting given the limited size of the dataset. The Albumentations library [60] was used to apply the augmentations to the training set and exact augmentation values can be found in Table S1.

Fig. 4.

Sampling methods when injecting real images into a synthetic dataset. Direct injection, where real images are added to the synthetic dataset without sampling constraints. Balanced sampling, where each training batch includes at least one real image. Fine-tuning, where the model pretrained on synthetic data is subsequently fine-tuned using the selected real images.

2.3.2. Model architecture - decoders

To determine which decoder design best complements the encoder architecture and target application, several representative decoder structures were evaluated. Each option offered distinct trade-offs between complexity, multi-scale feature fusion, and semantic granularity.

UNet++ [61] is an advanced encoder-decoder architecture for semantic segmentation that introduces nested and dense skip connections to bridge the semantic gap between encoder and decoder feature maps. Unlike the original U-Net, which directly forwards encoder features to the decoder, UNet++ enriches these features through intermediate convolutional blocks, making them semantically closer to their corresponding decoder layers. This nested design simplifies optimization and enhances fine-grained detail recovery. Additionally, UNet++ integrates deep supervision, allowing for flexible model pruning and improving performance across multiple segmentation tasks, especially in the medical imaging domain. However, this nested design comes with the cost of increased computational overhead and memory usage, potentially limiting resource constrained hardware [62].

Feature Pyramid Network (FPN) [63] is designed to efficiently handle multi-scale object recognition by leveraging the inherent pyramidal hierarchy of convolutional neural networks (CNNs). It combines low-resolution, semantically strong features with high-resolution, spatially precise features through a top-down pathway with lateral connections. This design enables FPN to produce feature maps with rich semantics at all scales, significantly enhancing performance in object detection and segmentation tasks. Importantly, FPN achieves this without the computational burden of traditional image pyramids, making it both effective and efficient. The trade-off is that as the number of parameters increase, the architectural complexity can make model training difficult in custom pipelines [64].

SegFormer [65] is a transformer-based semantic segmentation framework that balances efficiency, accuracy, and robustness. It introduces a hierarchical Transformer encoder [66] without positional encodings, which produces multi-scale features that generalize well across varying input resolutions. SegFormer also fuses features from multiple stages, combining local and global context effectively. This simple yet powerful architecture avoids heavy computation while achieving state-of-the-art performance on several benchmarks, making it ideal for both real-time and high-accuracy applications. The limitation is that scaling model size requires careful hyperparameter fine-tuning and dropout [67].

2.3.3. Model architecture - encoders

We hypothesize that in this relatively constrained binary segmentation task, lightweight architectures are less prone to overfitting on synthetic data. These smaller models may be better suited to integrating the small signal introduced by real data, perhaps due to stronger gradient responses or more stable optimization behavior. This aligns with findings from Brigato and Iocchi [68], who demonstrated that low-complexity networks generalize better than high-capacity architectures when training data is limited.

EfficientNet [69] is a family of CNNs designed for computational efficiency while maintaining accuracy. The key innovation is a compound scaling method that uniformly scales the network's depth, width, and input resolution using fixed coefficients. The compound method scales depth, width, and resolution together according to user-defined coefficients, ensuring balanced model scaling with image resolution under a fixed computational budget. This stands in contrast to traditional approaches that scale only one dimension of the model, often leading to diminishing returns. EfficientNet models are built upon a baseline architecture called EfficientNet-b0, which was discovered using neural architecture search. Larger models (e.g., b1–b8) are then derived by scaling b0 using the compound method, offering a principled way to trade off between accuracy and computational cost. We test the b0, b5 and b8 variants as our encoders to evaluate how the number of trainable parameters influence model performance and domain adaptation.

2.3.4. Loss function

Before evaluation, the model is trained using a multi-class Dice loss. This loss function directly optimizes spatial overlap between predicted and ground truth regions across all classes, making it especially suitable for imbalanced segmentation tasks. For each training batch, the network outputs unnormalized logits, which are passed directly to the Dice loss. The multi-class Dice loss is computed by comparing each class prediction channel against the corresponding ground truth binary mask and averaging the per-class Dice scores. The loss is defined as:

$${\mathcal{L}}_{\text{Dice}}=1-\frac{1}{C}\sum _{c=1}^C\frac{2{\sum }_i{p}_{i,c}\cdot g_{i,c}+ϵ}{{\sum }_i{p}_{i,c}^2+{\sum }_i{g}_{i,c}^2+ϵ}$$ (1)

where:

• •C is the number of classes,
• •p_i,c is the predicted softmax probability for pixel i belonging to class c,
• •g_i,c is the corresponding one-hot ground truth label (0 or 1),
• •ϵ is a small constant to prevent division by zero.

This formulation encourages the model to maximize overlap between predicted and ground truth masks for each class. Compared to cross-entropy [70,71], Dice loss is more robust to class imbalance, as it directly penalizes both false positives and false negatives in its overlap computation.

2.3.5. Evaluation criteria and metrics

To quantitatively evaluate segmentation performance, we use three standard metrics: Intersection over Union (IoU) [72], Dice coefficient, and Pixel Accuracy [73]. These metrics are calculated per pixel across the entire dataset, making them well-suited for evaluating dense prediction tasks like semantic segmentation. Each pixel is treated as an independent classification decision, assigned to a single class label.

For a multi-class segmentation problem with C classes, the confusion statistics are computed for each class c ∈ {1, …, C} by comparing predicted pixel labels ${\hat{y}}_{ij}$ and ground truth labels y_ij over all pixel locations (i, j) in an image:

• •True Positives (TP): Pixels correctly predicted as class c.
• •False Positives (FP): Pixels incorrectly predicted as class c (they belong to another class in ground truth).
• •False Negatives (FN): Pixels belonging to class c in the ground truth but predicted as another class.
• •True Negatives (TN): Pixels correctly predicted as not belonging to class c.

IoU measures the agreement between the predicted and ground truth regions, penalizing both oversegmentation and undersegmentation. For class c, it is computed as:

$${\text{IoU}}_c=\frac{{\text{TP}}_c}{{\text{TP}}_c+{\text{FP}}_c+{\text{FN}}_c+ϵ}$$ (2)

where ϵ is a small constant (e.g., 1e⁻⁷) to ensure numerical stability. A high IoU indicates precise localization and accurate region coverage.

The Dice coefficient is closely related to IoU and can be derived from it (Dice = 2 ⋅IoU/(1 + IoU)). While both metrics measure overlap between predicted and ground truth regions, we report both. It is defined as:

$${\text{Dice}}_c=\frac{2\cdot {\text{TP}}_c}{2\cdot {\text{TP}}_c+{\text{FP}}_c+{\text{FN}}_c+ϵ}$$ (3)

Pixel Accuracy [73] evaluates the overall proportion of pixels correctly classified (either as class c or not):

$${\text{PixelAccuracy}}_c=\frac{{\text{TP}}_c+{\text{TN}}_c}{{\text{TP}}_c+{\text{FP}}_c+{\text{FN}}_c+{\text{TN}}_c+ϵ}$$ (4)

While intuitive, pixel accuracy can be misleading in imbalanced datasets dominated by background pixels, as a model predicting only background could still achieve high accuracy. We include it for completeness, but rely primarily on IoU and Dice for evaluating segmentation quality, as these metrics directly measure region overlap and are robust to class imbalance. In addition to aggregate scores, we also report per-class performance for interpretability and diagnostic purposes. For example, in our case, metrics are computed for background, weed, and plant classes. To completely evaluate model performance, we make the following comparisons: to train using only synthetic data, only real data, and using both synthetic and real data.

To evaluate the quality and realism of generated thermal images, we employ the Fréchet Inception Distance (FID) [74], a widely used metric for assessing generative models. Unlike the pixel-level segmentation metrics described above, FID operates in a learned feature space and measures the distributional similarity between real and generated images. A lower FID score indicates greater similarity between the distributions, suggesting that the generated images are more realistic and diverse. An FID of zero would indicate identical distributions. Implementation details are provided in supplementary material.

2.4. GAN-based alignment

After performing segmentation on the RGB images, we transfer the resulting masks to their corresponding thermal images. To accomplish this, we first identify the matched regions between the RGB and thermal views. We translate the RGB images into the thermal domain using CycleGAN-Turbo [51], enabling direct template matching between the generated and real thermal images. Once the matched region is located, the segmentation mask from the RGB image is mapped onto the thermal image. Notably, this approach does not require explicit geometric alignment between the RGB and thermal images, as the template matching operates effectively without camera calibration.

2.4.1. RGB-to-thermal translation

We fine-tuned CycleGAN-Turbo on our dataset to perform unpaired RGB-to-thermal image translation. CycleGAN-Turbo enables single-step image-to-image translation, offering significant speed improvements over traditional multi-step diffusion approaches. The model was initialized with the pretrained “stabilityai/sd-turbo” weights using the Transformers library provided by Huggingface [75] and fine-tuned using Low-Rank Adaptation (LoRA) [76]. Training was performed using unpaired RGB images (domain A) and thermal images (domain B) collected from our field dataset. We used a combination of cycle-consistency loss, identity loss, and adversarial loss with CLIP-based discriminators [77] to ensure structure preservation during translation. The thermal images were normalized to 8-bit grayscale by min-max scaling before training. Images were preprocessed by resizing to 286 pixels and randomly cropping to 256 × 256 with horizontal flipping for augmentation.

2.4.2. Multi-scale template matching

To align generated thermal (RGB-translated) and real thermal images, we apply a multi-scale template matching algorithm using OpenCV [78]. Since the RGB and thermal cameras have different intrinsic parameters (Table 1) and a relative pose offset, direct pixel-wise correspondence is not possible. Instead, we search for corresponding RGB-translated images within a defined time frame based on image timestamps.

For each candidate pair, the RGB-translated image serves as the search image and the real thermal image serves as the template. Both images are first converted to grayscale, and the thermal image is cropped to remove the bottom 20% to exclude non-overlapping regions caused by the camera offset. The RGB-translated image is similarly cropped by 15% on the left and right edges to remove border artifacts from the translation process.

Template matching is performed across 30 discrete scales ranging from 1.0 × to 0.1 × of the original RGB-translated image size. At each scale, the RGB-translated image is resized using OpenCV's bilinear interpolation function, and normalized cross-correlation (NCC) is computed using cv2.matchTemplate with the TM_CCOEFF_NORMED method.

The scale and location yielding the highest NCC value are recorded as the best match. The corresponding bounding box coordinates are then transformed back to the original RGB image coordinate space using the inverse scale factor r = w_original/w_resized. This bounding box defines the region from which the segmentation mask is extracted and transferred to the thermal image. This approach enables effective alignment of images across modalities without requiring camera calibration or geometric rectification. The complete pipeline is summarized in Fig. 1, Fig. 5.

Fig. 5.

Overview of our GAN-based alignment and mask transfer method. We first translate the RGB image into the thermal domain. This enables template matching between the generated and real thermal images. Once the matched region is located, the segmentation mask from the RGB image can be mapped onto the target thermal image.

3. Results

3.1. Data scaling model performance with real data

Using the FPN (31M) model, weed and plant class IoU performance for k real images included into the synthetic dataset using the balanced sampling method. Horizontal lines mark test results using only real data for training (k = 80). Each data point represents the mean of six runs with different random seeds to ensure variability in real image sampling across iterations.

Significant increase in Weed Dice scores with only 20 real data points. Results are shown across different model scales and parameter sizes.

Impact of model parameter size on performance. Shown as IoU difference between sample-based training (k = 10,20,80) and baseline (k = 0). Comparisons across real data inclusion strategies, class-specific trends, and model scale. Total real images used is k = 80.

By training solely on synthetic data, our best performing model is a SegFormer with an Efficientnet-b8 encoder, achieving a mean IoU of 0.56 and mean Dice of 0.64, as seen in Table 2. A UNet++ with an Efficientnet-b8 performed similarily with a mean IoU of 0.57 and a mean Dice of 0.63. However, incorporating real images into the training set led to substantial performance gains. Using the balanced sampling method revealed stronger results compared to direct and fine-tuned, displayed in Fig. 7a. The figure also shows that the mean IoU of a UNet++ model consistently improves with the addition real images, reaching up to a 25% relative increase compared to the synthetic-only baseline. We further evaluate per-class performance using the sample-based strategy and compare it to training on the full real dataset (k = 80), as illustrated in Fig. 7b. Each data point in Fig. 7b represents the mean of six runs with different random seeds to ensure variability in real image sampling across iterations. The weed class (red) and plant class (blue) surpasses the full-data benchmark with 30 real images. This improvement can be visually observed in Fig. 6, where weed segmentation improves as k number of images increases.

Table 2.

Comparison of segmentation models using different encoders across IoU, Dice, and Pixel Accuracy metrics using a synthetic only dataset. Bold numbers are the highest mean performance values across all models for the respective metric. Each b0 variant has 6M parameters, b5 has 31M parameters, and b8 has 86M parameters.

Decoder	Encoder	IoU (%)				Dice (%)				Pixel Accuracy (%)
		Other	Weed	Plant	Mean	Other	Weed	Plant	Mean	Other	Weed	Plant	Mean
UNet++	Efficientnet-b0	0.89	0.06	0.63	0.53	0.94	0.1	0.73	0.6	0.92	0.95	0.9	0.92
	Efficientnet-b5	0.91	0.12	0.66	0.56	0.95	0.19	0.76	0.63	0.93	0.94	0.91	0.93
	Efficientnet-b8	0.91	0.13	0.66	0.57	0.95	0.19	0.76	0.63	0.93	0.94	0.91	0.93
FPN	Efficientnet-b0	0.92	0.06	0.68	0.56	0.95	0.11	0.77	0.61	0.94	0.95	0.92	0.94
	Efficientnet-b5	0.9	0.1	0.66	0.55	0.94	0.15	0.76	0.62	0.93	0.94	0.91	0.93
	Efficientnet-b8	0.91	0.09	0.65	0.55	0.95	0.15	0.75	0.62	0.93	0.95	0.91	0.93
Segformer	Efficientnet-b0	0.9	0.06	0.65	0.54	0.95	0.09	0.75	0.6	0.93	0.95	0.91	0.93
	Efficientnet-b5	0.89	0.11	0.66	0.55	0.94	0.17	0.76	0.62	0.92	0.92	0.9	0.91
	Efficientnet-b8	0.9	0.14	0.65	0.56	0.94	0.21	0.75	0.64	0.92	0.93	0.91	0.92

Fig. 7.

Using the UNet++ (31M) model, a comparison between a synthetic-only dataset and a synthetic plus k real dataset using mean IoU scores for each real data injection method: direct, sample, and finetune.

Fig. 6.

Sample predictions across different model architectures (each are 31M variants). Blue masks are plants and red masks are weeds. Each column represents the result with k number of target images included in the training set.

We also observed that incorporating just a small number of real samples can significantly boost model performance across architectures with varying scales and parameter counts. However, this improvement is not uniform: larger models with excessive parameter sizes may have diminished returns or even performance degradation. As shown in Fig. 7c, the UNet++ model with 86 million parameters experienced a decline in IoU score from 0.13 to 0.08 when real data was added. Similarly, the 86-million-parameter SegFormer model showed only a modest improvement, increasing from 0.14 to 0.17, suggesting that over-parameterization may hinder effective learning when real data is limited.

3.2. Cross-modal image generation and alignment

We leveraged CycleGAN-Turbo to synthesize thermal counterparts of our images, enabling accurate template matching against real thermal data. It took around 3 s in total to convert one RGB image to a thermal image and apply template matching on a NVIDIA RTX Titan 24 GB. The FID scores, shown in Fig. 8, were computed using image subsets from each growth stage. While FID scores notably decrease during the emergence and vegetative stages, indicating improved realism in the translated thermal images, they rise again during the flowering and late-season stages. This suggests increasing translation difficulty as canopy complexity grows. Furthermore, we observed that template matching often fails when images are dominated by vegetation with few distinctive structural features, as illustrated in Fig. 9.

Fig. 8.

CycleGAN-Turbo examples and calculated FID scores using the real thermal image as the target distribution.

Fig. 9.

Template matching examples with red denoting successful matches and magenta denoting a failed match.

4. Discussion and limitations

4.1. Data scaling

Training a segmentation model on over 1000 synthetic images established a solid representational foundation of the domain. While, the synthetic only baseline already performs sufficiently well (see Fig. 7a and c), we found that incorporating just a handful of real images, around 20 images, can yield noticeable gains in model performance. But, as we continuously incorporate labeled target data, the marginal influence of each additional real images diminishes, therefore proving that minimal manual labeling is required. These performance improvements are also partially influenced by the specific real images selected, which were curated to include a sufficient representation of both plant and weed classes. Additionally, the full set of real images spanned multiple phenological stages, suggesting that class diversity and temporal coverage both play roles in enhancing generalization.

Interestingly, models with a moderate number of trainable parameters consistently outperformed their larger counterparts. As mentioned in Section 2.3.3, we hypothesized that in this relatively constrained binary segmentation task, lightweight architectures are less prone to overfitting on synthetic data. These smaller models may be better suited to integrating the small signal introduced by real data, perhaps due to stronger gradient responses or more stable optimization behavior. Over-parameterized models, however, appeared unable to effectively leverage the sparse injection of real examples. With a more complex dataset or architecture-specific hyperparameter tuning, these larger models may yet prove advantageous.

A major limitation of this study is the small number of test samples and the imbalanced class distribution. Future work could address these constraints by evaluating performance across diverse field sites, crop types, and growth stages. Additionally, improved sampling strategies could reduce the need for manual image selection while boosting model performance with fewer training samples. Since the ultimate goal is to train segmentation models using only synthetic data, further research should focus on improving synthetic data generation pipelines to better capture the variability and complexity of real-world agricultural scenes.

Beyond this task, this domain adaptation strategy can definitely be applied to other applications and public datasets. One could pretrain on a similar dataset and then inject a handful of labeled images from the target domain to minimize the annotation effort while still reaching near-optimal performance. This can be further investigated to see how much more manual labels are needed for complex tasks. Or with a very large pretrained dataset (synthetic or not).

4.2. Multi-modal alignment

CycleGAN-Turbo enables RGB-to-thermal translation, facilitating cross-modal template matching for mask transfer. This approach is especially effective when plant features are clearly defined in both modalities. However, alignment performance degrades when thermal images are dominated by vegetation without distinctive structures, a common occurrence in dense canopies during mid-to-late season. This is expected, as thermal imagery suffers from lower spatial resolution and diminished texture detail compared to visible light. The absence of gradients and feature-rich regions in thermal views limits the reliability of pixel-wise correspondence. Nonetheless, the improved FID scores observed in early growth stages suggest that CycleGAN-Turbo effectively preserves modality-specific structure when alignment conditions are favorable.

This methodology could be expanded to other multi-modal imagery scenarios where the object of interest retains a recognizable structure, enabling robust template matching across modalities such as LiDAR or other multi-spectral data. In platforms equipped with multiple sensors, our alignment approach offers an alternative to purely timestamp-based synchronization, allowing computer vision algorithms originally developed for RGB inputs to be applied to new modalities. However, performance remains constrained when the scene lacks definitive visual features, for example, late-season canopies with heavy, uniform vegetation. So, future work should explore ways to consider correspondence under homogeneous conditions, perhaps by integrating temporal context, leveraging auxiliary sensor cues, or incorporating learned attention mechanisms to focus on even subtle structural cues.

5. Conclusion

In conclusion, this study presents an integrated framework for plant segmentation in weedy field environments that addresses two key bottlenecks in thermal phenotyping: the cost of manual annotation and the difficulty of cross-modal mask transfer. By training on over 1000 synthetic images generated with Helios and strategically injecting as few as 20 real images using balanced sampling, we achieved up to a 25% relative improvement in mean IoU over the synthetic-only baseline, demonstrating that near-optimal segmentation performance is attainable with minimal manual labeling. Our comparison of encoder–decoder configurations further revealed that moderately sized models (31M parameters) consistently outperformed larger architectures, underscoring the importance of matching model capacity to task complexity and data availability. To bridge the RGB-to-thermal gap, we employed CycleGAN-Turbo for unpaired image translation followed by multi-scale template matching, enabling segmentation mask transfer without geometric calibration. While limitations remain — including a small test set, class imbalance, and degraded alignment in dense, late-season canopies — the proposed pipeline offers a practical, scalable path toward automated extraction of thermal traits in complex field settings. More broadly, the core strategy of synthetic pretraining with minimal real-data injection and GAN-based cross-modal alignment is not restricted to agriculture and can be extended to other domains where labeled data is scarce and multi-sensor imagery is available.

Author contributions

Earl Ranario for method creation, evaluation and manuscript editing.

Brian N. Bailey for generating the synthetic images and manuscript editing.

Ismael Mayanja for proposing the research problem and manuscript editing.

Heesup Yun for contributing to alignment code and data collection.

J. Mason Earles for manuscript and methodology feedback.

Funding

This work was supported by the Bill and Melinda Gates Foundation, Project ID: INV-002830, and USDA NIFA Hatch project 7003146. Under the grant conditions of the Foundation, a Creative Commons Attribution 4.0 Generic License has already been assigned to the Author Accepted Manuscript version that might arise from this submission.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Mason Earles reports financial support was provided by Gates Foundation. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article:

Data availability

Data can currently be accessed through Huggingface [75]. The real data is found here: https://huggingface.co/datasets/earlranario/cowpea-segmentation. The synthetic data is found here: https://huggingface.co/datasets/earlranario/cowpea-synthetic-segmentation.

They can also be found through AgML1 [79], a centralized framework for agricultural machine learning. AgML provides access to public agricultural datasets for common agricultural deep learning tasks, with standard benchmarks and pretrained models, as well the ability to generate synthetic data and annotations.