A parallel resnet-transformer and cross-modal attention network for identification of rice blast resistance using UAV-based visible-infrared imagery

Abstract

UAV-based visible-infrared image fusion can offer a promising tool for rapid identification of rice blast resistance (RBR), which is significant to promote rice breeding and ensure global food security. However, effectively combining the benefits of visible and infrared modalities for RBR identification still remains a challenge. This challenge stems from the difficulty of adaptively balancing the contribution of both modalities, and preserving the integrity of local and global representation for decision making. Also, the widely employed multi-scale feature fusion brings extra computational costs. To address the aforementioned limitations, this paper proposes a parallel resnet-transformer and cross-modal attention (PRCA) network for rapid RBR identification. The hybrid architecture processes the local and global feature learning in separate branches, retaining the representation ability of local and global features to maximum extent. Later, we introduce a visible-infrared cross-modal (VICM) attention mechanism to build the integration of visible and infrared modalities, striving to balance the contribution of visible and infrared imagery. Next, we present a lightweight multi-scale feature fusion (LMFF) method to further improve the precision, introducing no extra parameters and low computational costs. Comparison results of field tests on 530 rice plots showed that the PRCA model outperformed other SOTA methods in OA (+1.85%), kappa (+3.22%), F1-score (+0.24%), and recall (+1.38%). Finally, the effectiveness of our method was validated through ablation study. The experimental results indicate that the PRCA method has potential in RBR identification as well as quantitative inversion of other phenotypic traits.

1. Introduction

Rice blast is a serious challenge to rice production, and poses a significant threat to global food security. Cultivating blast resistant varieties is the main solution to control the blast infestation, which may reduce the overuse of chemicals and ensure rice production. Traditional manual inspection is subjective and time-consuming, which is not suitable for large-scale breeding programs [1]. Recently, unmanned aerial vehicle (UAV) has emerged as a promising platform for high throughput rice phenotyping. Extensive works [[2], [3], [4], [5]] have been conducted based on the UAV-based hyperspectral imagery and multi-temporal analysis. However, capturing hyperspectral imagery requires expensive hardware that is not accessible by most users, and data collection for multi-temporal data is relatively time consuming. In general, UAV-based visible camera has a higher resolution than other sensors, but its resolution is still insufficient to depict the structural details of rice leaves. With the infection of rice blast fungus Magnaporthe oryzae, a series of physiologic responses of rice plants will change which may result in the fluctuation of canopy temperatures [6]. Infrared sensors can capture the thermal radiation from rice canopy [7], which can detect the temperature variation caused by blast infestation and provide complementary information to visible images. Therefore, the integration of UAV-based visible and infrared images has potential in rapid identification of rice blast resistance (RBR) in large-scale field conditions. However, the progress on UAV-based visible-infrared rice phenotyping remains limited.

In recent years, significant advancements have been made in visible-infrared fusion researches [8]. The infrared-visible image fusion (IVIF) methods [9,10] generate a composite image that leverage the benefits of both modalities and improve scene representation. To the high throughput rice phenotyping, the UAV-based visible and infrared sensors cannot capture clear details of rice canopy, which is not feasible to conduct the keypoint alignment and generate a fused image. Other researches conduct the feature learning of visible and infrared images at separate branches, and fuses representation from both modalities to support advanced vision tasks. The typical applications involve visible-infrared person re-identification (VI-ReID), and visible-infrared object detection. However, the methods of VI-ReID [11] focus on exploiting the modal-shared representation and bridge the modality gap, ignoring the cross-modal integration for decision-making process. The visible-infrared object detection approaches [12] usually consider the visible image as dominant modal, and overlook the contribution of infrared image during the cross-modal integration stage.

To support the researches of visible-infrared feature fusion, various novel network architectures have been exploited to validate their generalization ability to fuse both modalities for advanced vision task. Despite the great success of CNN in computer vision, the local-level processing of convolution may result in limitation of spatial correlation. The vision transformer (ViT) model the global context with its multi-head self-attention mechanism, which can capture global context and outperforms the CNN on several public datasets [13]. In contrast to CNN, ViT are more dependent on large datasets, and more likely to suffer from limited generalization on small dataset [14]. Therefore, the concept of hybrid vision transformer, also known as CNN-transformer, was proposed to leverage the benefits of both architectures [15]. In addition, the multi-scale feature fusion strategy was also utilized to enhance the feature representation [16]. However, most hybrid vision transformers build the network architecture with sequential integration, and feed the feature maps of CNN to ViT for further processing. Such architecture cannot remain the representation ability of local and global features, and fails to take full advantages of both features. Also, the employed multi-scale feature fusion strategy mainly adopt the FPN-like feature fusion with different pathways, inevitably introducing extra computational costs [17].

To address the above issues, this paper proposes a parallel resnet-transformer and cross-modal attention (PRCA) network for RBR identification using UAV-based visible-infrared imagery. First, we established a UAV-based visible-infrared platform to collect the high throughput phenotyic data in rice field conditions. Second, we presented a parallel resnet-transformer framework, processing the feature learning of CNN and ViT at separate branches, ensuring the integrity of local and global representation before feature fusion. Next, we introduced a visible-infrared cross-modal (VICM) attention mechanism, integrating the visible and infrared modalities for RBR identification. With regard to the importance of infrared images for blast detection, the designed VICM module comprised a visible dominant and an infrared dominant branch, taking full advantage and complementarity of both modalities for RBR identification. Finally, we proposed a lightweight multi-scale feature fusion (LMFF) module, which introduced no extra parameters and low computational costs. The designed LMFF can be well adapted on the parallel resnet-transformer architecture, boosting the model performance in RBR identification. The overall methodology effectively combines the visible-infrared features, local-global representation, and shallow-deep layers in a compact network architecture. In conclusion, the main contribution of this paper are outlined as follows:

• •We proposed a parallel resnet-transformer framework, preserving the integrity of local and global representation during feature learning process.
• •We introduced a visible-infrared cross-modal attention mechanism, building the integration of both modalities for RBR identification using UAV-based visible-infrared imagery.
• •We developed a lightweight multi-scale feature fusion module, combining the representation of shallow layers with deep features with no extra parameters and low computational costs.
• •Experiments on 530 rice plots demonstrate the effectiveness of the proposed methodology on RBR identification with superior performance.

The remainder of this paper is organized as follows. Section 2 elaborates the latest progress in the related research realm. Section 3 discusses the details of dataset construction on the UAV-based visible-infrared imagery for RBR identification. Section 4 gives a thorough description of the proposed PRCA method. Section 5 showcases the experimental results, including comparison with the state of the art and the ablation study, followed by the discussion on the advantages and limitation of our method. Section 6 draws the conclusion and further perspectives.

2. Related works

In this section, we outline the latest advancements in the realm of rice disease monitoring using UAV remote sensing, visible-infrared fusion schemes, and hybrid vision transformers. Later, we present the distinction of our work with the existing approaches.

2.1. Rice disease monitoring using UAV remote sensing

In recent years, UAV remote sensing has demonstrated great potential for monitoring the disease stress in rice breeding applications. Liu et al. [18] deployed the hyperspectral sensor on the UAV platform to enhance the rice blast diagnosis. Xue et al. [19] collected the UAV-based hyperspectral images and the ground-based canopy reflectance, and introduced a two-step approach to estimate the severity of rice blast. Bai et al. [20] utilized time series UAV imagery with historical meteorological data to evaluate the rice disease severity. Huang et al. [21] also used multi-temporal UAV data to assess the blight stress, with transfer learning and spatiotemporal model for phenotypic analysis. Despite the aforementioned developments, the progress of rice disease monitoring using UAV remains limited. Different from pine tree and other crops [22,23], the rice leaves are small in size with complex structure and occlusion, which cannot be clearly delineated by UAV-based optical or multispectral sensors. Several studies attempted to address this limitation by utilizing the richness of spectral information with onboard hyperspectral cameras, and other literature applied multi-temporal remote sensing to compensate the insufficiency of single-temporal UAV imagery. However, the UAV-based hyperspectral camera is too expensive, and the time-series UAV-based monitoring is time-consuming, limiting its further development and application.

2.2. Visible-infrared fusion schemes

The visible sensor records the structural and texture details, and the infrared sensor capture the thermal radiation information [24]. The fusion of the both modalities plays a crucial role in visual perception under complex and extreme scenarios. Infrared-visible image fusion (IVIF) aims to integrate the unique characteristics of visible and infrared spectra, and generate a composite image for the downstream tasks [25]. Another way is to feed both modalities into separate networks, allowing for decision-level fusion to accomplish various vision tasks [26]. The most popular applications in this domain are visible-infrared person re-identification (VI-ReID) and visible-infrared object detection. Ye et al. [27] integrated the non-local attention and generalized mean pooling to support the Re-ID tasks. Feng et al. [28] decorrelated modality-shared cues and promoted the feature learning for discriminative representation. Yu et al. [29] explored the complementary nature across two modalities for the pedestrian re-identification. Hua et al. [30] introduced multi-scale semantic correlation mining network (MSCMNet) to build correlations of visible and infrared features at different scales. Helvig et al. [31] proposed cross-attention between different modalities and achieved state of the art in object detection. Wang et al. [32] presented cross-scale dynamic convolution-driven YOLO Fusion (CDC-YOLOFusion) for visible-infrared object detection in autonomous driving, introducing cross-scale dynamic learning with a supervised kernel.

With the limited resolution of UAV images and the complexity of rice canopy, it is infeasible to align the visible and infrared modalities, limiting the deployment of IVIF methods in rice phenotyping. The VI-ReID approaches can skip the step of generating composite image and directly handle the person re-identification task. However, most of the VI-ReID researches focus on reducing the modality gap between visible and infrared images, ignoring their integration for decision making. The visible-infrared object detection applications explored correlations between modalities, but most previous works overlook the contribution of infrared image, and fail to balance the contribution of two modalities.

2.3. Hybrid vision transformers

In recent years, transformers have emerged as an effective baseline for vision tasks [33]. The multi-head self-attention mechanism of transformer can dynamically capture the global relationship of the input, and hence outperforms convolutional operations in various vision tasks [34]. However, transformers may suffer from limited generalization on small dataset as they ignore the 2D structure and local correlation [13]. Therefore, hybrid vision transformers (HVT) was proposed to combine the advantages of CNN and transformer. Dosovitskiy et al. [35] made the first attempt to feed intermediate representation of resnet into ViT with patch size of one pixel. Qian et al. [36] proposed a hybrid resnet-transformer hierarchical aggregation architecture (HAResformer) to enhance feature discrimination of visible and infrared modalities. Lu et al. [37] introduced a novel CNN-ViT hybrid network called SBCFormer, that strike the balance of accuracy and efficiency on single-board computers. Fang et al. [17] proposed a hybrid network architecture combined with multi-scale feature learning for large-scale pest identification. Sun et al. [38] proposed a bidirectional-modulation frequency-heterogeneous network (BMFH-Net) to leverage the benefits of CNN and transformer representation, and extract their task oriented characteristics.

While these literature present significant progress, most hybrid vision transformers exploit the complementarity by feeding the feature maps of CNN to the multi-head self-attention mechanism of ViT [39]. Such approaches build the correlation of CNN-transformer features, hindering the integrity of local and global representation before decision making. Other studies attempted to address this limitation with parallel integration, processing the feature extraction of CNN and transformer in separate branches [38]. However, the cross-branch interaction like signal differential still fails to decorrelate the local and global representation in feature learning.

2.4. Comparison with existing approaches

In contrast to the prevalent studies, this paper utilizes the UAV-based visible and infrared images to conduct the RBR identification. Inspired by the fact the blast stress severity can be reflected by the canopy temperature, this study established the UAV-based visible-infrared platform for RBR identification. We then introduced a parallel resnet-transformer framework to process the visible and infrared images. Different from related hybrid vision transformers, the designed hybrid architecture conducts the feature learning of resnet and ViT in separate branches, preserving the integrity of local and global features. We further proposed a visible-infrared cross-modal attention mechanism to balance the contribution of visible and infrared modalities, containing a visible dominant branch and infrared dominant branch. Finally, we present a lightweight multi-scale feature fusion method, with no extra parameters and low computational costs.

3. Materials

High quality dataset plays a crucial role in developing effective RBR identification model. To our knowledge, the public dataset on RBR identification using UAV-based visible-infrared imagery remains absent. The main challenges to create benchmark dataset are the limitation of germplasm resources, and substantial manual effort to investigate the ground truth for each variety. To address this issue, this section constructs a UAV-based visible-infrared dataset for RBR identification (UVI-RBR), covering a total of 530 rice plots and various varieties with different resistance to rice blast.

3.1. Study site and field condition

The study site is a rice field located in Liuyang city, Hunan province, China (28°25′24″N, 114°1′23″E), as shown in Fig. 1 (a). The average temperature is 28 °C, with a relative humidity of 56-99% during data collection. It is warm and humid and is likely to harbor rice blast disease. The studied field includes 530 plots with an area of 26 × 30 cm for each plot. The planting span between varieties is 20 cm, and 530 rice varieties were planted.

Fig. 1.

Geographic location of the study site and collected samples of different disease levels.

Rice seedlings were transplanted to the field on May 13, 2024, and irrigation and fertilization management were applied according to regional standards. Blast-susceptible cultivars were planted surrounding each variety, and evenly sprayed with the spore suspension mixed with blast fungi. The artificially inoculated experiment was conducted by Yuan Longping Agriculture High-Tech Company (Changsha, China). The blast-susceptible cultivars started to present symptoms around 15 days after inoculation. Each variety in the experimental site was surrounded by infected rice cultivars, which may evaluate its resistance to rice blast.

3.2. UAV campaign and ground investigation

Flight campaign was carried out with DJI M300 UAV with onboard DJI ZenMuse H20T camera system. DJI ZenMuse H20T gimbal and camera features a wide angle camera and longwave infrared thermal camera, which can capture visible and infrared images simultaneously. Specifically, the spatial resolution of visible and infrared image are 1920 × 1020 and 640 × 512. Data collection was undertaken on July 5, 2024. The UAV flight was set to 12.0 m, with the ground sample distance (GSD) of 0.41 cm/pixel for visible images and 1.07 cm/pixel for infrared images. The forward lap and side lap were 80% and 70 %, respectively. The UAV images were imported into the Agisoft PhotoScan, generating the orthophotos of the whole field. The visible and infrared orthophotos were later reoriented to align the plots with the image axes, and sub-image for each plot was manually cropped and named with its variety name. The visible and infrared image were aligned by the plot center to minimize spatial misalignment, and only the central region of the plot was cropped to avoid the edge effects.

Ground investigation was carried out by phytosanitary staff of Yuan Longping Agriculture High-Tech Company immediately after the UAV flight. A total of 5 leaves were randomly selected from the center rows of each plot for manual investigation. The severity level of each leaf can be rated to 0-5 scales corresponding to the lesion coverage from 0% to 100%, according to the national standard of China (GB/T 15790–2009) [20]. To promote the efficiency of ground investigation, the severity level of each plot were classified into 3 categories with the value of disease index (DI), following the work of [19]. The value of DI is calculated as followed:

$$DI=\frac{\sum _{i=0}^5\left(n_i{L}_i\right)}{N{L}_{\mathit{max}}}$$ (1)

Where ${\mathrm{n}}_{\mathrm{i}}$ denotes the leaf number of severity level i, ${\mathrm{L}}_{\mathrm{i}}$ refers to the severity level. N is the total number of investigated leaves, and ${\mathrm{L}}_{\max }$ represent the maximum severity level. Three disease levels (DLs) were established according to the computed DI value: DL1 (light severity, DI: 0.0-0.25), DL2 (medium severity, DI: 0.25-0.5), and DL3 (heavy severity, DI: 0.5-1.0). These disease levels correspond to the high-resistance, middle-resistance, low-resistance to rice blast.

Several samples of the UAV-based visible images, infrared images, and ground images for different severity levels were demonstrated in Fig. 1 (b). It can be seen that the UAV images are insufficient in spatial resolution to depict the details of blast infection. This problem is more severe for UAV infrared images, which can only provide a general temperature information for each rice variety. Therefore, it is challenging to perform the RBR identification using the UAV-based visible-infrared images.

3.3. Dataset split

According to the results of ground investigation, we obtained 250 DL1 samples, 228 DL2 samples, and 52 DL3 samples. Each sample included a UAV visible image, a UAV infrared image and a label indicating the disease level. The dataset was randomly split into training, validation, and testing set with a ratio of 7:1:2 according to most related researches [40]. Specifically, there are 370 training samples, 52 validation samples, and 108 testing samples. As demonstrated in Fig. 1(a), each tested variety was surrounded by the blast-susceptible cultivars manually sprayed with blast fungi. Therefore, each plot can be an isolated environment to test the variety resistance to rice blast, which can avoid the problem of spatial correlation and confirm the rationality of the dataset.

4. Methods

In this section, we present the parallel resnet-transformer and cross-modal attention network, a robust visible-infrared feature fusion method for RBR identification as well as other vision tasks. We then elaborate each component of this framework, including the overall framework of the parallel resnet-transformer architecture, visible-infrared cross-modal attention (VICM), and lightweight multi-scale feature fusion (LMFF).

4.1. Overall architecture of parallel resnet-transformer framework

The proposed parallel resnet-transformer architecture simultaneously capture the local correlation and global context of visible and infrared modalities. As illustrated in Fig. 2, the visible and infrared images are first subjected into two convolutional networks through four residual blocks, following the introduced LMFF module for efficient multi-scale feature fusion. Later, the two modalities are integrated with patch tokens, class tokens, and position embedding. The visible and infrared tokens are then sequentially fed into the proposed VICM and LMFF modules, building the integration of different modalities and multi-scale features. It is worth noticing that the feature encoding of resnet and transformer is processed in parallel mode, preserving the integrity of the local and global features. The visible representation ${\mathrm{R}}_{\mathrm{v}}$ and infrared representation ${\mathrm{R}}_{\mathrm{i}}$ generated by resnet mainly capture the local correlation of single modality, and the cross representation ${\mathrm{R}}_{\mathrm{c}}$ models the global dependency of cross modalities. All features from the aforementioned branches are concatenated, followed by a classification head to output the probability distribution of the disease grades. The feature learning and aggregation of the parallel resnet-transformer architecture can be formulated as:

$$R_v=\text{LMFF}\left(\text{residual}\_\text{encode}\left(I_v\right)\right)$$ (2)

$$R_r=\text{LMFF}\left(\text{residual}\_\text{encode}\left(I_r\right)\right)$$ (3)

$$R_c=\text{LMFF}\left(\text{VICM}\left(I_v,I_r\right)\right)$$ (4)

$$R_{fuse}=\text{concat}\left(R_v,R_r,R_c\right)$$ (5)

Fig. 2.

General architecture of the parallel resnet-transformer architecture.

4.2. Visible-infrared cross-modal attention

Suppose (H₀,W₀) is the spatial resolution of the input image, and P denotes the patch size. Visible and infrared modalities are first processed by patch embedding, resulting in shallow features of ${\mathrm{F}}_{\text{vis}}\in {\mathrm{R}}^{\mathrm{C}\times \mathrm{H}\times \mathrm{W}}$ and ${\mathrm{F}}_{\text{ir}}\in {\mathrm{R}}^{\mathrm{C}\times \mathrm{H}\times \mathrm{W}}$, where C, H=H₀/P, W=W₀/P represent the embedding size, height, and width of the representation. These features are then flattened into a set of tokens of ${\mathrm{T}}_{\text{vis}}^1\in {\mathrm{R}}^{\text{HW}\times \mathrm{C}}$ and ${\mathrm{T}}_{\text{ir}}^1\in {\mathrm{R}}^{\text{HW}\times \mathrm{C}}$. We concatenate a class token to embedded tokens and then add positional embedding.

$$T_{vis}^2=\left(T_{vis}^1,{CT}_{vis}\right)+E_{pos}$$ (6)

$$T_{ir}^2=\left(T_{ir}^1,{CT}_{ir}\right)+E_{pos}$$ (7)

Where class token and positional embedding are both learnable parameters, helping to aggregate global information and encode spatial information to embedded tokens. Using self-attention mechanism with N heads, the visible tokens and infrared tokens are reshaped to the size of $\mathrm{N}\times \left(\text{HW}+1\right)\times \left(\mathrm{C}/\mathrm{N}\right)$. After that, the ith head obtains the visible tokens ${\mathrm{T}}_{\text{vis}}^{\mathrm{i}}\in {\mathrm{R}}^{\left(\text{HW}+1\right)\times \left(\mathrm{C}/\mathrm{N}\right)}$ and infrared tokens ${\mathrm{T}}_{\text{ir}}^{\mathrm{i}}\in {\mathrm{R}}^{\left(\text{HW}+1\right)\times \left(\mathrm{C}/\mathrm{N}\right)}$ as the input of visible dominant branch and infrared dominant branch as shown in Fig. 3(a). In light of correlation between canopy temperature and blast infection, the visible and infrared signals are processed in parallel fashion, balancing the contribution of both modalities for collaborative perception.

Fig. 3.

General architecture of the visible-infrared cross-modal attention and lightweight multi-scale feature fusion.

We map the tokens to queries, keys, and values through $1\times 1$ convolution as shown below:

$$Q_{vis}^i=T_{vis}^i\otimes W_Q^{vis},K_{vis}^i=T_{vis}^i\otimes W_K^{vis},V_{vis}^i=T_{vis}^i\otimes W_V^{vis}$$ (8)

$$Q_{ir}^i=T_{ir}^i\otimes W_Q^{ir},K_{ir}^i=T_{ir}^i\otimes W_K^{ir},V_{ir}^i=T_{ir}^i\otimes W_V^{ir}$$ (9)

Next, the attention scores are computed with matrix multiplication and normalized with softmax function. The refined features are obtained by multiplying the attention matrix with its values as shown below:

$$R_{vis}^i=\text{softmax}\left(\frac{Q_{vis}^i{\left(K_{vis}^i\right)}^T}{\sqrt{d_k}}\right)V_{vis}^i$$ (10)

$$R_{ir}^i=\text{softmax}\left(\frac{Q_{ir}^i{\left(K_{ir}^i\right)}^T}{\sqrt{d_k}}\right)V_{ir}^i$$ (11)

Where ${\mathrm{d}}_{\mathrm{k}}$ denotes the dimension of the keys. The multiplication with the correlation matrix can help address the spatial mismatch of different modalities, as semantically related elements may produce larger correlation coefficient through normalization. The refined features in N heads (${\mathrm{R}}^{\mathrm{i}}\in {\mathrm{R}}^{1\times \left(\text{HW}+1\right)\times \left(\mathrm{C}/\mathrm{N}\right)},\mathrm{i}=1,2,\dots ,\mathrm{N}$) will be concatenated in the first dimension and combined with the input tokens ${\mathrm{T}}^1$ to build a residual learning structure:

$$R_{vis}=\text{concat}\left(R_{vis}^1,R_{vis}^2,\cdots ,R_{vis}^N\right)+T_{vis}^1$$ (12)

$$R_{ir}=\text{concat}\left(R_{ir}^1,R_{ir}^2,\cdots ,R_{ir}^N\right)+T_{ir}^1$$ (13)

The refined features for visible and infrared modalities will later be concatenated to obtain the cross representation, covering the integration of visible and infrared features for collaborative perception. With the VICM module, we extend the self-attention of ViT to cross-modal attention for modelling global relationship between different modalities.

4.3. Lightweight multi-scale feature fusion

The popular multi-scale feature fusion approaches mainly apply convolution operation to align the spatial resolution or channel dimension using different pathways, introducing extra model complexity and computational burden. To address this issue, this paper proposes a lightweight multi-scale feature fusion module. Considering that image classification and object detection only requires one dimensional vector as input of classification heads, this paper proposes 2D adaptively average pooling for dimensionality reduction, as shown in Fig. 3(b).

Given a layer feature ${\mathrm{C}}^1\in {\mathrm{R}}^{\mathrm{N}\times \mathrm{C}\times \mathrm{H}\times \mathrm{W}}$ in CNN, its 2D adaptively average pooling can be demonstrated below, resulting in a low-dimensional feature ${\mathrm{C}}^2\in {\mathrm{R}}^{\mathrm{N}\times \mathrm{C}}$:

$$C_{ij}^2=\left(\sum _{a=1}^H\sum _{b=1}^W{C}_{ijab}\right)/\left(a\times b\right)$$ (14)

After that, the multi-scale features can be aggregated using concatenation operation. We can see that the UAV-based visible and infrared images cannot clearly delineate the details of rice canopy, as shown in Fig. 1(b). Therefore, we consider the last two layers for feature fusion:

$$C_{fuse}=\text{concat}\left(C_{-1}^2,C_{-2}^2\right)$$ (15)

Where ${\mathrm{C}}_{-1}^2$ and ${\mathrm{C}}_{-2}^2$ denote the features of last and penultimate convolutional blocks in ResNet. For ViT, the layer feature has been flattened into lower dimension by the VICM module as can be illustrated as: $\mathrm{V}\in {\mathrm{R}}^{\mathrm{N}\times \mathrm{C}}$. Therefore, the multi-scale feature fusion can be directly processed on the representation of intermediate layers:

$$V_{fuse}=\text{concat}\left(V_{-1}^2,V_{-2}^2\right)$$ (16)

Where ${\mathrm{V}}_{-1}$ and ${\mathrm{V}}_{-2}$ represent the features of last two layers in ViT. Compared with the element-wise summation, feature concatenation can preserve the data integrity at different scales before RBR identification. To the widely adopted multi-scale channel attention module (MS-CAM), its computation of correlation matrix raises the model complexity to O(N²), which requires more computational resources than our LMFF module. From the equations above, it can be seen that the proposed LMFF module introduces no extra parameters with the computational complexity of O(N).

5. Results

5.1. Implementation details

ResNet-50 was utilized as the backbone of CNN branch, and ViT-Base/16 was used as the basic architecture of transformer branch. The batch size was maintained at 10 with 5 sub-processes to accelerate data loading, and the training was iterated for 50 epoches. The loss function was cross entropy, and the optimization was conducted by Adam optimizer with the learning rate of 1e-4. All images were resized to the 224 × 224. The validation set and testing set were used for model selection and performance evaluation. Our experiments were conducted on a NVIDIA 3090 GPUs, utilizing PyTorch as the programming environment. According to previous researches of [18,41], the overall accuracy (OA), Kappa coefficient, F1-score, Recall, Param, and FPS were used as evaluation metrics. Specifically, Kappa coefficient can be calculated according to the confusion matrix, and a large value (0.8-1.0) of this parameter indicates excellent classification performance across all categories.

5.2. Comparison with state-of-the-art methods

According to the discussion in the section of Related works, we compared our method with the state of the art in visible-infrared fusion approaches and hybrid vision transformers. There are ten methods used for comparison, including five representative works in visible-infrared fusion approaches and five state of the art methods in hybrid vision transformers. For fair comparison, the official codes of these methods were employed and carefully finetuned on our dataset.

5.2.1. Comparison with the visible-infrared fusion approaches

Table 1 gives the comparison results of our method with the SOTA visible-infrared fusion approaches. For the implementation of CAFF-DINO [31], the weights pretrained on the COCO dataset are applied and finetuned on our experiments. SGIEL [28], CSDN [29] and MSCMNet [30] are proposed to minimize the disparities of the visible and infrared modalities, failing to exploit the complementary characteristics of dual modalities for RBR identification. CAFF-DINO [31] proposes the cross-attention features fusion (CAFF) module to build the integration of the different modalities for scene perception, and raises the overall accuracy for rice disease phenotypes to 0.9167. The hierarchical self and cross-attention mechanisms significantly increase the model complexity and computational complexity, obtaining the worst results in parameter size and frame rates. PMFA [42] develops a pose-guided feature enhancement (PFE) block to enhance feature discrimination by introducing key part features, helping to eliminate the intraclass interference and obtaining an overall accuracy to 0.9259. However, the introduced modal feature alignment causes heavy computational burden, and decrease the frame rate to 15.65. The proposed PRCA method outperformed other counterparts in OA (+1.85%), Kappa (+3.22%), F1-score (+0.24%) and Recall (+1.38%) with the inference speed close to the state of the art. We argue that the VICM module effectively extracts the complementary features of visible and infrared modalities for RBR identification, and the balancing strategy takes full advantages of both modalities. Also, the LMFF method combines the advantages of shallow and deep features with a computationally efficient way, contributing to precision boost with acceptable speed.

Table 1.

Quantitative results of our method and the SOTA visible-infrared fusion approaches.

Model	OA↑	Kappa↑	F1-score↑	Recall↑	Param/M↓	FPS↑
SGIEL [28]	0.8981	0.8254	0.9123	0.9013	72.86	59.76
CAFF-DINO [31]	0.9167	0.8565	0.9250	0.9130	499.28	9.68
CSDN [29]	0.8889	0.8078	0.8648	0.8461	61.20	48.59
MSCMNet [30]	0.8426	0.7350	0.7894	0.8107	155.75	47.51
PMFA [42]	0.9259	0.8734	0.9447	0.9441	134.17	15.65
PRCA	0.9444	0.9056	0.9471	0.9579	142.99	41.26

Fig. 4 illustrate the qualitative comparisons. The category of DL2 is difficult to recognize with its similarity with other two levels in color and texture, resulting in misclassification by most methods. On the contrary, our method obtains high recall value (87.5%) with least false positives for this class, as shown in Fig. 4(a). Specifically, our method achieves an AUC value of 0.9841, 0.9780 and 1.00 for the three disease levels, demonstrating its generalization ability across all categories. The most approximate approach with our method is PMFA [42], with a slight higher AUC value for DL2 category. However, our method is nearly 3 times faster than the PMFA [42] method, highlighting that the PRCA method strike the balance among all metrics.

Fig. 4.

Confusion matrices and ROC curves of our method and the SOTA visible-infrared fusion approaches.

5.2.2. Comparison with the hybrid vision transformers

Table 2 showcases the quantitative comparison of our method and the SOTA hybrid vision transformers. It can be seen from Table 2 that our method achieves the best results on all accuracy metrics. The compared methods [[17], [35], [37]] leveraged the benefits of CNN and transformer architectures following the strategy of sequential integration, and feed the intermediate feature maps into ViT. To our knowledge, the sequential connection between CNN and transformer cannot preserve the integrity of local and global features, inevitably decreasing the generalization capability. BMFH-Net [38] utilizes parallel CNN blocks and transformer blocks to ensure the independence of the local and global feature extraction, and raises the overall accuracy to 0.9259. However, BMFH-Net [38] apply pixel-wise subtraction to the CNN and Transformer features, still generates correlation between the local pattern and global context. The proposed PRCA method build parallel branches for CNN and transformer, preserving the integrity of local and global features and obtaining the best performance in precision.

Table 2.

Quantitative results of our method and the SOTA hybrid vision transformers.

Model	OA↑	Kappa↑	F1-score↑	Recall↑	Param/M↓	FPS↑
R50ViT [35]	0.8889	0.8085	0.8918	0.8710	143.16	46.38
SBCFormer [37]	0.8889	0.8094	0.8554	0.8470	35.99	22.83
Pest-ConFormer [17]	0.9166	0.8561	0.9118	0.8906	174.54	23.70
BMFH-Net [38]	0.9259	0.8721	0.9186	0.8976	38.78	3.30
HTR-VT [39]	0.8519	0.7451	0.8647	0.8444	267.34	26.70
PRCA (ours)	0.9444	0.9056	0.9471	0.9579	142.99	41.26

Multi-scale feature fusion is an effective approach to enhance the network's ability to capture representative features, which has been widely adopted by most hybrid vision transformers. Pest-ConFormer [17] adopts a dual-path feature aggregation method to extract and fuse multi-scale features, using a top-down and bottom-up pathway with attention mechanisms. BMFH-Net [38] firstly utilizes deformable convolutions to extract high-frequency features at different scales, then carried out fine-grained weight allocation to highlight the importance of different scales. A drawback of the aforementioned multi-scale feature fusion approaches is their relatively high computational complexity, and the frame rate of Pest-ConFormer [17] and BMFH-Net [38] are only 23.70 and 3.30. In contrary, the proposed LMFF module requires no extra network parameters with a frame rate of 41.26, which is acceptable in rice phenotyping scenarios.

Fig. 5 demonstrates the visual comparison of confusion matrix and ROC curves. It can be seen from Fig. 5(a) that the PRCA method performs consistently across all disease levels. For the minor category of DL3, our method significantly surpasses other counterparts without false negative samples. We argue that the VICM module retain the representation capability of local and global features to maximum extent, helping to overcome the prediction bias for most classes. Also, it is obvious from Fig. 5(b) that our method achieves the best or close to best performance in ROC curves, with the AUC value of 0.9841, 0.9780 and 1.0000 for three disease levels. Notably, BMFH-Net [38] obtains approximate performance in AUC value. However, the multi-scale feature fusion and frequency computation significantly decrease its inference speed with the frame rate of 3.30.

Fig. 5.

Confusion matrices and ROC curves of our method and the SOTA hybrid vision transformers.

5.3. Ablation study

This paper first employed the UAV-based visible-infrared imagery for RBR identification, and then proposed three innovative modules to enhance the visible-infrared feature fusion. Therefore, we conducted ablation study on different input modalities and each proposed component to validate its effectiveness.

5.3.1. Study on different input modalities

We conducted experiments on input modalities to investigate the influences of different modalities for RBR identification. The ablation study involved three configurations: visible-only, infrared-only, and visible-infrared fusion. To evaluate the performance of RBR identification using single modal, we evaluated eight mainstream classification architectures from four literatures. These methods include four representative networks in CNN, and four popular methods in vision transformer. It is clear from Table 3 that ConvNeXt-B [43] achieves 0.8426 and 0.7307 in overall accuracy for visible-only and infrared-only modal, with the performance of visible-only identification significantly outperforming that of infrared-only. Similar trends can be observed from other methods. As illustrated in Fig. 1(b), the UAV-based infrared images contain less texture details and color information than visible images, limiting the performance of RBR identification using infrared-only modal.

Table 3.

Ablation study on different input modalities.

Modality	Method	OA↑	Kappa↑	F1-score↑	Recall↑	Param/M↓	FPS↑
Visible	ConvNeXt-B [43]	0.8426	0.7338	0.8709	0.8850	87.57	58.25
	ConvNeXt-S [43]	0.8333	0.7155	0.8762	0.8762	49.46	97.00
	EfficientNet-B0 [44]	0.8426	0.7325	0.8485	0.8588	4.01	126.32
	EfficientNet-B1 [44]	0.8519	0.7512	0.8383	0.8659	6.52	97.26
	ViT-B/16 [35]	0.8148	0.6794	0.8354	0.8139	85.80	133.68
	ViT-L/16 [35]	0.8519	0.7445	0.8644	0.8431	303.30	70.23
	Swin-T [45]	0.8611	0.7635	0.8969	0.8979	27.52	67.14
	Swin-S [45]	0.8240	0.6997	0.8568	0.8484	48.84	30.51
Infrared	ConvNeXt-B [43]	0.7037	0.4958	0.6632	0.6638	87.57	58.25
	ConvNeXt-S [43]	0.7222	0.5194	0.6985	0.6771	49.46	97.00
	EfficientNet-B0 [44]	0.7500	0.5918	0.7312	0.7904	4.01	126.32
	EfficientNet-B1 [44]	0.7315	0.5401	0.7397	0.7307	6.52	97.26
	ViT-B/16 [35]	0.6667	0.4152	0.6771	0.6350	85.80	133.68
	ViT-L/16 [35]	0.6481	0.3710	0.5831	0.5499	303.30	70.23
	Swin-T [45]	0.7037	0.4737	0.6540	0.6148	27.52	67.14
	Swin-S [45]	0.7222	0.5183	0.7071	0.6788	48.84	30.51
Visible		0.8704	0.7747	0.8626	0.8328	142.99	41.26
Infrared	PRCA	0.7778	0.6204	0.7633	0.7636	142.99	41.26
Visible-Infrared		0.9444	0.9056	0.9471	0.9579	142.99	41.26

The proposed PRCA method explores the feature fusion of visible and infrared images, and outperforms the single modal with 8.33% in overall accuracy, 14.21% in kappa, 7.09% in F1-score, and 6.00% in recall. The other 10 compared methods (Table 1, Table 2) for visible-infrared fusion approaches generally outperform the single-modal images, deriving benefits of the complementary characteristics of both modalities for RBR identification. This result demonstrates that the thermal radiation of infrared image can provide complementary information for rice blast identification, even with its insufficient spatial resolution. The combination of visible and infrared modalities increases the model complexity and inference time. However, our method still achieves a frame rate of 41.26 on a single RTX 3090, which is acceptable in rice phenotyping applications. We have also provided the single-modality benchmarks (visible-only and infrared-only) using the PRCA framework. It can be seen from Table 4 that our model slightly outperforms others in precision, demonstrating the superiority of the hybrid architecture to individual CNNs or ViTs. The integration of the visible and infrared imagery significantly outperforms the single modality in PRCA framework (+7.40% in overall accuracy, +13.09% in kappa, +8.45% in F1-score, and +12.51% in recall), which may further support our hypothesis that the visible and infrared modalities are complementary for RBR identification.

Table 4.

Ablation study on each component.

Method			OA↑	Kappa↑	F1-score↑	Recall↑	Param/M↓	FPS↑
PRTA	VICM	LMFF
			0.8889	0.8085	0.8918	0.8710	143.16	47.38
✓			0.9074	0.8425	0.9191	0.9299	143.18	43.07
	✓		0.9167	0.8570	0.9259	0.9147	143.16	47.11
✓	✓		0.9352	0.8887	0.9394	0.9280	142.98	42.08
✓	✓	✓	0.9444	0.9056	0.9471	0.9579	142.99	41.26

Kendall's tau (τ) statistics was calculated to measure the correlation of the mean canopy temperature and disease index (DI). The Kendall-tau correlation coefficient and p-value was 0.4545 and 3.2824e-12, demonstrating the agreement between the mean canopy temperature and DIs was moderate. The statistics result gives a quantitative explanation to the experimental results that the infrared-only approaches cannot obtain satisfactory accuracy, but the infrared modality can provide complementary information to visible imagery for RBR identification.

5.3.2. Study on each component

The main contribution of this paper is parallel resnet-transformer architecture (PRTA), visible-infrared cross-modal attention (VICM), and lightweight multi-scale feature fusion (LMFF). We progressively remove each component to evaluate their individual contribution to RBR identification using UAV-based visible-infrared images. Specifically, the baseline architecture is the parallel combination of the encoders of ResNet-50 and ViT-Base/16, followed by feature concatenation and several fully connected layers for RBR classification. As indicated in Table 4, integrating PRTA improves the network performance by 1.85% in OA, 3.40% in Kappa, 2.73% in F1-score, and 5.89 % in Recall. Utilizing VICM alone enhances the model in OA (+2.78%), Kappa (+4.85%), F1-score (+3.41%), and Recall (+4.37%). The combination of PRTA and VICM further facilitates the enhancement. It is noticeable that adding LMFF module yields improvement in accuracy with little increase in model complexity and inference time. The dimensional reduction and feature fusion strategy of LMFF introduces no extra parameters, and the increase in model parameters (+0.01 M) stems from the modification of final classification head with increased input features.

Fig. 6 demonstrates the Grad-CAM of our model to the identification of different disease levels. It is clear that the transformer branch focuses on the global scale, while the resnet branch captures the local pattern. This phenomenon highly corresponds to our intuition that PRTA may simultaneously capture the local feature and global context. To the samples in light severity, the visible imagery obtains more attention than the infrared modality. We suppose that the variety in light severity generally exhibit health status across the whole plot, and this state is mainly reflected by the color and texture characteristics. To the samples in medium severity, the visible and infrared feature maps in resnet locate different discriminative regions, as shown in Fig. 6(c–d) where the discriminative regions are marked in white frames. To the identification of heavy severity, the infrared imagery becomes the dominant modality. We argue that the infected blast fungus has significantly changed thermal radiation of rice plants at this stage, thus the infrared imagery may well reflect this variation. Fig. 6(g–h) gives two failure examples. The sample of Fig. 6(g) is in medium severity, where the salient region is marked in white frame. However, the visible and infrared feature maps both fail to locate this area, and misclassified this sample as light severity. The sample of Fig. 6(h) is in heavy severity, where its evidence is obtained from the global scale. However, the visible and infrared feature maps only concentrate on the left-up and right-bottom regions, leading to misclassification and judge this variety to medium severity.

Fig. 6.

Grad-CAM of our method on the samples of different categories. (a-b): light severity; (c-d): medium severity; (e-f): heavy severity; (g-h) failure samples.

5.4. Advantage and limitation

An overall comparison with the SOTA methods on all metrics is given in Fig. 7. It can be concluded from Fig. 7 (a) that the main advantage of our method is the perception accuracy, which surpass other counterparts in all metrics in terms of accuracy. We owe this advantage to the fusion of local and global features, and the integration of visible and infrared representation. Recognition accuracy is the most important indicator for RBR identification in rice phenotyping, which is significant towards the selection of better rice varieties. Therefore, the comparison results on accuracy demonstrates the potential and feasibility of our method in RBR identification.

Fig. 7.

Visual comparison on the overall metrics.

The main limitation is that the model parameters of our method is relatively large, as illustrated in Fig. 7(b). To our experiments, it was found that using shared encoder for the multi-modal feature learning decreases the overall precision, and we suggested that the effective representation of diverse modalities cannot be learned by the same encoder. Therefore, we built separate encoders for the visible and infrared images to boost the accuracy. This network design increases the model parameters by nearly 2 times. To the metric of FPS, our method achieves acceptable results and outperforms about half of the compared counterparts. It can be concluded that the network efficiency in terms of latency does not correlate well with the number of parameters, which is consistent with the conclusion of SBCFormer [37].

6. Conclusion

This paper exploited the potential of UAV-based visible-infrared imagery for RBR identification. First, we developed a benchmark dataset, UVI-RBR, to support RBR identification using UAV-based visible-infrared imagery. Second, we proposed a parallel resnet-transformer framework to obtain the local and global features. The resnet and ViT streams are processed in separate branches, preserving the integrity of local and global features to maximum extent. Next, we presented a VICM method, integrating the visible and infrared modalities for RBR identification. The VICM module includes a visible dominant and an infrared dominant branch, balancing the contribution of both modalities for collaboration. Additionally, we introduced a LMFF module to combine the representation of shallow layers with deep features, introducing no extra parameters and low computational costs. The proposed method was compared with the state of the art approaches, including the visible-infrared fusion schemes, and hybrid vision transformers. Quantitative comparison demonstrated that the proposed PRCA model outperformed other counterparts with 1.85% in OA, 3.22% in kappa, 0.24% in F1-score, and 1.38% in recall. Also, the overall methodology was validated through ablation study, including the effectiveness of multi-modal fusion and each proposed modules. The progress of this study indicated that the PRCA method can serves as a fundamental tool for RBR identification, as well as the quantitative inversion for other phenotypic traits like canopy temperature and drought tolerance. In our future work, we will explore the knowledge distillation and pruning techniques to reduce the number of parameters and promote the inference speed, with the intention to minimize hardware requirements and facilitate its application.

Funding

The authors acknowledge support from the National Natural Science Foundation of China (Grant No. 62202110), the National key research and development project of China (Grant No. 2021YFD2000104-02), the Key Science and Technology Research and Development Program of Guangzhou City, China (Grant No. 2024B03J1302), the Science and Technology Planning Project of Guangzhou City, China (Grant No. 2025A04J5511), the Key Construction Discipline Research Capacity Enhancement Project of Guangdong Province, China (Grant No. 2022ZDJS014), and the Key Construction Discipline Research Capacity Enhancement Project of GPNU, China (Grant No. 22GPNUZDJS11, Grant No. 22GPNUZDJS18).

Author contributions

H.H., J.Z., and Y.T. conceived and managed the project. A.Q., Z.T., and M.F. designed the algorithm. H.H., Z.Z., and A.Y. programmed the algorithm, conducted the experiments, and generated the figures and tables. H.H., T.G., and Y.T. wrote the manuscript. H.H., Z.Z., and J.Z. collected the images. All authors revised the manuscript and approved the final version.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The dataset and codes are available at figshare: https://doi.org/10.6084/m9.figshare.30585938.