CMFGP-Net: RGBT salient object detection based on cross-modal feature global perception and multi-scale deformable convolution fusion

Abstract

Aiming at the challenges of insufficient multi-modal feature interaction, spatial-spectral misalignment, and poor robustness to target deformation and occlusion in RGB-T salient object detection, this paper proposes a detection framework (CMFGP-Net) based on cross-modal global perception and multi-scale cross-fusion. The framework adopts Swin Transformer with a hierarchical window attention mechanism as the backbone network, which significantly enhances the global context modeling capability. A cross-modal feature global perception (CMFGP) module is designed; through temporal correlation modeling and dynamic feature calibration mechanisms, it effectively alleviates the spatial and spectral misalignment between visible light and thermal infrared modalities. Furthermore, a multi-scale deformable convolutional cross-fusion (MSDCIF) module is introduced, which uses deformable convolution kernels to adaptively adjust the receptive field, enhancing the model’s adaptability to target deformation, occlusion, and complex backgrounds. Experiments on three mainstream RGB-T datasets (VT821, VT1000 and VT5000) demonstrate that CMFGP-Net outperforms existing state-of-the-art methods in key metrics such as E-measure, weighted F-measure and MAE. Ablation experiments further verify the effectiveness of each module and its contribution to performance improvement, and the model exhibits stronger robustness and generalization ability especially in complex scenarios.

Introduction

In the field of computer vision, salient object detection aims to automatically and accurately locate and segment the most visually prominent target regions from complex scenes. This technology provides crucial foundational support for various practical applications such as image compression¹, autonomous driving², image retrieval³, video surveillance⁴, and infrared guidance⁵. However, traditional single-modal salient detection methods based on visible light (RGB) often suffer from performance limitations in challenging scenarios including extreme illumination variations, complex background textures, high similarity between targets and backgrounds, or target occlusion, making it difficult to achieve stable and robust detection results.

To overcome the limitations of single-modal information representation, multi-modal fusion-based salient detection technology has emerged. In recent years, RGB-Thermal (RGB-T) salient detection, which fuses visible light and thermal infrared images, has become a research hotspot. By integrating the rich spatial texture and color information of the RGB modality with the temperature-sensitive characteristics of the thermal infrared modality, this method exhibits outstanding advantages in complex scenarios such as low-light conditions, haze, thermal crossover, and significant temperature differences between targets and backgrounds. For instance, Li et al.⁶ proposed a category fusion method based on grayscale and thermal imaging images and released the corresponding dataset, laying an important foundation for research in this field. Compared with single-modal data, RGB-T data enables effective complementarity between visible light details and thermal radiation characteristics, thereby significantly improving detection performance in low-light, occlusion, and low-contrast environments. Relevant research mainly focuses on feature alignment, adaptive fusion, and network structure design. Wang et al.⁷ first introduced manifold ranking into this task and proposed the MTMR algorithm, which treats thermal information as an additional channel and achieves adaptive fusion based on a superpixel graph model and dual ranking. Subsequent studies have further refined fusion strategies from various perspectives: Peng et al.⁸ designed TCAINet integrated with channel attention and enhanced fusion modules; Huang et al.⁹ utilized bimodal edge information to generate more accurate superpixels and constructed a multi-graph fusion model; Chen et al.¹⁰ proposed a style transfer fusion module and a multi-scale channel attention mechanism; Pang et al.¹¹ built a cross-modal aggregation unit combining self-attention and cross-attention.

In addition, a series of studies have explored novel network architectures to optimize the feature learning and fusion process. Xu et al.¹² proposed a lightweight architecture based on a convolutional capsule network; Hu et al.¹³ designed a progressive network consisting of encoding, fusion, and decoding components for RGB-D tasks; Li et al.¹⁴ and Guo et al.¹⁵ proposed an adaptive fusion network and a dual siamese network based on response-level fusion, respectively, which were applied to RGB-T object tracking. In terms of feature optimization and post-processing, Makram et al.¹⁶ adopted a two-stage method that combines a background dictionary and the CascadePSP network to optimize the boundaries of saliency maps; Zhang et al.¹⁷ proposed a multi-layer fusion tracking method based on a dynamic siamese network; D. Zhang et al.¹⁸ designed a local attention fusion module with convolution-like operations; X. Zhang et al.¹⁹ constructed a parallel shallow network for feature extraction and fusion; Z. Yu et al.²⁰ proposed a network that achieves region-selective fusion through quality assessment to suppress the negative impact of low-quality features. These studies have collectively advanced the development of RGB-T salient detection technology.

Despite the progress made by the aforementioned studies, the field of RGB-T salient detection still faces several key challenges: First, most existing methods are based on convolutional neural networks (CNNs), whose local receptive fields limit the ability to model long-range dependencies and global contextual relationships, resulting in insufficient cross-modal feature interaction. Second, due to differences in imaging mechanisms, there exists an inherent spatial-spectral misalignment between visible light and thermal infrared images, and existing fusion methods lack effective dynamic calibration mechanisms, which affects the quality of feature alignment. Third, when targets undergo deformation, partial occlusion, or are located in complex backgrounds, standard convolutions with fixed structures cannot adaptively capture geometric changes, leading to decreased detection robustness.

To address the above challenges, this paper proposes an RGB-T salient object detection network based on cross-modal global perception and multi-scale cross-fusion. The main contributions of this paper are as follows:

1. Adopting Swin Transformer with a hierarchical window attention mechanism as the backbone network instead of traditional CNNs, which enhances the global context modeling capability while ensuring computational efficiency, providing a stronger representation foundation for cross-modal feature learning.

2. Designing a cross-modal feature global perception (CMFGP) module, which alleviates spatial-spectral misalignment between modalities through temporal correlation modeling and dynamic feature calibration mechanisms, achieving adaptive feature alignment and complementary fusion.

3. Introducing a multi-scale deformable convolutional cross-fusion (MSDCIF) module, which uses deformable convolution kernels to adaptively adjust the shape of the receptive field, enhancing the model’s adaptability to target deformation, occlusion, and complex backgrounds.

4. Experiments on three mainstream RGB-T datasets demonstrate that the proposed network outperforms existing methods in multiple evaluation metrics such as E-measure, weighted F-measure, and MAE, and exhibits stronger robustness and stability especially in complex scenarios including low-light conditions and thermal crossover.

Related work

RGB-T salient object detection

Early research on RGB-T SOD mainly relied on handcrafted features and traditional machine learning methods, which localized targets through shallow cues such as color contrast and thermal radiation differences. However, these methods suffer from insufficient robustness in complex scenarios and struggle to address challenges including illumination variations and background interference. With the advancement of deep learning, methods based on Convolutional Neural Networks (CNNs) have gradually become the mainstream. For example, Tu et al.²¹ proposed to use convolutional neural networks to extract multi-level features from RGB and thermal modalities separately, and then aggregate multi-modal features via an attention mechanism to achieve more accurate salient object detection. Pang et al.¹¹ presented a hybrid attention mechanism based on global information alignment and transformation, which effectively enhanced the model’s capability to model long-range dependencies. Fu et al.²² adopted a siamese network architecture to realize deep fusion of multi-modal features. Zhou et al.²³ proposed a bidirectional fusion mechanism that interactively integrates foreground and background information to extract complete boundary representations, and adaptively embeds such boundary information into multi-level features, thereby improving the model’s detection accuracy and robustness under low-light conditions. Nevertheless, existing methods still have limitations in handling spatial-spectral misalignment and target geometric deformation. How to achieve more accurate cross-modal alignment and more flexible feature adaptation mechanisms remains an area for further exploration.

Application of transformer in salient object detection tasks

The Transformer architecture initially achieved breakthroughs in the field of natural language processing, and its core self-attention mechanism can effectively model long-range dependencies. Swin Transformer is an important variant of the Transformer series models. It adopts a pyramid architecture, gradually merging image patches at different stages to construct multi-scale feature representations, and reduces computational overhead through self-attention calculations within local windows. The shifted window mechanism enables cross-window information interaction, achieving a global receptive field while maintaining local computational efficiency. These characteristics make Swin Transformer a powerful feature extraction foundation for RGB-T SOD with wide applications. For instance, Zhang et al.²⁴ first introduced the class token mechanism into the field of RGB-D co-salient object detection, modeling intra-image and inter-image co-saliency patterns through adaptive class tokens and shared class tokens, respectively. Pang et al.²⁵ abandoned the convolution-based local fusion paradigm, reconstructed the multi-modal feature integration process from the perspective of global information alignment and transformation for the first time, and realized efficient cross-modal fusion through a sequence-to-sequence propagation mechanism based on view-mixing attention. Liu et al.²⁶ combined the hierarchical modeling capability of Swin Transformer with an edge-guided mechanism, achieving effective complementarity of cross-modal features and contour refinement through spatial alignment, channel recalibration, and edge-guided decoding.

Deformable convolution

Traditional convolutional neural networks use fixed sampling grids, whose receptive field shape and size cannot adapt to target deformation and multi-modal spatial misalignment, limiting the accuracy of feature representation and fusion. To this end, deformable convolution²⁷ was proposed. By introducing learnable offsets for each convolution sampling point, the network can dynamically adjust the receptive field shape according to input content, thereby significantly enhancing the model’s capability to model geometric deformations, which is particularly suitable for visual tasks requiring fine-grained spatial alignment. In RGB-T salient object detection (SOD), visible light and thermal infrared images often suffer from non-rigid spatial misalignment due to differences in imaging mechanisms. Directly fusing unaligned features tends to cause information confusion. Therefore, using deformable convolution to achieve adaptive alignment of cross-modal features has become a key research direction. For example, Zhu et al.²⁸ used deformable convolution to achieve multi-level feature alignment, improving the consistency of cross-modal representations; Huang et al.²⁹ designed a dedicated deformable convolution alignment module to solve the feature misalignment problem during upsampling. Although the above methods effectively alleviate the problem of spatial inconsistency, their alignment process mainly relies on geometric offset learning and fails to explicitly introduce cross-modal semantic guidance, leading to the alignment process possibly deviating from salient regions. To address this issue, this paper proposes an attention-guided deformable convolution alignment module. When deformable convolution learns sampling point offsets, this module uses attention maps generated by cross-modal interaction for guidance, so that feature alignment not only adapts to geometric deformations but also focuses on salient-related complementary regions, thereby achieving more discriminative multi-modal feature fusion and improving detection robustness in complex scenarios.

Methods

Overall architecture of the model

This paper proposes an RGB-T salient object detection framework (CMFGP-Net) based on cross-modal feature global perception and multi-scale deformable convolution fusion, whose overall architecture is illustrated in Fig. 1. Following the encoder-decoder paradigm, the model is designed to address the key challenges in RGB-T salient detection, including insufficient multi-modal feature interaction, difficult cross-modal alignment, and weak robustness to target deformation and occlusion, by constructing systematic modules.

Fig. 1.

Overall Architecture of the CMFGP-Net Model.

Firstly, the model employs a two-stream Swin Transformer as the encoder to extract features of RGB and thermal infrared images separately. Leveraging the shifted window attention mechanism, the two-stream Swin Transformer efficiently models the long-range dependencies of images, and outputs four feature levels with decreasing resolution and increasing semantic information, laying a structural foundation for subsequent multi-scale fusion.

Secondly, a cross-modal feature global perception (CMFGP) module is incorporated into the model architecture. Acting on each corresponding feature level, this module effectively alleviates the feature misalignment between RGB and thermal infrared modalities caused by differences in imaging mechanisms through temporal correlation modeling and dynamic feature calibration mechanisms. Its core function is to suppress modality-specific noise and enhance shared salient semantic information, thereby providing high-quality and aligned bimodal fusion features for the decoding stage. The core of the decoding process is the Multi-Scale Deformable Convolution Feature Fusion Module. This module receives fused features from different levels and adopts deformable convolution instead of standard convolution for feature transformation. By learning adaptive offsets of sampling points, deformable convolution can dynamically adjust the shape of the receptive field, so as to better adapt to the geometric deformation and complex contours of targets, realizing adaptive and refined fusion of multi-scale features.

Finally, the model outputs saliency maps through a top-down progressive fusion strategy: high-level semantic features are upsampled and then concatenated with adjacent low-level features rich in details, and a global context module is used to fuse multi-scale contextual information, gradually restoring spatial details. The decoder finally outputs four saliency maps at different scales, forming multi-level deep supervision to synergistically optimize the semantic consistency, boundary clarity, and regional integrity of the predicted maps. The entire model is trained in an end-to-end manner, achieving accurate and robust detection of RGB-T salient objects in complex scenarios.

Cross-modal feature global perception fusion module

To effectively alleviate the spatial and spectral feature misalignment between RGB and thermal infrared (T) modalities caused by differences in imaging mechanisms, and to achieve deep fusion of cross-modal complementary information, this paper designs a cross-modal feature global perception (CMFGP) module. As a key component connecting the dual-stream encoder and decoder, the module takes as input the RGB features and thermal infrared features from the-th layer of the Swin Transformer backbone, as illustrated in Fig. 2. For the given RGB features and thermal features , the cross-modal difference features are first computed as:

1

Fig. 2.

Structure of the cross-modal feature global perception Fusion Module.

These difference features are used to preliminarily capture complementary information between the two modalities. Next, axial bidirectional mLSTM (Bi-mLSTM) is adopted to model long-range contextual dependencies along spatial dimensions. Specifically, global average pooling is performed along the vertical direction and horizontal direction , respectively, to generate axial spatial enhancement weights:

2

Here, Bi-mLSTM is employed to process the spatial directional sequences generated by and (vertical and horizontal pooling operations), thereby enhancing the perception of continuous image structures. denotes the bidirectional mLSTM module, and is the Sigmoid activation function. Simultaneously, the spatial detail enhancement weights for each modality are computed as:

3

4

where denotes element-wise addition. Subsequently, the axial weights are fused with the modality-specific detail weights and applied to the original features:

5

6

After concatenating the enhanced bimodal features, the SimAM attention mechanism is introduced to enhance global perception capability and suppress redundant information:

7

where is a parameter-free attention computation process, as detailed in³⁰. Finally, feature reconstruction is performed via depthwise separable convolution (DSConv):

8

The CMFGP module is repeatedly applied across the four encoder layers, generating fused features at different scales. These features not only contain rich cross-modal complementary information but are also aligned and purified, providing a robust and discriminative feature foundation for the subsequent decoding stage.

Multi-scale deformable convolution fusion module

To enhance the model’s adaptability to target deformation, scale variation, and complex backgrounds, this paper designs a Multi-Scale Deformable Convolutional Feature Fusion Module (MSDCIF). This module introduces a deformable convolution mechanism to achieve dynamic alignment and fine-grained fusion of multi-scale features by adaptively adjusting the shape of the receptive field and sampling positions. Taking features from three adjacent levels , , as input, the module aims to achieve complementary enhancement between high-level semantics and low-level details through deformable convolution and an indirect guidance mechanism (Fig. 3).

Fig. 3.

Structure of the Multi-Scale Deformable Convolution Feature Fusion Module.

The core of the module lies in using the mid-level feature as a medium to guide indirect interaction between the high-level feature and the low-level feature . First, deformable convolution is applied to adaptively transform each input feature spatially:

9

Here, deformable convolution dynamically adjusts sampling positions through learned offsets, enabling the receptive field to conform to the geometric contours of the target, effectively overcoming the limitations of fixed-grid convolution in deformation and misalignment scenarios.

Subsequently, cross-layer guidance is achieved through weight generation. The processed is added to the upsampled after deformable convolution, and a high-level guidance weight is generated via the Sigmoid function to emphasize salient regions in low-level features:

10

Symmetrically, the low-level guidance weight is generated from and the downsampled to modulate high-level features:

11

Finally, the original feature details are preserved through weighted fusion:

12

By endowing the model with flexible spatial modeling capability through deformable convolution, and combining it with an indirect interaction mechanism to achieve bidirectional enhancement of semantics and details, this module significantly improves fusion robustness and localization accuracy under target deformation, occlusion, and complex backgrounds.

Loss function design

To effectively guide the model to learn the semantic and structural information of salient objects, this paper designs a multi-level supervised loss function, including binary cross-entropy loss, smoothness loss, and Dice loss, so as to achieve stepwise optimization of the prediction results.

Binary cross-entropy loss

To measure the discrepancy between the predicted saliency map and the ground truth label, we adopt the binary cross-entropy loss as the basic supervision signal. Considering that features at different levels in the decoder contribute differently to saliency modeling, we impose binary cross-entropy supervision on both the final output layer and the three intermediate feature layers:

13

where denotes the ground truth saliency map, represents the saliency prediction map output by the layer, and is the total number of pixels. This loss function encourages the model to learn the semantic consistency of salient objects across multiple scales.

Smoothness Loss

To enhance the clarity of salient boundaries and the coherence of object structures, we introduce the smoothness loss on the final prediction map :

14

where and denote the gradient operators in the horizontal and vertical directions, respectively. By constraining the gradient magnitude of the prediction map, this loss effectively suppresses noise and sharpens object contours.

Dice loss

To further optimize the integrity and internal consistency of salient regions, we adopt the Dice loss function, which achieves pixel-level alignment by measuring the overlap between the predicted region and the ground truth region:

15

This loss has good robustness to the class imbalance problem, which helps improve the model’s ability to detect small objects or weakly salient regions.

Total loss function

Integrating the above loss terms, the total loss function defined in this paper is as follows:

16

where are the weights of the binary cross-entropy loss at each layer, set as , , , and in descending order of feature depth. and are the balance coefficients of the smoothness loss and Dice loss, respectively, both set to 1.0 in the experiments. This multi-level loss design enables the model to simultaneously optimize semantic alignment, boundary clarity, and regional consistency during training, thereby improving the overall detection performance.

Experiments

Datasets

Comprehensive evaluations of this study were conducted on three benchmark datasets widely adopted in the field of RGB-T salient object detection, namely VT821³¹, VT1000³², and VT5000³³. The detailed data distributions are as follows:

• VT821: Consists of 821 pairs of strictly aligned RGB-thermal modality data with pixel-level annotations. As an early benchmark dataset, its samples cover indoor and outdoor scenes under conventional illumination conditions.

• VT1000: Expands to 1000 cross-modal sample pairs. By improving the hardware synchronization mechanism of multispectral imaging equipment, it significantly enhances the spatial alignment accuracy between RGB and thermal infrared images.

• VT5000: Currently the largest multispectral salient detection dataset, containing 5000 professionally calibrated sample pairs. It covers 11 categories of complex environmental challenges, such as haze, low light, and thermal crossover.

To verify the generalization capability of the model, the experiments strictly followed the standard evaluation protocol: 2500 sample pairs officially divided by VT5000 were used as the training set, and the remaining 2500 sample pairs, together with the full sets of VT1000 and VT821, constituted the cross-domain test set. During the training phase, random horizontal flipping, rotation, and color jittering were applied to the input data for augmentation. In the test phase, the original resolution was retained to avoid artificial intervention.

Evaluation metrics

To comprehensively evaluate model performance, this paper adopts four widely used evaluation metrics: E-measure³⁴, Weighted F-measure³⁵, F-measure³⁶, and Mean Absolute Error (MAE)³⁷. The E-measure integrates image-level statistics with pixel-level matching to holistically assess the global consistency and local accuracy of the predicted saliency maps. The Weighted F-measure alleviates the foreground-background pixel imbalance problem by introducing weights. The F-measure is the harmonic mean of precision and recall. The Mean Absolute Error directly computes the average pixel-level absolute difference between the predicted saliency map and the ground truth, reflecting the magnitude of prediction deviation. Furthermore, to assess the practical deployment efficiency of the model, this paper also reports the following efficiency metrics: number of parameters (Params, M), computational cost (FLOPs, G), inference speed (FPS), and GPU memory footprint (Mem, GB). The number of parameters and FLOPs measure the storage overhead and theoretical computational complexity, respectively. Inference speed (Frames Per Second) is measured on a single GPU and reflects the model’s real-time processing capability. GPU memory footprint refers to the peak memory consumption during forward inference and is a critical hardware resource indicator for evaluating model deployment feasibility.

Experimental settings

The experiments are conducted on the Ubuntu 22.04 operating system, with hardware configuration including an Intel(R) Xeon(R) Gold 6330 CPU (2.00 GHz), 128 GB RAM, and a single NVIDIA RTX 3090 GPU (24 GB VRAM). The software environment is based on the PyTorch 2.1.0 framework for model implementation, with CUDA 12.1 for acceleration. The model is trained using the AdamW optimizer, with an initial learning rate of 1e-4, weight decay of 0.1, batch size of 6, and a total of 200 training epochs. Input images are uniformly resized to 384 × 384 × 3 resolution, and data augmentation strategies such as random horizontal flipping, multi-scale rotation, and color jittering are applied to improve generalization. During training, model parameters are saved every 5 epochs, and mixed precision training (AMP) is enabled throughout the process to optimize GPU memory utilization. The experimental code is implemented in Python 3.8, and all tests are conducted in a single-GPU environment to ensure consistency of experimental conditions.

Comparison methods

To comprehensively evaluate the performance of CMFGP-Net, quantitative comparisons were conducted against 14 state-of-the-art methods in recent years on three widely used RGB-T salient object detection datasets (VT821, VT1000, and VT5000). These methods include ECFFNet (2021)³⁸, SwinNet (2021)³⁹, APNet (2021)⁴⁰, DCN (2022)⁴¹, MFENet (2022)⁴², TNet (2022)⁴³, CoLA (2023)⁴⁴, LSNet (2023)⁴⁵, RGBTScribble (2023)⁴⁶, EAEFNet (2023)⁴⁷, LAFB (2024)⁴⁸, UniTR (2024)⁴⁹, and MSEDNet (2024)⁵⁰.The comparative experiments adopted four commonly used evaluation metrics: E-measure (Em), weighted F-measure (wFm), F-measure (Fm), and mean absolute error (MAE). The prediction results of all comparison methods were obtained from their officially released codes or saliency prediction maps to ensure the fairness of the evaluation.

Visual comparison

Based on the qualitative comparison results shown in Fig. 4, the proposed CMFGP-Net demonstrates superior visual performance across various challenging RGB-T saliency detection scenarios. Representative samples from three public datasets are selected, covering practical challenges such as low illumination, target deformation, complex backgrounds, and multiple object coexistence. In the first group of examples, when input images suffer from imaging noise or modality defects, comparison methods such as ECFFNet and DCN produce obvious artifacts or incomplete regions in their predictions. In contrast, CMFGP-Net effectively suppresses noise interference through its cross-modal feature global perception module, achieving alignment and enhancement of cross-modal features, thereby maintaining the structural integrity and internal consistency of salient objects. In extreme low-light scenarios (as shown in the third and fourth rows), the visible modality information is severely degraded, making the thermal infrared modality the key discriminative cue. By adaptively integrating bimodal information through the multi-scale deformable fusion module, CMFGP-Net achieves accurate target localization even under near-zero illumination conditions, with significantly superior boundary sharpness and spatial continuity compared to existing methods such as LAFB and TNet. Furthermore, in cases involving small objects, multiple objects, and significant scale variations (as in the last three examples), CMFGP-Net combines the hierarchical feature extraction capability of the Swin Transformer backbone with multi-scale interaction mechanisms. This enables balanced detection of objects at different scales while maintaining model light weightness, effectively avoiding missed detection or over-segmentation issues observed in methods such as RGBT Scribble and LSNet. The qualitative comparison results indicate that CMFGP-Net exhibits strong robustness, structural integrity, and scene adaptability at the visual level.

Fig. 4.

Comparison of result visualization effects on the VT821 (Li et al., 2017), VT1000 (Tu et al., 2019), and VT5000 (Tu et al., 2020) datasets. The VT5000 dataset is available under the Creative Commons Attribution 4.0 International License (CC BY 4.0; see http://creativecommons.org/licenses/by/4.0/). All image samples are used for academic research in accordance with the datasets’ terms of use.

Comparative experiments

From the quantitative comparison results shown in Table 1, it can be observed that the proposed method achieves excellent overall performance across the three datasets, particularly outperforming existing mainstream methods on key evaluation metrics. On the VT821 dataset, our method attains the highest scores in E-measure (0.932) and weighted F-measure (0.848), while also achieving the lowest MAE (0.027) among all methods, indicating clear advantages in region localization accuracy and structural consistency. On the VT1000 dataset, our method ranks first in E-measure (0.969), weighted F-measure (0.918), and MAE (0.015), demonstrating superior detection accuracy in well-aligned spatial scenarios. On the larger and more complex VT5000 dataset, our method also leads in E-measure (0.958) and achieves the lowest MAE (0.021), further verifying its strong robustness and generalization capability across datasets with varying complexity. It is also noted that CMFGP-Net achieves slightly lower F-measure scores compared to some methods (e.g., MSEDNet achieves an F-measure of 0.931 on VT5000), but it delivers the best performance in three key metrics: E-measure, weighted F-measure, and MAE. This indicates that our model excels in salient region localization precision, boundary clarity, and structural integrity, while F-measure emphasizes the balance between recall and precision and may be more sensitive to false detections in certain complex scenarios. Moreover, our model achieves a better balance among parameter efficiency, computational complexity, and inference speed (see Table 1), making it more suitable for practical deployment. Therefore, CMFGP-Net remains superior to existing methods in terms of comprehensive performance and practicality.

Table 1.

Quantitative Comparisons on Three Widely Used Datasets. Four commonly adopted evaluation metrics are selected to assess the model performance, including E-measure (Em, the higher the better), weighted F-measure (wFm, the higher the better), F-measure (Fm, the higher the better), and mean absolute error (MAE, the lower the better). The optimal results are marked in red font.

Models	VT821	VT1000	VT5000	Params (M)	Flops (G)	FPS	Mem (GB)
	Em, wFm, Fm, MAE	Em, wFm, Fm, MAE	Em, wFm, Fm, MAE
2021_ECFFNet	0.908, 0.801, 0.865, 0.034	0.948, 0.885, 0.930, 0.021	0.912, 0.802, 0.872, 0.038	-	-	-	-
2021_SwinNet	0.928, 0.818, 0.903, 0.030	0.957, 0.894, 0.948, 0.018	0.945, 0.846, 0.915, 0.026	199.18 M	122.20G	30.96	3.37
2021_APNet	0.912, 0.792, 0.865, 0.034	0.951, 0.883, 0.930, 0.021	0.918, 0.806, 0.875, 0.035	30.38 M	46.14G	47.76	0.56
2022_DCN	0.914, 0.808, 0.817, 0.041	0.916, 0.907, 0.938, 0.020	0.913, 0.791, 0.850, 0.040	24.06 M	207.21G	24.45	0.70
2022_MFENet	0.924, 0.832, 0.876, 0.030	0.963, 0.901, 0.932, 0.019	0.931, 0.825, 0.881, 0.033	-	-	-	-
2022_TNet	0.924, 0.841, 0.904, 0.030	0.953, 0.895, 0.937, 0.021	0.930, 0.840, 0.895, 0.033	87.41 M	54.90G	50.68	1.48
2023_CoLA_main	0.912, 0.828, 0.895, 0.032	0.953, 0.897, 0.934, 0.021	0.930, 0.839, 0.891, 0.033	194.77 M	86.29G	9.76	3.25
2023_LSNet-main	0.918, 0.809, 0.872, 0.033	0.952, 0.887, 0.932, 0.023	0.920, 0.806, 0.873, 0.037	4.35 M	1.23G	41.08	0.07
2023_RGBTScribble	0.932, 0.828, 0.883, 0.038	0.961, 0.909, 0.939, 0.021	0.935, 0.829, 0.878, 0.033	66.87 M	13.06G	21.85	1.09
2023_EAEFNet	0.926, 0.830, 0.884, 0.037	0.964, 0.913, 0.941, 0.019	0.932, 0.828, 0.879, 0.031	77.97 M	17.97G	31.61	1.27
2024_LAFB	0.921, 0.811, 0.882, 0.036	0.960, 0.905, 0.937, 0.019	0.933, 0.838, 0.829, 0.032	118.76 M	166.29G	16.54	2.15
2024_UniTR	0.907, 0.808, 0.863, 0.032	0.965, 0.907, 0.945, 0.018	0.932, 0.831, 0.909, 0.031	139.54 M	134.95G	18.12	2.44
2024_MSEDNet	0.915, 0.827, 0.891, 0.040	0.947, 0.910, 0.945, 0.021	0.948, 0.895, 0.931, 0.026	94.24 M	37.98G	36.39	1.56
Ours	0.932, 0.848, 0.893, 0.027	0.969, 0.918, 0.944, 0.015	0.958, 0.866, 0.907, 0.021	177.00 M	95.49G	11.75	2.98

To further evaluate the comprehensive performance of the model, Fig. 5 presents the precision-recall (PR) curves of CMFGP-Net and mainstream comparison methods on the three RGB-T datasets. It can be observed that the PR curves of CMFGP-Net on VT821, VT1000, and VT5000 are consistently positioned closer to the top-right corner of the coordinate graph, indicating that the model maintains high precision and recall simultaneously under different confidence thresholds. The overall position and shape of the curves further corroborate the advantages of CMFGP-Net in cross-threshold stability and comprehensive detection performance. This result, from the perspective of threshold robustness, validates the effectiveness of the proposed cross-modal feature alignment and fusion framework.

Fig. 5.

Precision-recall curves of 11 methods on three datasets.

Model efficiency analysis

To comprehensively evaluate the practicality of the model, Table 1 also lists four efficiency metrics: number of parameters (Params), computational cost (FLOPs), inference speed (FPS), and GPU memory footprint (Mem). The metrics for CMFGP-Net are measured under the experimental environment described in the section “Experimental settings”; the Params, FLOPs, and FPS of the comparison methods are reliably reproduced results, while their Mem values are theoretically estimated based on their model structures. In terms of model complexity, CMFGP-Net has 177.00 M parameters, which is comparable to SwinNet (199.18 M) and CoLA (194.77 M), but its computational cost is only 95.49G FLOPs, significantly lower than DCN (207.21G) and LAFB (166.29G). This is attributed to the extensive use of depthwise separable convolution (DSConv) and the lightweight sequential design of the CMFGP module, which effectively controls computational complexity while enhancing feature interaction. In terms of actual inference performance, CMFGP-Net achieves 11.75 FPS on an RTX 3090. Although this is lower than the extremely lightweight LSNet (41.08 FPS), it achieves a balance between accuracy and speed compared to models with similar accuracy, such as UniTR (18.12 FPS) and EAEFNet (31.61 FPS). Its theoretical memory footprint is approximately 2.98 GB, which is comparable to other high-performance models and within the acceptable range of mainstream graphics cards. Therefore, through efficient architectural design, CMFGP-Net maintains leading detection accuracy while keeping computational cost, memory footprint, and inference speed at practical levels, reflecting a balanced design oriented toward real-world applications.

Ablation experiments

To validate the effectiveness of each core component in CMFGP-Net and its contribution to the final performance, this section conducts systematic ablation experiments. Starting from a baseline model, key modules are progressively introduced, and performance is evaluated on three datasets: VT821, VT1000, and VT5000.

Ablation Setting B (Baseline): Contains only the basic encoder-decoder structure, using conventional convolutions for feature extraction and fusion.

Ablation Setting B + SWIN: Replaces the backbone network with Swin Transformer featuring hierarchical window attention to enhance global context modeling capability.

Ablation Setting B + SWIN + CMFGP: Introduces the cross-modal feature global perception module on top of the previous configuration to alleviate spatial and spectral misalignment between modalities.

Ablation Setting B + SWIN + CMFGP + MSDCIF: The complete model, which further introduces the multi-scale deformable convolutional feature fusion module on top of the previous configuration to adaptively adjust the receptive field and enhance robustness to deformation and occlusion.

Based on the ablation experiment results shown in Table 2, the contribution of each module to model performance can be systematically verified. Taking Res2Net as the baseline model (B), the introduction of the Swin Transformer backbone (B+SWIN) improves the E-measure on the VT5000 dataset from 0.935 to 0.943, the weighted F-measure from 0.845 to 0.863, and reduces the MAE from 0.031 to 0.027, demonstrating that its hierarchical window attention mechanism effectively enhances global dependency modeling capability. With the further addition of the cross-modal feature global perception module (CMFGP), the model’s E-measure on the VT821 dataset rises to 0.933 and the weighted F-measure reaches 0.849, indicating that this module alleviates semantic and spatial misalignment between modalities through dynamic feature calibration. Finally, the introduction of the multi-scale deformable convolution fusion module (MSDCIF) to form the complete model (B+SWIN+CMFGP+MSDCIF) further increases the E-measure on the VT5000 dataset to 0.958, the weighted F-measure to 0.866, and significantly reduces the MAE to 0.021, demonstrating that this module enhances the model’s capability to model target deformation and complex backgrounds through adaptive receptive field adjustment. Parameter scale analysis shows that the Swin Transformer introduces approximately 105.38 M parameters, bringing significant performance gains, while the CMFGP and MSDCIF modules further optimize model performance with almost no increase in parameter count, reflecting the lightweight and efficient design of these modules. Therefore, the modules synergistically improve model performance from three perspectives—global context modeling, cross-modal alignment, and multi-scale deformation adaptation—collectively supporting CMFGP-Net in achieving state-of-the-art performance in the RGB-T saliency detection task.

Table 2.

Quantitative comparison on three widely used datasets: VT821, VT1000, and VT5000. Four commonly used evaluation metrics are employed: E-measure (Em, higher is better), weighted F-measure (wFm, higher is better), F-measure (Fm, higher is better), and mean absolute error (MAE, lower is better). The best results are highlighted in red.

Settings	Params (M)	Flops (G)	VT821	VT1000	VT5000
			Em, wFm, Fm, MAE	Em, wFm, Fm, MAE	Em, wFm, Fm, MAE
B	118.76 M	166.29G	0.922, 0.838, 0.894, 0.034	0.964, 0.912, 0.941, 0.017	0.935, 0.845, 0.892, 0.031
B + SWIN	224.14 M	205.10G	0.928, 0.845, 0.899, 0.028	0.965, 0.916, 0.942, 0.015	0.943, 0.863, 0.907, 0.027
B + SWIN + CMFGP	176.35 M	98.51G	0.933, 0.849, 0.892, 0.028	0.968, 0.916, 0.943, 0.015	0.945, 0.861, 0.906, 0.026
B+SWIN+CMFGP+MSDCIF	177.00 M	95.49G	0.932, 0.848, 0.893, 0.027	0.969, 0.918, 0.944, 0.015	0.958, 0.866, 0.907, 0.021

Loss function hyperparameter sensitivity analysis

To validate the rationality of hyperparameter selection in the loss function, we conduct systematic hyperparameter ablation experiments. The hyperparameters include the hierarchical weights , , , (corresponding to the supervision strength of feature maps at different depths), the smoothness loss weight, and the Dice loss weight . Using MAE as the primary evaluation metric on the validation set, nine different configurations are compared (Fig. 6) and ranked according to MAE values. All experiments are conducted under identical training settings.

Fig. 6.

MAE comparison of nine hyperparameter configurations.

The experimental results demonstrate the following:

Impact of hierarchical weight configuration: The results show that the configuration with decreasing weights (1.0, 0.8, 0.6, 0.5) achieves the best MAE (0.02163), outperforming the equal-weight configuration (Baseline_equal, MAE = 0.02221). This indicates that deeper features should be assigned higher supervision weights, which is consistent with the fact that deeper features contain richer and more stable semantic information. A steeper decreasing configuration (Steep_decreasing, 1.0, 0.6, 0.4, 0.2) yields a slight performance degradation (MAE = 0.02165), suggesting that moderately preserving low-level feature details benefits boundary recovery.

Impact of smoothness loss weight: When (No_smooth), MAE increases significantly to 0.02679; when (Low_smooth), MAE is 0.02287. This indicates that smoothness loss is crucial for boundary sharpness, and the full weight effectively suppresses noise while enhancing contour sharpness.

Impact of Dice loss weight: Removing Dice loss , No_dice) results in an MAE increase to 0.02297; halving the weight , Low_dice) further degrades performance (MAE = 0.02412). This validates the positive role of Dice loss in improving region integrity and alleviating class imbalance.

Necessity of auxiliary supervision: When only the final layer supervision is used (No_auxiliary, ), MAE reaches 0.02682, representing a significant performance drop. This demonstrates that multi-level supervision facilitates gradient propagation and enhances feature representation.

The hyperparameter configuration ultimately selected in this paper achieves the lowest MAE on the validation set, indicating that it strikes an optimal balance among semantic supervision, boundary smoothness, and region consistency. This selection is based on grid search experiments and has clear empirical justification.

Conclusion

This paper proposes CMFGP-Net, an RGB-T salient object detection framework based on cross-modal global perception and multi-scale deformable convolution fusion, to address the challenges of insufficient multi-modal feature interaction, spatial-spectral misalignment, and poor robustness to target deformation. By introducing the Swin Transformer backbone to enhance global context modeling capability, designing the cross-modal feature global perception (CMFGP) module to achieve dynamic feature alignment and calibration between modalities, and incorporating the Multi-Scale Deformable Convolution Fusion (MSDCIF) module to adaptively capture geometric deformations of targets, a complete RGB-T saliency detection framework is constructed. Experiments on three standard datasets, VT821, VT1000, and VT5000, demonstrate that CMFGP-Net outperforms existing mainstream methods in key metrics such as E-measure, weighted F-measure, and MAE, exhibiting particularly stronger robustness in challenging scenarios including low illumination, thermal crossover, and target occlusion. Ablation studies further validate the effective contribution of each core module to performance improvement. Meanwhile, through lightweight designs such as depthwise separable convolution, the proposed model achieves a favorable balance among parameter count, computational cost, and inference speed, providing a high-precision and practical solution for RGB-T salient object detection in complex environments.

Author contributions

The individual contributions of the authors to this work are specified as follows: Lijuan Shi (First Author): Conceptualized the research framework, designed and implemented the core model architecture, conducted the majority of the experiments, performed data analysis and result validation, and drafted the initial version of the manuscript. Haitang Li (Corresponding Author): Provided overall research guidance and supervision, contributed to the methodological design (especially the cross‑modal fusion strategy), critically revised and refined the manuscript, and managed the submission and editorial correspondence. All authors have reviewed and approved the final version of the manuscript.

Funding

This work was supported by the Key Research Project Plan of Henan Universities under Grant 26B520050.

Data availability

The datasets used and analysed during the current study are publicly available. The VT5000 dataset is available at https://github.com/lz118/RGBT-Salient-Object-Detection. The VT1000 dataset is available from the authors’project page at https://github.com/lz118/RGBT-Salient-Object-Detection. The VT821 dataset is available at https://github.com/lz118/RGBT-Salient-Object-Detection.

Declarations

Competing interests

The authors declare no competing interests.