Enhancing cross-modal retrieval via label graph optimization and hybrid loss functions

Abstract

Cross-modal retrieval, particularly image-text matching, is crucial in multimedia analysis and artificial intelligence, with applications in intelligent search and human-computer interaction. Current methods often overlook the rich semantic relationships between labels, leading to limited discriminability. We introduce a Two-Layer Graph Convolutional Network (L2-GCN) to model label correlations and a hybrid loss function, Circle-Soft, to enhance alignment and discriminability. Extensive experiments on the NUS-WIDE, MIRFlickr, and MS-COCO datasets demonstrate the effectiveness of our approach. The results show that the proposed method consistently outperforms current baselines, achieving accuracy improvements of 0.5%, 0.5%, and 1.0%, respectively. The source code is accessible via https://github.com/buzzcut619/L2-GCN-CIRCLE-SOFT.

Introduction

The rapid advancement of artificial intelligence (AI) has led to the widespread deployment of AI-driven solutions for complex challenges^1,2. A critical innovation in this domain is cross-modal retrieval, which allows for semantic-based search across different data formats (e.g., using text to find images or videos). By leveraging techniques from computer vision, natural language processing, and representation learning, this technology is revolutionizing applications in intelligent video surveillance, interactive multimedia platforms, and AI-assisted content creation, where understanding the semantic correlation between heterogeneous data is paramount.

To address the challenges in this field, extensive research has been conducted to enhance feature alignment and model discriminability. Representative methods, such as Stacked Cross Attention (SCAN)³, Iterative Matching with Recurrent Attention Memory (IMRAM)⁴, Semantically Supervised Maximal Correlation (S2MC)⁵, and the Unified Multi-level Cross-Modal Similarity (MCMS) framework⁶, have made significant strides in bridging the semantic gap. Additionally, self-supervised approaches like Self-Align⁷ have been proposed to force alignment at both concept and context levels. More recently, sophisticated frameworks leveraging contrastive learning and deep interaction have further pushed the boundaries of retrieval performance. For instance, IConE⁸ introduces an instance contrastive embedding method that combines instance loss with contrastive loss to extract fine-grained representations. To enhance information interaction, CDISA⁹ proposes a cross-modal deep interaction and semantic aligning module, utilizing bidirectional cosine matching to improve feature differentiation. Furthermore, to tackle inconsistent learning objectives, a multimodal embedding transfer approach¹⁰ has been developed, which employs a soft contrastive loss to realize selective optimization of multimodal pairs.

Despite these advancements, adapting retrieval models to complex, real-world scenarios remain a major challenge. Specifically, individual image-text pairs often exhibit multi-label characteristics due to the intricate interconnections of semantic concepts—a factor that most existing methods fail to fully capture. This “multi-label” issue manifests in two key aspects: first, a single image-text pair frequently contains multiple parallel or hierarchical semantic labels; second, distinct image-text pairs may share common semantic labels, thereby forming complex many-to-one relationships in the semantic space. For instance, as illustrated in Fig. 1, an image focusing on marine life might simultaneously contain labels such as “Whale,” “Sea,” “Wind,” and “Sky”. Meanwhile, another visually distinct image (e.g., a coastal sunset) could also share the “ocean” label. However, existing methods (e.g.^3–10) primarily focus on optimizing cross-modal alignment while treating labels as semantically independent entities. This oversight prevents the construction of comprehensive semantic graphs and weakens the semantic bridges between heterogeneous modalities, limiting retrieval accuracy.

Fig. 1.

Illustration of the multi-label characteristics and semantic correlations in cross-modal retrieval. The original image on the left displays a complex natural scene, while the right side lists its corresponding semantic labels: “1. Sky,” “2. Whale,” “3. Sea,” and “4. Wind.” These labels are not isolated but share rich semantic relationships (e.g., the co-occurrence of “Whale” and “sea,” where “sea” can also co-occur with other labels), which collectively describe the image content.

To overcome this limitation, we propose a novel Two-Layer Graph Convolutional Network (L2-GCN) for modeling label correlations. In this framework, labels are represented as nodes in a semantic graph, with initial features obtained from semantic embeddings. By iteratively aggregating features from each node and its first-order neighbors, L2-GCN captures high-order semantic dependencies, thereby enhancing the structural consistency and discriminative power of cross-modal representations.

Moreover, single loss functions demonstrate limited effectiveness in addressing multidimensional challenges. Modal heterogeneity (i.e., distribution discrepancies between image and text features) and suboptimal feature discriminability (characterized by insufficient intra-class compactness and rigid inter-class distance adjustment) constitute two fundamental bottlenecks. To address these, we propose a hybrid loss function termed Circle-Soft, which integrates the adaptive margin property of Circle Loss with the sample weighting strategy of contrastive learning. A learnable parameter is incorporated to dynamically balance the contributions of these two components. This design effectively mitigates modal heterogeneity while enhancing intra-class compactness and inter-class separation in the embedding space.

The main contributions of this work are summarized as follows:

1. Proposing a Two-Layer Graph Convolutional Network (L2-GCN) for label relationship modeling. Unlike existing methods that treat labels as independent, L2-GCN explicitly models label semantics by constructing a label graph (nodes = labels, edges = semantic relationships). It first processes node features (labels) and their first-order neighbor features separately, then fuses these processed features to generate comprehensive label representations. This design enables the capture of high-order semantic dependencies between labels, strengthening semantic connections across modalities.

2. Developing a Circle-Soft Loss function for joint optimization of alignment and discriminability. Circle-Soft Loss integrates the dynamic margin adjustment mechanism of Circle Loss (to optimize intra-class/inter-class relationships) and the cross-modal alignment capability of Contrastive Loss (to mitigate modal heterogeneity). A learnable weight parameter balances the contribution of the two components, while an adversarial loss is further incorporated to refine feature consistency, addressing both challenges simultaneously.

3. Validating the proposed approach on three public benchmarks. Extensive experiments on the NUS-WIDE, MIRFlickr, and MS-COCO datasets demonstrate that our method consistently outperforms baselines (e.g., achieving 0.5%, 0.5%, and 1.0% improvements, respectively) and shows strong robustness, verifying the effectiveness of the proposed innovations.

The remainder of this paper is organized as follows: “Related works” reviews related work on cross-modal retrieval and label modeling; “Proposed method” details the proposed L2-GCN and Circle-Soft Loss; “Experiments” presents experimental settings and results; “Limitations and future work” concludes the Limitation and Future Work.; “Conclusion” concludes the work.

Related works

Cross-modal retrieval methods

Generally, cross-modal retrieval approaches can be categorized into two main streams: traditional methods and deep learning-based methods. Traditional techniques primarily utilize linear projections to map multi-modal data into a common feature space, where similarity calculations are performed to achieve image-text matching,^11,12.Hardoon et al.¹³ utilized Canonical Correlation Analysis (CCA) and its kernelized version (KCCA) to map heterogeneous data into a common subspace by maximizing inter-modal correlations, thereby effectively handling both linear and non-linear alignments.

Sharma et al.¹⁴ proposed Generalized Multiview Analysis (GMA), a supervised extension of CCA. By leveraging class labels, this model effectively minimizes the distance between samples of the same class in the latent space, thereby enhancing the discriminability of cross-modal features. Similarly, Joint Representation Learning (JRL)¹⁵ utilizes both label information and correlation maximization within a unified optimization framework. Specifically, JRL incorporates semantic constraints into the subspace learning process, ensuring that the learned common representation preserves the underlying semantic structure of the multi-modal data.

With the rapid development of deep learning, various deep learning-based cross-modal retrieval methods have been proposed. In AFA¹⁶, single-modal features are adaptively aggregated by a Feature Enhancement Module, and the retrieval reliability is enhanced by an improved ranking loss function. In TGDT¹⁷, coarse and fine-grained representations are combined into a unified framework, and semantic consistency is ensured by a Consistent Multimodal Contrastive loss. In Deep Supervised Cross-modal Retrieval (DSCMR)¹⁸, discriminative features are supervised by minimizing losses in both label and common spaces, and cross-modal discrepancy is eliminated via a weight sharing strategy. However, the aforementioned methods largely overlook label correlations, which are pivotal for learning discriminative representations. To address this limitation, we propose the L2-GCN framework. Furthermore, we introduce a Circle-Soft Loss function designed to enhance intra-class compactness and inter-class separability, thereby significantly improving cross-modal matching accuracy.

Graph convolutional neural networks

Graph Convolutional Neural Networks(GCNs)^19,20, are widely applied in image recognition^21,22, social network analysis^23,24, and image re-ranking^25,26 because of their powerful graph feature expression capabilities. GCNs can effectively capture the relationships between different modalities and leverage the structure of the data to improve the representation learning. By processing information through GCNs, the system can better understand the semantic relationships and shared meanings across modalities. Naturally, GCNs can also be applied to the processing of label data and modality data. Qian et al.²⁷ proposed ALGCN, which employs GCNs on an adaptive correlation matrix to explicitly model label dependencies. This mechanism optimizes inter-dependent classifiers, thereby generating discriminative representations for cross-modal retrieval. Qian et al.²⁸ proposed a new method called DAGNN, which utilizes dual generative adversarial networks to create a common representation space and employs multi-hop graph neural networks to capture label correlations, where a layer aggregation mechanism is proposed to leverage multi-hop propagation information, capturing label correlations and learning interdependent classifiers. Qian et al.²⁹ proposed I-GNN. Compared to DAGNN, I-GNN utilizes an iterative approach to process label information, enabling it to adaptively learn a better graph and further capture fine-grained relationships between different labels. Wang et al.³⁰ proposed FGAP, which utilizes graph attention and pooling networks to learn graph-level relevance and transform fragment-level features into a shared subspace. Similarly, Wang et al.³¹ introduced a framework combining Dynamic Semantic Graph Enhancement (DSGE) and Progressive Semantic Alignment (PSA), which adaptively adjusts graph connectivity to prevent over-connection and captures hierarchical cross-modal dependencies.

These methods have achieved good results; however, they overlook the relationships between nearest neighbors within the label graph, failing to explore the deeper, fine-grained relationships within the labels. Different from these methods, we propose a novel framework that explicitly models high-order label dependencies via L2-GCN and optimizes feature alignment using a hybrid Circle-Soft Loss. The details of our approach are presented in the following section.

Proposed method

This section provides a detailed explanation of the proposed framework, covering the processing of raw feature data, the generation of common space representations, the optimization of label representations via the L2-GCN model, and the enhancement of final matching through the proposed hybrid loss function.

Overview

As illustrated in Fig. 2, the proposed framework is designed to bridge the heterogeneity gap between vision and language by integrating adversarial learning with fine-grained label modeling. The overall architecture comprises two mutually reinforcing sub-modules: the Modality Discrimination Adversarial Network and the L2-GCN Label Set Optimization Model.

Fig. 2.

Schematic illustration of the proposed framework, comprising two key sub-processes: (a) Modal discrimination adversarial network: this component depicts the data processing flow for image-text pairs, illustrating how the adversarial network and Circle-Soft Loss align heterogeneous features into a shared embedding space to enhance cross-modal similarity. (b) L2-GCN Label Set Optimization Model: This component demonstrates the label refinement process via L2-GCN, followed by the calculation of classification loss.

First, in the feature extraction stage, raw image and text inputs are processed by their respective encoders to generate initial feature representations ( and ). These features are then passed through a Feature Generator and subsequently challenged by a Modality Discriminator. Under the supervision of the Adversarial Loss, this adversarial process eliminates modality-specific patterns, rendering the features from both modalities more similar. Simultaneously, to enhance the discriminability of these features, we apply the Circle-Soft Loss to the initial feature representations, which optimizes intra-class compactness and inter-class separability.

Parallel to this, the L2-GCN module operates on the label space. By aggregating features from label nodes and their first-order neighbors, it explicitly models the high-order semantic dependencies between labels, resulting in a semantically enriched label set.

Finally, the optimized label representations are used to supervise the visual and textual features via a Classification Loss. This step ensures that the learned common representations align accurately with the refined semantic categories, completing the end-to-end training process.

To provide a high-level perspective of the workflow, Fig. 3 presents a simplified overview, encompassing the core components detailed in the subsequent subsections of “Proposed method”.

Fig. 3.

Detailed flowchart of the proposed method.

Feature extraction

Introduce the notation used throughout the paper. We used a pair of networks (VGG19 and BoW) to coarsely extract the raw features of images and text. The extracted features are denoted as V and T, respectively. Additionally, we denote the training dataset as , where n represents the total number of image-text pairs in this dataset. denotes the i-th image-text pair in the dataset S, and represents the feature dimension of the raw input data. is the label set used for the corresponding training set S. m is the total number of categories in the label set. If the i-th image-text pair in S belongs to the m-th category in , then , and all other entries set to 0.

Our approach consists of two main parts:

1. Modality Discriminator Adversarial Network, which will be explained in detail in the following sections, specifically addressing common feature extraction and the adversarial alignment process.

2. Detailed Process of L2-GCN, which will be explained in detail in the following sections, specifically addressing the processing of the label set.

After completing sections (a) and (b), a classification loss is applied using the common representations created in part (a) and the processed label set, which is also a classic operation in supervised learning.

Modality discriminator adversarial network

The main goal of this part is to extract common features and perform modality discrimination. By using VGG19, BoW, and MLP to extract the raw features of images and text, these features are projected into a shared space. In this shared space, the similarity between the two modalities can be directly computed, which is a significant branch of cross-modal retrieval and is fundamental to its implementation.

For images, we employ a seven-layer VGG-19 network¹⁵ pre-trained on a public dataset³² to extract raw image features. The output from the seventh layer (fc7) typically yields a feature vector with a dimension of 1024.

1

MLP refers to a Multilayer Perceptron consisting of several stacked fully connected linear layers. Here, d represents the dimension of the image features after projection into the common space.

For text, similar to image feature extraction, we utilize the widely used Bag-of-Words (BoW) approach³³, to obtain raw text features. These features are then mapped into the common embedding space.

2

Similar to image representation, MLP refers to a Multilayer Perceptron consisting of several stacked fully connected linear layers. Here, d represents the dimension of the text features after projection into the common space.

Modality discrimination: the common representation of images and text is fed into the respective decoders of the feature generator. After obtaining these generated representations, they are passed to the modality discriminator for modality classification. This adversarial interplay between the Feature Generator and Modality Discriminator is the essence of GAN networks, ensuring modality-invariant feature learning.

3

4

where and are the features of image and text common representations obtained after passing through the generator. These features are later used in the GAN loss.

L2-GCN pre-optimization of the label set

In cross-modal datasets, labels are rarely independent; instead, they exhibit strong statistical co-occurrence and correlations. For instance, “sky” frequently co-occurs with “airplane” but rarely with “trees” (in certain contexts), yet it can also co-occur with “dolphin,” implying that a single object concept can appear across diverse thematic contexts.

The L2-GCN is designed to capture the topological structure of these semantic correlations. As illustrated in the L2-GCN module in Fig. 2, we construct a label graph where the adjacency matrix represents these co-occurrence and correlation probabilities (indicated by values such as 0.7 in the figure). By propagating information within this graph, the model enriches individual label representations using complementary context from correlated neighbors, thereby enhancing the accuracy of cross-modal retrieval.

The label set is denoted as , N represents a node , a node represents a category, each category has a corresponding feature matrix, the feature matrix of all nodes denoted as F, E represents edge , The connections between nodes are represented by a matrix that links these relationships together, forming an adjacency matrix, denoted as A.

5

6

First, update the adjacency matrix A, Diag is a square matrix with zeros on the main diagonal and all other elements being one. Subtract this matrix from a matrix of all ones to obtain a matrix with ones on the diagonal and zeros elsewhere, denoted as .

Next, use a weight matrix and bias to update the feature matrix, and then multiply it with , resulting in . This feature matrix retains the features of all nodes themselves.

7

8

9

10

is a square matrix with zeros on the diagonal, while retaining the original values elsewhere.

For matrix , in the first dimension, any value greater than 0 is treated as indicating a relationship with a nearest neighbor, and thus these values are set to 1. The sum along the first dimension is then calculated to get the number of one-hop neighbors for each node, denoted as neighbor.

Next, multiply by and divide by neighbor to obtain the mean-pooled feature matrix .

Finally, pass and through one linear layer each, and then add the results to obtain the final feature matrix.

Circle-soft-loss-category loss and modality loss

Circle-Soft-Loss is an innovative loss function that integrates the strengths of Circle Loss and Soft Contrastive Loss. These two algorithms excel in handling intra-class, inter-class, and modality-specific relationships, respectively. By combining the advantages of both approaches and introducing a threshold for fine-tuning, Circle-Soft-Loss achieves superior performance in global similarity matching.

Category loss-circle loss

One of the core essences of cross-modal retrieval is to make data from the same category more similar in a shared embedding space while preserving their intrinsic characteristics. To achieve this goal, we adopt Circle Loss. Circle Loss reweights each similarity score to emphasize those that are under-optimized and provides a unified formulation for both category-level and pair-wise labels. Through this method, Circle Loss demonstrates significant advantages in handling complex cross-modal data, particularly in tasks that require fine-grained differentiation between categories. Below is the process by which Circle Loss processes data:

Normalize the obtained image common features and text common features V and T, where V and T are denoted as and from the previous context:

11

where denote the raw feature vectors of the image and text for the i-th sample, respectively.

Based on the normalized features, we calculate the pairwise cosine similarity within each modality. The similarity matrices for the image modality and text modality are defined as:

12

where represents the cosine similarity between the ith and jth images.

We directly utilize the label adjacency matrix to determine positive and negative pairs. Here, indicates that sample i and sample j are positive pair, while , indicates they are negative pair.

Based on A, we define the set of positive pairs P and negative pairs N as:

13

14

Circle Loss simultaneously optimizes class relationships via an adaptive weighting mechanism. Taking the image modality as an example, the loss function is formulated as:

15

where is the scale factor, and are the preset margins. The adaptive weighting coefficients and are calculated as:

16

Similarly, the loss for the text modality is calculated in the same manner. The final total Circle Loss is the sum of the losses from both modalities:

17

Modality loss-soft contrastive loss

Inspired by I-GNN, we utilize its contrastive loss as the modality Loss. Specifically, Soft Contrastive Loss optimizes cross-modal alignment by exploiting the similarity matrix between views. Its normalization mechanism plays a critical role in enhancing the model’s discriminability across different modalities. Compared to traditional contrastive loss functions, the Soft Contrastive Loss places greater emphasis on the similarities between modalities, thereby offering stronger adaptability and robustness in multi-modal tasks.

18

19

20

21

22

Ultimately, integrate the category loss and modality loss to obtain the final formula.

23

Classification loss

The classification loss serves to calculate the distance difference between the predicted labels and the actual labels. Classification loss is typically used to help the model learn semantic consistency between different modalities, ensuring that the same concept or entity can be correctly associated across modalities.

24

represents the predicted label, R represents the actual label, for generality, we use the (Euclidean norm) to calculate the distance between them, leading to the final loss function.

Adversarial loss

Similar to AGCN, we made a small improvement by introducing a weight to merge the losses from both views, reducing redundancy and balancing the impact of different modalities on the final loss. We use GAN Loss to further update the features of both modalities. In the field of cross-modal retrieval, GAN Loss (the loss function of Generative Adversarial Networks) plays a crucial role. It encourages the model to deeply learn the intrinsic connections between modalities through the adversarial game between the generator and discriminator, achieving effective mapping and transformation between different data types such as images and text.

25

is the feature obtained from the common representation of images after passing through the decoder, is the feature obtained from the common representation of text after passing through the decoder, and denotes the cross-entropy loss function.

Overall objective

Based on the above losses, the final objective loss is defined as follows, where the values of , and will be discussed in “Experiments”.

26

Retrieval process

While the training phase involves complex optimization objectives and label graph modeling (L2-GCN) to learn discriminative representations, the actual retrieval stage operates in a deterministic, feed-forward manner. To provide a clear understanding of how the trained model is deployed for cross-modal tasks, we illustrate the detailed retrieval workflow in Fig. 4.

Fig. 4.

Schematic illustration of the cross-modal retrieval process.

As illustrated in Fig. 4, the retrieval process operates in a deterministic, feed-forward manner. Specifically, taking the Image-to-Text task as an example, given a query image and a text database, we first extract their raw features using the pre-trained VGG-19 and BoW models, respectively. These features are then projected into the common embedding space via the trained Linear Layers, resulting in the final feature vectors. Subsequently, we calculate the semantic similarity between the query image vector and each vector in the text database using Cosine Similarity. Finally, all items in the database are sorted in descending order based on these similarity scores to generate the final ranked retrieval list.

Experiments

To validate the effectiveness of our method, we tested it on three different and representative datasets. First, we introduce the evaluation metrics, then we describe the datasets used, and finally, we conduct ablation experiments and various parameter analyses.

Evaluation metrics

We use mAP^34,35 (Mean Average Precision) as the evaluation metric. mAP is a widely used metric in information retrieval and recommendation systems to measure a model’s overall retrieval performance across multiple query tasks. After calculating the mAP values for both modalities, the final value is obtained by averaging them, which reflects the effectiveness of our method.

For a single query q, the formula for calculating average precision is as follows:

27

28

N is the number of retrieved results. P(k) is the precision of the top k retrieved results. is an indicator function that represents whether the k-th result is relevant (1 if relevant, 0 if not). is the total number of relevant results for the query q. Q denotes the total number of queries in the test set.

Datasets

The experiments were conducted on three widely used datasets. To ensure a fair comparison with existing state-of-the-art methods (e.g., DSCMR³⁴, DAGNN²⁸, I-GNN²⁹), we strictly follow the standard data partition protocol widely used in the literature, and the detailed descriptions of these datasets are as follows:

MIRFlickr: The MIRFlickr dataset contains 25,000 image-text pairs collected from the Flickr website, covering a wide range of scenes and subjects. Each image and its corresponding text are annotated with tags that describe the content of both the image and the text. These tags can be used for image category classification and retrieval tasks. The dataset includes a variety of images, such as natural landscapes, urban scenes, animals, people, and events. This diversity makes the dataset highly useful for various research tasks. We selected 2000 image-text pairs as the test set, with the remaining pairs used as the training set.

NUS-WIDE: NUS-WIDE is a large-scale dataset for multimedia information retrieval and image annotation, primarily provided by the National University of Singapore (NUS). This dataset is widely used in research on image classification, retrieval, and cross-modal retrieval. We selected 190,421 image-text pairs for our experiments. Among these, we randomly chose 2000 image-text pairs as the test set, with the remainder serving as the training set.

MS-COCO: This dataset contains over 320,000 images, which are richly annotated with information about objects, scenes, and their contextual relationships. The MS-COCO dataset covers 80 common object categories, including humans, animals, everyday items, vehicles, and more. The images are diverse in content, often containing multiple objects with complex interactions and background information. To obtain the mAP evaluation metric, 2,000 image-text pairs were randomly selected as the test set, with the remaining pairs used as the training set.

Baseline methods

We compared our method with thirteen baseline methods, including three traditional multivariate statistical methods: CCA³⁶, ML-CCA³⁷, and PLS-C2A³⁸; three cross-modal hashing methods: DCDH³⁹, GCH⁴⁰, MGCH⁴¹;and eleven deep learning-based cross-modal retrieval methods: DCCA⁴², ACMR⁴³, DSCMR¹⁸, DAVAE⁴⁴, ALGCN²⁷, DAGNN²⁸, IConE⁸, FGAP³⁰, DSGE³¹ ,CDISA⁹ and I-GNN-CON²⁹. We directly referenced the results from the respective papers or reproduced these methods’ codes based on their models to obtain comparative results.

Implementation details

All experiments were implemented using the PyTorch framework and conducted on a single NVIDIA GeForce RTX 4060 Ti GPU. We utilized the Adam optimizer to train the entire model. The learning rate was universally set to 0.00005 across all datasets, with a total training duration of 40 epochs. To accommodate different dataset scales, the batch sizes were set to 2048 for NUS-WIDE, 100 for MIRFlickr, and 512 for MS-COCO.

Regarding the specific hyperparameters used in our proposed method, the loss weights and were set to 1.1 and 2.2, respectively, for all three datasets. The threshold parameter was empirically set to 0.6. Additionally, the depth of the L2-GCN was configured to 5 layers for the MIRFlickr and MS-COCO datasets, whereas it was set to 6 layers for the NUS-WIDE dataset.

In terms of network architecture, both the image features (processed by VGG-19) and text features (processed by BoW) are mapped into a common embedding space with a dimension of 1,024 via their respective MLPs. The Image Decoder consists of two fully connected layers with 512 and 300 hidden units, respectively, activated by LeakyReLU with a negative slope of 0.2. Conversely, the Text Decoder comprises two fully connected layers with 2,048 and 4,096 hidden units, respectively, activated by Rectified Linear Unit (ReLU). For the L2-GCN module, the output dimension of each layer is fixed at 1,024. The construction of the adjacency matrix and input vertex features for the label graph follows the protocol described in²⁶.

Results and discussion

All experimental results and comparison results are shown in Table 1. The first three experiments use traditional multivariate statistical methods, the middle three methods utilize cross-modal hashing approaches, and the remaining methods use deep learning-based cross-modal retrieval techniques.

Table 1.

mAP scores on NUS-WIDE, MIRFlickr, and MS-COCO Datasets.

Method	NUS-WIDE			MIRFlickr			MS-COCO
	I2T	T2I	Average	I2T	T2I	Average	I2T	T2I	Average
CCA	0.656	0.664	0.660	0.712	0.722	0.717	0.652	0.656	0.654
PLS-C2A	0.632	0.631	0.631	0.730	0.740	0.735	0.643	0.637	0.640
ml-CCA	0.669	0.668	0.668	0.734	0.742	0.738	0.637	0.634	0.635
DCCA	0.637	0.649	0.643	0.736	0.746	0.741	0.635	0.630	0.632
ACMR	0.684	0.675	0.680	0.736	0.748	0.742	0.706	0.708	0.707
DCDH	0.684	0.680	0.682	0.742	0.758	0.750	0.610	0.604	0.607
MGCH	0.684	0.690	0.687	0.751	0.757	0.754	0.661	0.669	0.665
GCH	0.677	0.697	0.687	0.762	0.786	0.774	0.559	0.560	0.560
DSCMR	0.706	0.739	0.722	0.752	0.799	0.775	0.813	0.810	0.811
DAVAE	0.728	0.728	0.728	0.760	0.801	0.781	0.821	0.809	0.815
IConE	0.736	0.747	0.741	0.788	0.799	0.793	0.811	0.822	0.816
ALGCN	0.747	0.758	0.753	0.792	0.819	0.806	0.831	0.810	0.811
FGAP	0.750	0.766	0.758	0.791	0.830	0.810	0.833	0.828	0.830
DSGE	0.762	0.754	0.757	0.810	0.810	0.810	0.839	0.822	0.830
DAGNN	0.755	0.761	0.758	0.806	0.820	0.812	0.836	0.830	0.833
CDISA	0.766	0.782	0.773	0.793	0.799	0.796	0.834	0.854	0.844
I-GNN-CON	0.757	0.763	0.760	0.807	0.825	0.816	0.843	0.840	0.841
L2-GCN (our)	0.760	0.770	0.765	0.813	0.827	0.821	0.850	0.852	0.851

From the experimental results in Table 1, it can be observed that, firstly, among these three methods, models using deep learning techniques achieve the best results. This is because deep learning strategies are effective in detecting both linear and nonlinear relationships between data. Secondly, the results of the three cross-modal hashing methods are generally inferior to those of deep learning methods. This is because hashing methods map high-dimensional real-valued features to low-dimensional binary hash codes, inevitably resulting in some information loss. This information loss may lead to a decrease in feature discriminability, thus affecting retrieval accuracy. Thirdly, the results of the DSCMR and DAVAE models are slightly behind those of the other three deep learning methods because ALGCN, DAGNN, and I-GNN-CON all use their respective GCN models to explore the semantic relationships between labels, achieving significant data improvements. This indicates that the exploration of label processing methods in cross-modal retrieval is far from complete and warrants further research. While CDISA achieves superior accuracy on the NUS-WIDE dataset compared to other methods, its performance on the MIRFlickr and MS-COCO datasets is suboptimal. Ultimately, L2-GCN achieves the best performance on three datasets compared to other methods.

For example, in the MIRFlickr dataset, Table 1 shows that our method (L2-GCN) achieves the best mAP scores of 0.813 and 0.827 for the Image2Text and Text2Image tasks, respectively. Compared to the traditional multivariate statistical method CCA, L2-GCN improves performance by 10% and 10.5% for Image2Text and Text2Image tasks, respectively. Compared to the best cross-modal hashing method GCH, L2-GCN improves average performance by 4.6%. Compared to the best deep cross-modal retrieval method I-GNN-CON, our method improves performance by 0.6% and 0.2% for the Image2Text and Text2Image tasks, respectively.

To evaluate the effectiveness of the proposed L2-GCN module and the impact of the loss component , we conducted comprehensive ablation studies comparing performance across different layer depths and loss combinations. Specifically, the number of L2-GCN layers was varied from 1 to 8, and five distinct loss scenarios were evaluated: retaining all losses, retaining only , removing , removing , and removing . These five scenarios are denoted as conditions 1 to 5, respectively. Based on these combinations, we generated a 3D surface plot (Fig. 5) and a bar chart (Fig. 6) to visually illustrate the joint impact of layer depth and loss components on retrieval performance.

Fig. 5.

3D surface plot illustrating the joint impact of L2-GCN layer depth and different loss function configurations on retrieval precision.

Fig. 6.

Ablation study on the MIRFlickr dataset illustrating the impact of different loss component configurations on retrieval performance. The vertical axis represents the Precision score, while the horizontal axis denotes various ablation settings.

The results in Fig. 5 reveal distinct trends regarding both model depth and loss composition.

First, concerning the L2-GCN layer depth, the performance exhibits a clear convex trend. With shallow networks (1–2 layers), the model fails to capture sufficient high-order semantic correlations, resulting in suboptimal precision. The accuracy peaks at 5 layers, indicating the optimal balance for feature aggregation. However, increasing the depth further (6–8 layers) leads to a performance degradation. This is likely due to the “over-smoothing” phenomenon common in deep GCNs, where repeated aggregations cause node features to converge and lose discriminability, while simultaneously increasing computational complexity. Consequently, we set the optimal layer count to 5 for the MIRFlickr dataset.

Second, regarding the loss configurations, the plot demonstrates that the highest accuracy is consistently achieved when all losses are retained, regardless of the layer count. Notably, removing causes a sharp decline in performance. These findings underscore the pivotal role of in enhancing the overall data representation and retrieval accuracy.

Figure 6 further illustrates the impact of loss on accuracy. The removal of losses and results in only a minor decline in overall accuracy, but removing our proposed loss leads to a 2% decrease, demonstrating the indispensable role of loss in affecting accuracy. For the MIRFlickr dataset, when the number of layers is 5 and all loss conditions are retained, the accuracy reaches the highest value of 0.82.

To further investigate the impact of L2-GCN on experimental results, we replaced L2-GCN with GCN, GAT and BaGFN to explore the semantic relationships between labels. Figures 7 and 8 show the experimental results of L2-GCN, GCN, GAT and BaGFN⁴⁵ on the MIRFlickr datasets. It can be seen that L2-GCN outperforms both GAT and GCN in the Image2Text and Text2Image retrieval tasks. Despite BaGFN’s superior I2T performance on both datasets, L2-GCN holds a slight edge in the overall average, a finding that serves to confirm its effectiveness.

Fig. 7.

Bar chart illustrating the retrieval accuracy of different label processing models on the MIRFlickr dataset. The horizontal axis represents the retrieval tasks, while the vertical axis represents the prediction precision (percentage). The colored bars distinguish between the baseline models—GCN (red), GAT (blue), and BaGFN (green)—and the proposed L2-GCN (purple).

Fig. 8.

Bar chart illustrating the retrieval accuracy of different label processing models on the NUS-WIDE dataset. The horizontal axis represents the retrieval tasks, while the vertical axis represents the prediction precision (percentage). The colored bars distinguish between the baseline models—GCN (red), GAT (blue), and BaGFN (green)—and the proposed L2-GCN (purple).

From Figs. 7 and 8, it can be seen that our L2-GCN model shows significant improvement compared to the three traditional foundational models, indicating notable progress in exploring the fine-grained relationships between labels.

Considering the critical role of GAN Loss, we analyzed the sensitivity of the retrieval accuracy to the hyperparameter . To identify the optimal configuration, we jointly optimized and through repeated experiments on two datasets. The corresponding results are visualized in Figs. 9, 10, 11 and 12.

Fig. 9.

Parameter sensitivity analysis on the MIRFlickr dataset for the Image-to-Text (i2t) task. The 3D plot illustrates the joint impact of hyperparameters and on retrieval precision.

Fig. 10.

Parameter sensitivity analysis on the MIRFlickr dataset for the Text-to-Image (t2i) task. The 3D plot illustrates the joint impact of hyperparameters and on retrieval precision.

Fig. 11.

Parameter sensitivity analysis on the NUS-WIDE dataset for the Image-to-Text (i2t) task. The 3D plot illustrates the joint impact of hyperparameters and on retrieval precision.

Fig. 12.

Parameter sensitivity analysis on the NUS-WIDE dataset for the Text-to-Image (t2i) task. The 3D plot illustrates the joint impact of hyperparameters and on retrieval precision.

As shown in Figs. 9, 10, 11 and 12, for both datasets, the accuracy decreases dramatically as the parameter is adjusted from 2.2 towards 0, reaching its lowest value when is 0. For the parameter, the data distribution forms a normal distribution with a peak around 1.1, creating an arch-like shape. Consequently, the optimal accuracy is achieved when and are set to 1.1 and 2.2, respectively.

As shown in Fig. 13. To better balance the trade-off between Soft Contrastive Loss and Circle Loss, we introduce a threshold between them using an empirical modulation strategy. Under optimal prior conditions, we conducted experiments with set to 0.3, 0.4, 0.5, 0.6, and 0.7. The highest accuracy across three datasets was achieved when .

Fig. 13.

Parameter sensitivity analysis illustrating the impact of the threshold on retrieval accuracy across three datasets. The horizontal axis represents the varying values of (ranging from 0.3 to 0.7), while the vertical axis denotes the classification accuracy.

To provide a more intuitive illustration of the superior performance of the L2 GCN model in retrieval tasks, we present qualitative examples from several randomly selected image-to-text queries. Figure 14 displays the top-two retrieved results for each query, showing that our model achieves more accurate matches than the DSCRM baseline in almost every case.

Fig. 14.

Visualization of top two retrieval results for several randomly selected image queries.

To better demonstrate the convergence of the objective function, Figs. 15 and 16 show that during the training process on the MIRFlickr and NUS-WIDE datasets, the loss decreases rapidly and eventually stabilizes. We set the number of iterations to 40, with the final loss function value converging to approximately 0.153 for the MIRFlickr dataset and around 0.01 for the NUS-WIDE dataset.

Fig. 15.

Visualization of the training loss curve on the MIRFlickr dataset. The horizontal axis represents the number of training epochs, while the vertical axis represents the value of the objective function.

Fig. 16.

Visualization of the training loss curve on the NUS-WIDE dataset. The horizontal axis represents the number of training epochs, while the vertical axis represents the value of the objective function.

To demonstrate the practical deployability and the balance between efficiency and performance of the proposed L2-GCN, we conducted a comparative analysis of computational complexity against the baseline method(I-GNN). The experiments were performed on a hardware platform equipped with a single NVIDIA RTX 4060 Ti GPU. We adopted Parameter Count, Floating-point Operations (FLOPs), and Training Time as the evaluation metrics.

The results are summarized in Table 2. Compared to the baseline, our method introduces a marginal increase in model size of 5.49 M parameters and a slight rise in computational cost of 0.07 GFLOPs. This slight overhead is attributed to the additional graph convolution layers and the enhanced feature aggregation mechanism designed to capture richer cross-modal correlations.

Table 2.

Comparison of computational complexity and performance on MIRFlickr.

Method	Params (M)	FLOPs (G)	Training time	mAP gain
Baseline	29.92	0.3010	7 m 19 s
L2-GCN	35.41	0.3744	8 m 01 s	+0.5 %

However, it is worth noting that the total FLOPs of our method remain at a very low level of 0.3744 G, which is significantly lower than standard deep visual backbones. Furthermore, the increase in training time is negligible, amounting to less than 1 minute. Considering the 0.5% improvement in retrieval accuracy, we argue that L2-GCN achieves a favorable balance between efficiency and performance. It provides enhanced retrieval capabilities while keeping computational costs within an acceptable range for real-time applications.

Limitations and future work

While the proposed framework demonstrates promising cross-modal retrieval capabilities, we acknowledge certain limitations regarding computational complexity, potential overfitting risks, and adaptability that merit further investigation.

First, regarding computational complexity: The inherent matrix multiplications and feature aggregations of the L2-GCN inevitably increase computational overhead. Although operating in a low-dimensional label space mitigates computational demands, it may still pose a deployment bottleneck on resource-constrained edge devices. Furthermore, the composite objective function—integrating Circle-Soft, adversarial, and classification losses—constructs a complex optimization landscape, potentially leading to local optima or overfitting on small-scale datasets. Finally, its performance on ultra-long texts or high-resolution images has not been fully explored, and the current scope is limited to image-text retrieval, excluding scenarios like video-text or cross-lingual tasks.

To address these challenges, future work will focus on the following directions: We plan to employ knowledge distillation techniques and decouple static graph computations from dynamic training to reduce inference latency and enhance efficiency. To address overfitting risks, we intend to introduce an adaptive loss weighting strategy to ensure stable optimization. Finally, for complex inputs, we plan to integrate Transformer-based backbones and extend the framework to broader scenarios, such as video-text and cross-lingual retrieval, thereby facilitating its wider application.

Conclusion

In this paper, we addressed the limitations of existing cross-modal retrieval methods, particularly their oversight of complex label correlations and insufficient feature discriminability. To this end, we proposed a novel Two-Layer Graph Convolutional Network (L2-GCN), which explicitly models high-order semantic dependencies by aggregating neighbor information within the label graph, rather than treating labels as independent entities. Furthermore, we introduced a hybrid Circle-Soft Loss function that integrates the benefits of Circle Loss and Contrastive Learning. This design dynamically balances intra-class compactness and inter-class separability, effectively mitigating modal heterogeneity. Extensive experiments on the NUS-WIDE, MIRFlickr, and MS-COCO datasets validate the robustness of our framework. Experiments confirm that our method outperforms baselines, achieving accuracy improvements of 0.5%, 0.5%, and 1.0%, respectively.

Author contributions

W L coordinated the overall progress of the article and contributed to some of the innovative points. WCC proposed innovative ideas and was responsible for text and chart production. P conducted research for the article.

Funding

This work was supported by the National Natural Science Foundation of China (52477226).

Data availability

The source code and datasets generated during this study are available in the GitHub repository, [https://github.com/buzzcut619/L2-GCN-CIRCLE-SOFT],Considering that the manuscript is currently under submission, we will release the full core code upon its formal acceptance.

Declarations

Competing interests

The authors declare no competing interests.