Multi-stage fusion of local and global features for few-shot image classification

Abstract

Few-shot classification is a very challenging task of computer vision. Recently, different from meta-learning, transfer-learning foregoing the episodic training strategy has gradually become popular in this community. Under this pipeline, how to learn a high-quality feature representation is vital for winning good performance. However, current works mainly build the classification model upon convolutional neural networks, which cannot extract discriminative features. To address the above problem, we propose exploring the non-local networks to construct classification model, which is trained by the joint learning of supervised and self-supervised tasks to obtain global invariant features. Further, we propose a few-shot classification algorithm using multi-stage fusion of local and global features, in which the fusion of features happens simultaneously during two stages of transfer-learning. The stage of pre-training implements parallel mechanism, in which the local feature network and global feature network mutually learn from each other, while the stage of few-shot testing implements serial mechanism through feature concatenation. We conducted extensive evaluations on multiple benchmark datasets to demonstrate the effectiveness of our method. Ablation studies have shown the effectiveness of the multi-stage feature fusion, and the comparison results have shown that our method can achieve better performance compared with other state-of-the-art methods.

Introduction

In recent years, the performance of image classification models based on deep convolutional neural networks (CNNs) has been constantly refreshed, however, these impressive achievements largely depend on the use of large-scale labelled data to optimize the representation parameters in the models. Manual labelling of large amounts of image data is labor-intensive and sometimes impossible to collect, which severely limits the practical application of CNNs in social and productive fields. In contrast, the human visual system can quickly identify an object and generalize it to other unseen observation samples by observing only a few example images. Inspired by this powerful learning capability, few-shot image classification^1,2 has now become very important in computer vision and artificial intelligence.

The few-shot image classification task involves making classification decisions for query samples in novel categories using only a few support sets per class. Due to the limited supervised information, it is often necessary to learn to obtain a priori knowledge with the help of base class samples as an additional auxiliary dataset. Meta-learning-based^3–5 methods quickly adapt to the classification task of unknown classes with little supervised information by constructing a series of few-shot image classification tasks in the base class dataset, and then employing segmental training to acquire the meta-knowledge embedded in these tasks. Few-shot image classification methods based on transfer learning usually adopt a two-stage learning approach, i.e., pre-training on the base class dataset and testing on the unknown class dataset. The key element of those methods is extracting good features for the unknown category⁶. Compared with data from unknown categories, the base-class dataset has sufficient labeled data. In most cases, the pretraining process treats it as a supervised learning task and optimizes the network parameters using the classic Softmax loss function. Hence, enhancing model generalization has emerged as a key challenge in few-shot image classification.. Unlike supervised learning tasks, self-supervised learning^7,8 constructs auxiliary tasks to directly learn useful feature representations for downstream tasks from unlabeled data. Recent works^9,10 introduce self-supervised tasks into few-shot image classification research, enabling the feature extractor to be invariant to certain variations in images through joint supervised and self-supervised learning during the pre-training step, thereby enhancing the generalization capability of the feature representations.

However,current few-shot image classification models are built upon CNNs whose receptive-field size limits them to extracting only local features from the input image. As shown in Fig. 1(a), when local changes such as occlusion, illumination, and pose occur in the content of the image, if local features are used, the bird can still be recognized even after its torso has been occluded; due to the occlusion of the bird’s torso by the tree leaves, it may be misclassified as a tree leaf when global features are used. This shows that local features have good robustness to local changes within the image. As shown in Fig. 1(b), when there are global changes in shape, color, texture, etc., if only local features are used, due to the fact that the local black region of the bird’s torso and the local black region of the tropical fish’s ventral fins are very close to each other, it may happen that the two are misclassified as being of the same kind; whereas, due to the difference in the overall shapes of the bird and the tropical fish, the global features are able to correctly differentiate between the two. This shows that global features have good discriminative properties. Leveraging the advantages of global features, we introduce non-local networks¹¹ into few-shot image classification to construct a global feature network, and use supervised and self-supervised joint learning to pre-train the model, yielding global invariant features. The algorithm constructs local and global feature networks separately and fuses them in the stage of pre-training and stage of testing. In order to reduce the computational effort in the pre-training stage, we use a parallel fusion method with mutual learning in this stage to minimize the Kullback-Leibler (KL) divergence between the two output probability distributions.The KL divergence measures the disparity between two probability distributions. In our mutual learning framework, we employ KL divergence to quantify the divergence among the output distributions of different networks. Amari¹² demonstrates that minimizing the KL divergence between output distributions can effectively enhance the model’s representational capacity, and the resulting representation consistency is more readily transferable to other related tasks. And the serial fusion method of splicing is used in the testing stage, which can map the samples to the high-dimensional feature space and increase the linear divisibility to improve the classification performance.

Fig. 1.

Illustration of our motivation.

The contributions of our work can be summarized as follows:

1. We propose a few-shot image classification model based on global features by adding a non-local computational module to the backbone neural network and pre-train the model using supervised and self-supervised joint learning. And for the first time, the non-local network is introduced to the few-shot image classification, and the established global network model can enable the feature extractor to have global invariance property, which improves the discriminative property of the classification model.

2. A few-shot image classification method based on multi-stage fusion of local and global features is proposed, and the fusion of local and global features in the pre-training stage is a parallel fusion method, which is realized through the interactive learning of information between the networks. The fusion of local and global features in the testing stage is a feature serial fusion method, which is realized by splicing features extracted from the local and global networks. We firstly implement multi-stage fusion of local and global features under the transfer learning paradigm, and the method can make full use of the complementary nature of local and global features, which makes the classification model robust and discriminative at the same time, and thus can effectively improve the performance of few-shot image classification.

3. The proposed method is experimentally validated on several datasets for few-shot image classification, and the comparison results with SOTAs show the effectiveness of the proposed method.

Related work

In recent years, few-shot image classification has attracted widespread attention. Based on the differences in the learning paradigms employed, these methods can generally be divided into two major categories: those based on meta-learning and those based on transfer learning.

Meta-learning based methods

Meta-learning simulates a real few-shot image classification testing environment through a series of fragments and accumulates meta-knowledge in several randomly sampled subtasks to quickly identify unknown classes. Depending on the type of meta-knowledge, meta-learning is further divided into optimization-based methods and metric-based methods. The former employs an explicit, two-tier optimization process to learn the key elements of the optimizer, allowing for rapid adaptation to new unseen tasks. For example, MAML¹³ obtained the optimal initialization parameters of the model through meta-training, so that the model can adapt to new tasks after a few gradient updates. In addition to initialization parameters, Meta-SGD¹⁴ and Meta-Learner LSTM¹⁵ learned the learning rate and update rules of the optimizer. Because bi-level optimization requires storing and computing higher-order derivatives, it incurs large memory and computational costs. Consequently, only low-capacity base learners can be adopted, which severely limits model expressiveness. To solve the above problems, MetaOptNet¹⁶ uses SVM instead of linear classifiers. Recently, in order to further improve the generalization performance of the model, TAML¹⁷ constrained all tasks on parameter updates, and MeTAL¹⁸ optimized the model using the task adaptive loss function in the inner loop. Meta-AdaM¹⁹ incorporated momentum into the optimization process via a double look-ahead mechanism.

The basic idea of the metric-based meta-learning methods is to learn a good feature embedding space, so that samples of the same class can be close to each other in this space, and samples of different classes can be far away from each other. In the meta-training process, the early methods constructed the feature embedding space based on the convolutional neural network, and used fixed metrics (such as cosine, Euclidean distance, etc.) or parameterized nonlinear metrics to calculate the similarity between samples. On the one hand, how to extract effectiveness features in the meta-training process is a continuous concern of researchers. For example, DSN²⁰ used the samples in the support set to find a discriminative subspace for each category, and maps the feature vectors of the query samples to this subspace for distance measurement, and IEPT¹⁰ introduced self-supervised tasks in the meta-training process to make the feature extractor obtain richer and more migratory feature representations. APP2S²¹ maps sample features to hyperbolic hyperspaces. On the other hand, several studies have carried out improvement work around the similarity measurement link. For example, CovMNet²² and Deep EMD²³ obtained rich similarity measures directly from local features, while DeepBDC²⁴ introduced a Brownian covariance measure function. Wu et al.²⁵ proposed a trilinear spatial-awareness network to handle subtle differences between images. Zheng et al.²⁶ proposed a bi-directional local alignment approach to reduce the interference of non-dominant objects. Zheng et al.²⁷ proposed a novel cooperative density-aware representation to encode the semantic information. Recently, Zheng et al.²⁸ proposed a semantic-guided generalization enhancement method to solve distributional bias.

Transfer learning based methods

Recent studies have shown that the transfer learning can achieve classification performance comparable to that of meta-learning methods in few-shot image classification tasks. These methods usually combine the pre-trained feature extractors on all base class datasets with any traditional classifier to complete the classification decision of unknown classes. Neg-Cosine²⁹ uses a non-negatively spaced Cosine loss function that increases the distance between the training sample and its parameterized prototype, while S2M2³⁰ borrows manifold blending, an effective regularization technique. In view of the fact that there is no complex and well-performing teacher network in few-shot image classification, self-distillation or mutual learning in knowledge distillation is used as knowledge transfer between models. In addition, the self-supervised task can improve the robustness of the classification model to image transformation, and can also improve the generalization performance of the model³¹. Under the framework of transfer learning, several works have improved the classification results. For example, simpleShot³² can achieve very good performance using the KNN classifier after simple whitening preprocessing of the extracted features, and CBM³³ uses the double-path similarity of the query sample and the correlation calculation cooperation between the sample set and the base class dataset as a measure in the KNN classifier. In addition to non-parametric classifiers, other parametric classifiers, such as logistic regressors³⁴, also showed excellent performance.

Recent research indicates that the transfer learning based methods have achieved superior performance compared to the meta-learning based methods, including PBML³⁵, S2M2³⁰,RFS⁶ and PAL³⁶. However, most of those methods are based on ordinary convolutional neural networks, which can only extract local features of the image. In this paper, the non-local network is used to construct the model, and the non-local mechanism is used to expand the visual receptive field of neurons and obtain global features to improve the discrimination of the classification model. In addition, in view of the good complementarity between local features and global features, this paper proposes to carry out multi-stage fusion of local features and global features, and through parallel fusion in the pre-training stage and serial fusion in the test stage, the information between the local feature network and the global feature network can be fully mined, so as to obtain excellent feature representation, so as to further improve the performance of few-shot image classification. Wu et al.³⁷ presented a geometric invariance module to learn the intra-instance variance. Zheng et al.³⁸ used one-vs-all contrastive learning to improve the generalization ability. Zheng et al.³⁹ proposed a data augmentation strategy and an alternating training manner to overcome domain discrepancies

Methodology

Overall framework

For the given dataset I, it can be randomly partitioned into a base training dataset which has categories, a validation set and a new class dataset for testing, and the category label spaces of each subset are disjoint. The image sample in is denoted as , and its label is denoted as , N categories are randomly sampled from , and K samples are randomly sampled from each category to obtain the support set, and a batch of samples are sampled from the remaining image collection to obtain the query set, and the few-shot image classification task requires the model to use each sample to classify the images. The task requires the model to distinguish N categories using K support samples for each category, which is called the N-way K-shot problem. Suppose that the support set is denoted as , where denotes the image and is its label. The query set is denoted as , where where denotes the query sample. We firstly construct local and global feature networks and fuses them in the training phase using deep mutual learning, then extracts features and performs serial fusion on the images in the support set and the query set using local and global feature networks, and finally trains a logistic regression classifier using the support sample features to complete the classification decision on the support set. The general framework diagram as well as the pseudo-code flow of the proposed method are shown in Fig. 2 and algorithm 1, respectively, and the detailed descriptions of the key steps in the algorithm are as follows.

Fig. 2.

Overview of the Framework. The network contains two stages: pretraining and testing, both of which perform feature fusion. The pretraining stage employs a parallel fusion strategy, realized through inter-network information exchange and learning. The testing stage adopts a serial fusion strategy, accomplished via feature concatenation. The local and global network does not share the parameters..

Algorithm 1.

Multi-stage Fusion of Local and Global Features for Few-shot Image Classification

Construction of local and global feature networks

In order to improve the generalization performance of the classification model, a self-supervised method is introduced to learn jointly with the supervised method during the training process, and for this purpose the image enhancement module is shared in the local and global feature networks to construct the self-supervised task. Here, we mainly choose two self-supervised tasks, rotation prediction(RP) and instance discrimination(ID), so in addition to the base class classifier that implements the original supervised task on the training dataset, the classification model also needs to add a rotation predictor and a projector. Assume that the local feature network is denoted as , which mainly consists of a backbone network with parameter , a base class classifier with parameter matrix , a rotational predictor with parameter matrix , and a projector with parameter matrix . Assume that the global feature network is denoted as , which mainly consists of a backbone network , a base class classifier , a rotational predictor and a projector .

Unlike the local feature network, the global feature network incorporates a non-local computational module into the backbone. This module captures long-range dependencies between global feature points by computing correlations across the entire feature space, instead of restricting to local neighborhood correlations as in local feature networks, as shown in Fig. 3. For a given feature map , the module performs a non-local operation on each of its feature points expressed as follows:

1

where , denote the C-dimensional feature vectors at the ith and jth positions in the feature map, computes the similarity between the feature vectors, denotes the spatial convolution operation, and denotes the normalisation of the features. Then the output of the non-local module is represented as follows:

2

where is the parameters to be optimized in the module.

Fig. 3.

Illustration of non-local module.

Feature fusion in pre-training stage

For model , its backbone network maps the image to a dimensional feature space to get , then goes through a base class classifier and a rotation predictor to get the bth base class score and the rth rotation score , while the projector maps it to a V-dimensional hyper-spherical space to get ,with the help of normalization operations, and the Softmax layer converts the base class scores and the rotation scores into predicted probability values:

3

For , it also maps the image to a dimensional feature space to get , then also get the bth base class score and the rth rotation score , while the projector is also used to get ,with the help of normalization operations, and then a Softmax layer is uesed to predicted probability values:

4

The cross-entropy loss functions between the samples in base dataset predicted by classification probabilities of and and the true value of the samples in base dataset label are calculated in Eq(5) as follows:

5

where denotes the bth component of the mth image sample whose label value is . Similarly, the cross-entropy loss functions between rotation prediction probabilities and true value with label are calculated in Eq(6) as follows:

6

where denotes the bth component of the mth image sample whose rotated label value is .

Assuming that the positive sample image of the image passes through the projectors of and to obtain V-dimensional projective feature representations as and , the computational formula for comparison learning by and is defined as follows:

7

where denotes the temperature parameter. Then the loss function for optimization and of supervised and self-supervised is:

8

where and are the adjustable hyper-parameters. In this stage of fusion of local and global features, an information interaction mechanism between models and is required, for this purpose, we use the KL divergence to measure the mutual information between them, with the expectation that the probability of their outputs to the image base class converges to the same level. The loss function of mutual information from local network to global network is formulated:

9

The mutual information loss function from to is formulated:

10

Then the total loss functions of and trained in the pre-training step is denoted as:

11

where is the adjustable hyper-parameter. The pre-training is mainly related with the joint learning of supervised learning and self-supervised learning and mutual learning. Wherein, the joint loss function of supervised and self-supervised is Lipschitz-smooth and has bounded gradients. And the training loss in the mutual learning is quasi-convex. Therefore, gradient descent converges to a stationary point.

Feature fusion in testing stage

Firstly, and are removed, and then the parameters in the backbone network in the fixed model are used to extract features for the support set and query set, in which the kth support sample passes through and to get the features and ,respectively. Then the feature representation of the support image samples is get by fusing the features:

12

Where denotes the feature splicing operation.Similarly, the features and are obtained for the qth query sample, and . The logistic regression classifier with parameter is trained using support sample features, and the final classification prediction of the query sample is denoted as:

13

Experiments and analysis

Experimental settings

In order to provide a fair comparison with SOTAs, both the the backbone network of local and global feature networks we used are ResNet-12. The first three residual blocks use a maximum pooling operation, while the last residual block uses an average pooling operation. The number of filters in each residual block is 64, 160, 320, and 640, respectively. In the training step, SGD is chosen as the Optimizer, where momentum is set to 0.9 and weight decay is set to 5 10-4.For all experiments, the model is trained for 160 rounds, and the initial learning rate is set to 0.025. When the model is trained for 70 rounds, every 30 rounds the learning rate is adjusted to one-tenth of the original one. The temperature coefficient in contrast learning is set to 0.1. In the evaluation stage, we test 5-way 1-shot and 5-way 5-shot test tasks on each dataset. 2000 episodes are randomly selected from the test dataset, and in each episode, 15 images of each class are randomly selected as query images for testing. The evaluation criterion for the classification performance of the algorithm is the average of the correct rates of all episodes, along with the standard deviation of the correct rates at 95% confidence intervals. The projector is the MLP.

We use five datasets to evaluate the proposed method: Caltech-UCSD Bird-200-2011(CUB)⁴⁰, which is the most widely used benchmark in fine-grained image classification, MiniImageNet dataset⁴¹, tiredImagenet dataset⁴², CIFAR-FS and Few-shot-CIFAR100 (FC100)⁴³.

Results comparison

We compares with the SOTAs in recent years, and the results are shown in Table 1. The comparison methods include DeepEMD²³, RENet⁴⁴, APP2S²¹, MFS⁴⁵, Meta-AdaM¹⁹, ALFA⁵, SM+CADS⁴⁶, BML³⁶, LPE⁴⁷, MBBS⁴⁸, PBML³⁵, DFR⁴⁹, AdaBC⁵⁰. Wherein, DeepEMD²³ used a Earth Mover’s Distance to calculate the similarity between image regions. RENet⁴⁴applied attention mechanisms to establish more discriminative features. APP2S²¹ maps sample features to hyperbolic hyperspaces. MFS⁴⁵embedded shallow self-attention mechanisms inside existing encoder architectures. Meta-AdaM¹⁹ incorporated momentum into the optimization process via a double look-ahead mechanism. ALFA⁵ used a task-adaptive weight update rule that greatly enhances the fast adaptation process. SM+CADS⁴⁶calibrated and utilizes the characteristics of different features. BML³⁶ combined the mutual learning paradigm and binocular parallax. LPE⁴⁷ represented each class by its several different parts with the help of class semantic knowledge. MBBS⁴⁸ employed a mixer module to spread the key semantic on the whole image. PBML³⁵ employed Laplacian estimation to approximate the integral term over the likelihood loss. DFR⁴⁹ adaptively decoupled the discriminative features that are modeled by the classification branch. As shown in Table 1, our method achieves the optimal classification performance in the 5-way 1-shot and 5-way 5-shot classification tasks on the five benchmarks.

Table 1.

Comparison results on benchmarks.’-’means the result is not given by the author.The best results are marked in Bold italic.

Method	Venue	miniImageNet		tiredImageNet		CIFAR-FS		FC100		CUB
		1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
Meta-learning based approach
DeepEMD23	CVPR’20	65.91±0.82	82.41±0.56	71.16±0.87	86.03±0.58			46.47±0.78	63.22±0.71	75.65±0.83	88.69±0.50
RENet44	ICCV’21	67.60±0.44	82.58±0.30	71.61±0.51	85.28±0.35	74.51±0.46	86.60±0.32			79.49±0.44	91.11±0.24
APP2S21	AAAI’22	66.25±0.20	83.42±0.15	72.00±0.22	86.23±0.15	73.12±0.22	85.69±0.16	47.64±0.21	63.56±0.22	77.64±0.19	90.43±0.18
MFS45	CVPR’22	68.32±0.62	82.71±0.46	73.63±0.88	87.59±0.57					79.60±0.80	90.48±0.44
Meta-AdaM19	NIPS’23	59.89±0.49	77.92±0.43	65.31±0.48	85.24±0.35			41.12±0.49	56.14±0.49		-
ALFA5	TPAMI’24	66.61±0.28	81.43±0.25	70.29±0.40	86.17±0.35	76.32±0.43	86.73±0.31	44.54±0.50	58.44±0.42		-
SM+CADS46	TNNLS’25	66.56±0.19	82.74±0.13	72.04±0.22	86.47±0.15	73.23±0.21	86.65±0.14			76.74±0.21	89.45±0.12
Transfer-learning based approach
BML36	ICCV’21	67.04±0.63	83.63±0.29	68.99±0.50	85.49±0.34	73.45±0.47	88.04±0.33			76.21±0.63	90.45±0.36
LPE47	WACV’23	65.74±0.53	83.03±0.33	73.82±0.58	86.76±0.52	74.13±0.71	87.54±0.51	44.82±0.73	61.31±0.54	80.76±0.40	88.98±0.26
MBBS48	TMM’23	65.40±0.20	81.90±0.14	71.74±0.23	86.34±0.15					73.71±0.69	88.13±0.40
PBML35	TNNLS’24	63.60±0.70	81.94±0.44	70.64±0.72	85.39±0.40	73.07±0.59	85.51±0.41	47.92±0.49	62.96±0.51	73.07±0.59	85.51±0.41
DFR49	TNNLS’24	68.12±0.86	82.79±0.59	72.38±0.95	86.00±0.61					81.67±0.78	93.76±0.47
AdaBC50	NEUCOM’25	68.95±0.20	84.62±0.13			75.57±0.21	87.42±0.15			81.20±0.19	91.49±0.10
ICCL51	KBS’25	68.05±0.88	82.46±0.58	74.41±0.92	87.30±0.60	78.51±0.91	88.30±0.59				-
our		69.55±0.43	85.77±0.30	74.01±0.48	87.77±0.32	77.19±0.44	89.71±0.31	47.93±0.42	64.25±0.42	82.26±0.40	93.87±0.18

Parametric analyses

Equation (11) shows the loss function optimised for local and global feature networks, where the coefficients of each loss term are important parameters in the proposed method, and the impact of these parameters on classification performance is analysed in this section on three datasets: miniImageNet, CIFAR-FS and CUB. In the experimental process, it is found that the rotational prediction term loss function coefficient ,the instance discrimination loss term coefficient , and the fusion loss term coefficient take the values in the ranges of [0, 1], [0, 0.1], and [0, 1], respectively. The classification results of different combinations of coefficients are shown in table 2, and it can be seen that the 1-shot and 5-shot tests have the highest performance on all datasets when , or and .

Table 2.

The mean accuracies with 95% confidence intervals of different parameters on MiniImageNet, CIFAR-FS and CUB under the 5-way 1-shot and 5-shot tasks.

Parameters	miniImageNet		CIFAR-FS		CUB
	1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
	69.10±0.44	85.54±0.27	76.30±0.46	89.33±0.31	81.11±0.39	93.30±0.20
	69.55±0.42	85.77±0.27	77.09±0.47	89.56±0.32	81.87±0.42	93.55±0.19
	69.37±0.43	85.45±0.27	77.19±0.44	89.71±0.31	82.26±0.40	93.87±0.18
	69.37±0.41	85.39±0.27	76.93±0.46	89.60±0.31	82.16±0.39	93.67±0.18

Time complexity and parametric capacity analysis

In the training step, assuming that T batches of images are randomly selected for model optimisation, the number of images in each batch is M and the number of classes to which all images belong is , the complexity of the local and global feature networks for base class prediction is , he complexity of the local and global feature networks for rotational prediction is O(4TM), the complexity of the local and global networks for instance adjudication is , and for fusion of local and global networks in the pre-training step, the total complexity is ,with a linear relationship with the number of image samples.

We use both local and global feature networks to build a model with comparable parameter capacity between the two. Assuming that the parameter capacity of any model is C, the evaluation stage fusion method fuses the local feature network and global feature network to extract features, and the parameter capacity of the model is 2C. Inspired by the method⁵², we reduce the number of convolution kernels in the convolutional block in the backbone network, and the results of the performance at this time compared with the performance without capacity parameter reduction on the CUB dataset are shown in Table 3. As shown in Table 3, when the parameter capacity of the local feature network and global feature network is reduced, the performance is comparable to the previous performance.But the runtime is significantly reduced from 3.18 seconds per image to 2.92 seconds per image. The reason for this is that there is redundancy in the original backbone network used, which does not affect the performance after reduction, so applying this technique to this paper’s method allows this paper’s method to avoid a significant increase in parameter capacity.

Table 3.

Comparison results with reduced parameter capacity on CUB dataset.

Parameter capacity	CUB		Runtime(second/per image)
	1-shot	5-shot
W/O decrease()	82.28±0.39	93.87±0.18	3.18
Decrease()	81.95±0.43	93.48±0.18	2.92

Visual analysis

We propose a global feature network to capture long range dependencies within an image, effectively highlighting its overall changes. In order to verify its effectiveness, the feature maps in layer 4 of this network are visualized via Grad-cam⁵³, and the visualization results are shown in Fig. 4. From the results, it can be seen that: (1) for the target objects in the original image in row 1, all the target objects can be highlighted consistently and holistically in the corresponding visualization image in row 2; (2) as shown in the original image in row 3 in the figure, when there are multiple example targets, small-sized targets, target locations near edge locations or complex scenes within the image, all the target objects can be highlighted consistently and holistically in the corresponding visualization image in row 4; and (3) when there are multiple example targets, small-sized targets, target locations close to edge locations or complex scenes within the image, all the target objects are highlighted consistently and holistically in the corresponding visualization image in row 4. visualization image, all the target objects can still be highlighted. The above analysis shows that the global feature network can effectively extract global features.

Fig. 4.

Visualization of feature maps in the global feature network.

In addition, in order to further verify the complementary nature of the local and global feature networks, we visualize what features are retained in their respective backbone networks. To this end, the features extracted from the backbone network are transformed into RGB images using the DIP technique⁵⁴ in this section. DIP adopted the autoencoder network with six blocks to produce the reconstruction images. And it used the loss as the reconstruction loss.The reconstruction results of the local and global feature networks are shown in Fig. 5. From the results, it can be seen that (1) for the four original images in row 1, local singularity regions appear in the content of each image, the red head of the white goose in row 1, the black dots on the body of the ladybird in row 2, the black pattern on the part of the dog’s torso in row 3, and the light dots within the bird’s image in row 4; and (2) when comparing the reconstructed images of rows 2 and 3, it can be found that the local features highlight the local singular regions within the image, such as the red head of the white goose, the black dots of the ladybird, the black pattern of the dog, and the bright spots within the image, while the global features highlight the target object as a whole within the image, and do not highlight the local singular regions therein, which suggests that the local and global feature networks in our method are capable of extracting complementary features; (3) Comparison of the reconstructed images of rows 2 and 4, it can be found that after the feature fusion in the pre-training stage, the highlighting of local singular regions in the reconstructed images of local features is weakened, and comparing the reconstructed images in rows 3 and 5, it can be found that the reconstructed images of global features after the feature fusion in the pre-training stage increase the highlighting of local singular regions, which indicates that the feature fusion algorithm in the pre-training stage has the effectiveness; (4) Comparing the reconstructed images in rows 4 and 5, it can be found that the reconstructed images of local features in the pre-training stage are able to extract complementary features. Reconstructed images are compared, it can be found that after the fusion in the pre-training stage, the reconstructed images of local and global features are similar, but still differentiated, which provides a basis for their fusion in the evaluation stage.

Fig. 5.

Illustration of reconstruction images of features.

Ablation study

To show the effectiveness of our method, we conduct the ablation studies on three datasets: miniImageNet, CIFAR-FS, and CUB.

Evaluation of self-supervision

Both local and global feature networks are trained using joint supervised and self-supervised learning, where self-supervised learning chose the rotation prediction (RP) and instance discrimination (ID). We first evaluate the effectiveness of self-supervision in the training process of local and global networks. denotes the rotation prediction using local features, and denotes the instance discrimination using local features, and denotes combination of Rot and ID using local features. Similarly, we can define , and . As the results shown in Fig. 6, the local feature network and the global feature network achieve consistent results in both the 5-way 1-shot and 5-way 5-shot tests on all three datasets, with the correct rate of being higher than that of and , and the correct rate of being higher than that of and , indicating that the two self-supervised tasks can facilitate the extraction of different invariant features by local and global feature networks, and their linear combination can jointly improve the classification performance of the model.

Fig. 6.

Performance of self-supervised learning.

The evaluation of multi-stage feature fusion

In order to illustrate the effectiveness of the multi-stage fusion of local and global features proposed in this paper, the classification results of un-fused local and global features, single-stage fusion in pre-training stage or testing stage, and multi-stage fusion in pre-training and testing step are given in this section, respectively. As shown the results in table 4, the methods Local an Global are denoted as unfused local and global features respectively, and are denoted as methods for single-stage fusion of local and global features in the pre-training stage, are denoted as methods for single-stage fusion of local and global features in the testing stage, and are denoted as methods for multi-stage fusion of local and global features multi-stage fusion methods, and all the methods employ Logistic regression.

Table 4.

The mean accuracy with 95% confidence intervals of multi-stage fusion of local and global features on MiniImageNet, CIFAR-FS and CUB under the 5-way 1-shot and 5-shot tasks.

Method	Backbone	miniImageNet		CIFAR-FS		CUB
		1-shot	5-shot	1-shot	5-shot	1-shot	5-shot
Local(baseline)	ResNet12	67.03±0.44	84.05±0.29	74.07±0.46	88.46±0.32	78.09±0.42	91.72±0.20
Global	ResNet12	67.52±0.44	83.93±0.28	73.96±0.46	87.99±0.32	78.58±0.42	92.00±0.20
	ResNet12	68.36±0.43	85.36±0.28	75.47±0.45	89.11±0.32	80.38±0.40	93.27±0.19
	ResNet12	69.30±0.42	85.22±0.28	77.05±0.44	89.45±0.31	82.13±0.39	93.58±0.18
	ResNet12	68.67±0.43	84.98±0.27	76.79±0.46	89.34±0.32	81.78±0.41	93.16±0.17
	ResNet12	69.55±0.42	85.77±0.27	77.19±0.46	89.71±0.31	82.28±0.39	93.87±0.18

From the results , we can see that:

1. The multi-stage fusion approach improves the performance of both local and global feature networks on 5-way 1-shot and 5-shot tasks. Specifically, on the miniImageNet dataset, is higher than Local by 2.52% and 1.72%, and Global by 2.03% and 1.84%; on the CIFAR-FS dataset, it is higher than Local by 3.12% and 1.25%, and Global by 3.23% and 1.72% on the CUB dataset, higher than Local 4.19% and 2.15%, and higher than Global 3.7% and 1.87%. From the above results, it can be seen that the proposed multi-stage fusion method is highly effective for feature fusion.

2. On the 5-way 1-shot and 5-shot tasks, the multi-stage fusion method performs higher compared to the single-stage fusion method that fuses only at the pre-training stage. Specifically, on the miniImageNet dataset, outperforms by 1.19% and 0.41%, and by 0.25% and 0.55%; on the CIFAR-FS dataset, it outperforms by 1.72% and 0.6%, and by 0.14% and 0.26%; on the CUB dataset, higher than 1.9% and 0.6%, and higher than 0.15% and 0.29%.

3. On the 5-way 1-shot and 5-shot tasks, the multi-stage fusion method performs higher compared to the single-stage fusion method that fuses only in the test phase. Specifically, is higher than by 0.88% and 0.79% on the miniImageNet dataset, by 0.4% and 0.37% on the CIFAR-FS dataset, and by 0.5% and 0.71% on the CUB dataset.

Conclusion

For the shortcomings of the current networks only extract the local features via convolutional neural network, which cannot guarantee the discriminative feature representation, we propose a few-shot image classification model based on the global feature network, so as to achieve the purpose of portraying the overall changes of the image. On this basis, we further propose a few-shot image classification algorithm with multi-stage fusion of local and global features. The algorithm is capable of exploiting the complementary characteristics of local and global feature networks and fusing them in parallel and serial in the pre-training and testing stage, respectively. In this paper, exhaustive experiments are conducted on several few-shot image classification datasets. The results of the ablation experiments and visual analysis experiments show that the local and global networks are capable of extracting complementary features, and the proposed multi-stage feature fusion algorithm is able to effectively fuse the local and global features to achieve better discriminative features. Compared with several SOTAs on five benchmarks, the proposed method achieves a better performance. On the one hand, the future work will focus on how to develop promising local and global feature embeddings. On the other hand, since this work used a stationary fusion rule, future work will explore adaptive fusion strategies to dynamically adjust the weights of local and global features. In addition, we will also address the issue of doubled parameters caused by employing two networks.

Notation

The symbols used in this paper are listed in Table 5.

Table 5.

Table of symbol introductions.

Symbol	Definition
	Dataset
	Base training dataset
	Categories of
	Validation set
	New class dataset
	Image sample in
	Label of
	Support sample set
	Query set
	Image enhancement module
	Local feature network
	Backbone network in
	Base class classifier in
	Rotational predictor in
	Projector in
	Global feature network
	Backbone network in
	Base class classifier in
	Rotational predictor in
	Projector in

Author contributions

Yi Gu contributed to drafting the article. Zhangyan Yao made the analysis and interpretation of data associated with the work contained in the article. Sai Yang approved the final version of the article as accepted for publication. All authors reviewed the manuscript.

Data availability

The datasets are available on the website: https://github.com/tristandeleu/pytorch-meta?tab=readme-ov-file.

Declarations

Competing interests

The authors declare no competing interests.