Channel-spatial attention modules in convolutional neural networks for image classification

Abstract

Many studies have established that the attention mechanism has great potential in improving the performance of Convolutional Neural Networks (CNNs) in image classification problems in recent years. Combining channel and spatial attention modules is one of the different kinds of attention mechanisms that are inspired by the visual perception of the human brain. So far, no paper has considered both parallel and sequential states of combining channel-spatial attention modules, so that while comparing them comprehensively and accurately, it can be definitively said which of them is more optimal in terms of a better balance between efficiency and computational complexity of the model. In this paper, we introduced two new types of channel-spatial attention modules, the Parallel Channel-spatial Attention Module (PCSAM) and the Sequential Channel-spatial Attention Module (SCSAM), to embed in the architecture of any CNN. Each of the proposed attention modules is composed of a channel and spatial attention sub-modules. The Channel Attention Module (CAM) and Spatial Attention Module (SAM) help the network in extracting the channels related to the architecture of the Region of Interest (RoI) and its location in the input feature maps, respectively. We increase the representation power of the attention-based networks by extracting the features using Global Average Pooling (GAP) and Global Maximum Pooling (GMP) in the CAM and SAM. Also, the Dilation Convolution (DC) layer is employed in the structure of the SAM instead of the standard convolution to better focus on the RoI in the feature maps. The PCSAM and SCSAM are implemented in the architecture of the ResNet18 and MobileNetv4 to produce the ResNet18PCSAM, ResNet18SCSAM, MobileNetv4PCSAM, and MobileNetv4SCSAM. All networks are trained and evaluated on three general image classification datasets, the CIFAR-10, CIFAR-100, and Tiny-ImageNet, with the same experimental conditions for 50 epochs. The classification results in the test step show that the MobileNetv4SCSAM has a better efficiency than other architectures on all datasets. It also achieved higher performance than the previous existing channel-spatial attention modules.

Introduction

According to cognitive science, enormous amounts of data are received by the human brain. Since the brain’s capacity is limited, it pays attention only to the necessary information by assigning weight to the input data. The cognitive attention mechanism is a procedure in the brain that focuses on important information and ignores others. The visual information are processed in the ventral and dorsal independent pathways based on the Two Visual Systems Hypothesis (TVSH). The ventral stream extracts the properties of color, form, and objects, while the dorsal stream engages in interpreting spatial organization, guiding action, and spatial attention. Hence, they are also called ’What-information’ and ’Where-information’ pathways^1–3.

Convolutional Neural Networks (CNNs) are the most famous kind of deep learning methods that are employed for a series of pattern recognition applications, including classification, detection, segmentation, and control problems. Using channel and spatial attention mechanisms in the architecture of different CNNs can simply increase their performance by recalibrating the weights of features. They enable the network to assign weights to information from different channels and positions related to the structure and location of the Region of Interest (RoI), making the model focused on features and locations that are more important to the image classification task^4–6.

Each of the channel and spatial attention modules analyzes the structure of the objects and their positions in the feature maps, respectively. Hence, they process information that determines what and where to pay attention. Therefore, the channel and spatial attention modules help each deep learning model to focus on the shape and location of RoI in the feature maps, but network complexities are increased by employing them. As a result, to produce a balance between the efficiency of the model and its complexity, examining how modules are merged and comprehensively comparing each state is very important in improving network performance^7–10.

In this paper, two attention sub-modules, including channel and spatial, are designed and developed in parallel and sequential types, and they are embedded in the architecture of the two state-of-the-art CNNs. The training and testing of the baseline and attention-based models are performed on the three general image classification datasets under the same conditions to determine which one achieves a better balance between efficiency metrics and complexity.

Our main contributions are as follows:

• By comparing different combinations of channel and spatial attention modules in this study and comprehensively evaluating the advantages and disadvantages of each state, the proposed channel-spatial attention module has been introduced.

• The Global Average Pooling (GAP) and the Global Maximum Pooling (GMP) are used in the channel and spatial modules to leverage the benefits of both approaches for offering an extra robust and informative demonstration of the input feature map.

• Dilation Convolution (DC) layer¹¹ can extract features from a larger kernel size while training and can well capture the fine details of the feature map. We employed a DC layer instead of a standard convolution layer to pay better attention to the RoI in the feature maps and to improve the efficiency of the spatial sub-module.

In the section related works, a review of the CNNs for image classification tasks and the attention mechanism is provided. In the proposed methodology section, the employed datasets, the architecture of the proposed networks, and attention modules are explained. The experiment setup, hyperparameter tuning, training and test steps, image classification results, ablation study, statistical analysis, and visualizing feature maps and Gradient-weighted Class Activation Mappings (Grad-CAMs) are presented in the section experiments. In the section discussion, we analyze the experimental results and performance comparison. The summary of findings and future works is summarized in the section conclusion.

Related works

In image classification using CNNs, the class of each sample is determined based on the features extracted by the network. Many different CNNs have been introduced for image classification tasks in recent years. AlexNet did not perform well for image classification due to the use of large filters and their lack of diversity. Although ZFNet achieved better results by using smaller and different filters, it had challenges such as growing the computational complexity of the network and gradient vanishing with the increase in the number of layers. Inception (GoogleNet) and ResNet could overcome these problems. MobileNet series are lightweight CNNs that were introduced to generate a balance between efficiency and complexity, and for use on devices with limited memory^12–14.

In recent years, channel-spatial attention modules have been employed in image classification problems to help the networks in highlighting RoI areas of the inputs^15–17. SE block is the first channel module to perform dynamic channel-wise feature attention¹⁸. It consists of two sub-modules: squeeze and excitation. They extract and weight the channels. Other channel attention blocks are inspired by the SE block. Global Second-order Pooling (GSoP)¹⁹and Frequency Channel Attention Networks (FCANet)²⁰ blocks have improved the squeeze sub-module, while Efficient Channel Attention (ECANet)²¹ has enhanced the excitation module. Style-Based Recalibration Module (SRM)²² and Gated Channel Transformation (GCT)²³ have attempted to enhance both squeeze and excitation sub-modules.

Recurrent Attention Model (RAM) is the first spatial module that is composed of Recurrent Neural Networks (RNNs) and Reinforcement Learning (RL)²⁴. Gather-Excite Network (GENet) emphasizes predicting the relevant region implicitly by providing a recalibration function in the spatial domain²⁵. Stand Alone Self-Attention (SASA) improves speed, number of parameters, and quality of CNN by replacing local self-attention instead of the spatial convolutions²⁶. Vision Transformer (ViT) employs a Multi-head Attention (MHA) module that takes a sequence as input²⁷. RAM and GENet select the RoI based on prediction, while SASA and ViT collect the information of the RoI using the attention map.

The first channel-spatial attention block is the Convolutional Block Attention Module (CBAM), where the attention blocks are connected in sequential state²⁸. Bottleneck Attention Module (BAM) was introduced as the first parallel channel-spatial attention block²⁹. They are different in the type of pooling and convolution layer in their sub-modules. Hybrid Attention Module (HAM)³⁰, Spatial Channel Attention (SCA)³¹, Efficient Spatial and Channel Attention (ESCA)³², and Spatial Perception and Channel Information Interaction (SPCII)³³ including channel-spatial atterntion blocks that presented for embedding in the architecture of the CNNs.

A summary of the previous channel-spatial attention modules based on their arrangement is presented in Table 1.

Table 1.

A summary of the previous channel-spatial attention modules.

Type	Module
Parallel	BAM, SCA, CSAM
Sequential	CBAM, HAM, ESCA, SPCII, CSAB, VDAB, NPVDAB

Proposed methodology

The employed datasets, the architecture of the proposed networks, and the structure of the suggested attention modules are described in this section.

The used datasets

We employed three standard and general image classification datasets, including CIFAR-10³⁴, CIFAR-100 ³⁴, and Tiny-ImageNet ³⁵. The CIFAR-10 dataset comprises 10 classes with 60,000 samples. It has images with RGB color and resolution of pixels. The number, color, and resolution of samples in the CIFAR-100 dataset are similar to the CIFAR-10 dataset, but it has 100 categories. The number of classes and images of the Tiny-ImageNet dataset is 200 and 120,000, respectively. It has samples with RGB format and pixels. The test data of the Tiny-ImageNet dataset doesn’t have labels. The original images of the datasets are used for training and evaluating the networks. Table 2 shows the splitting of samples on each dataset in this study.

Table 2.

The splitting of samples.

Dataset	Training	Test	Total
CIFAR-10	50,000	10,000	60,000
CIFAR-100	50,000	10,000	60,000
Tiny-ImageNet	100,000	10,000	110,000

The architecture of the proposed networks

Two different types of CNNs, including ResNet18³⁶ and MobileNetv4³⁷, are used for embedding the suggested attention modules. The architecture of the proposed networks in this study is described in the following.

ResNet18. Basic Blocks (BB0 and BB1) are the backbone of ResNet18. Their structure is similar to each other, but BB1 has a residual layer more than BB0. Table 3 demonstrates the architecture of the proposed attention-based ResNet18. Conv, MP, AP, BN, ReLU, and CSAM denote Convolution, Maximum Pooling, Average Pooling, Batch Normalization, Rectified Linear Unit activation function, and Channel-spatial Attention Module, respectively.

Table 3.

The architecture of the proposed attention-based ResNet18.

Block	Layer	Input dimension ()	K	Stride	Output dimension ()
Conv2D	Conv/MP/AP/BN/CSAM/ReLU			2
BB0	Conv/BN/ReLU/Conv/BN/CSAM/ReLU			1
BB0	Conv/BN/ReLU/Conv/BN/CSAM/ReLU			1
nBB1	Conv/BN/ReLU/Conv/BN/CSAM/ReLU			2
	Conv/BN			2
BB0	Conv/BN/ReLU/Conv/BN/CSAM/ReLU			1
BB1	Conv/BN/ReLU/Conv/BN/CSAM/ReLU			2
	Conv/BN			2
BB0	Conv/BN/ReLU/Conv/BN/CSAM/ReLU			1
BB1	Conv/BN/ReLU/Conv/BN/CSAM/ReLU			2
	Conv/BN			2
BB0	Conv/BN/ReLU/Conv/BN/CSAM/ReLU			1
Linear	Classifier		-	-	10

MobileNetv4. MobileNetv4 is the latest generation of MobileNets that uses different instances of Universal Inverted Bottleneck (UIB), including Fused Inverted Bottleneck (Fused IB), IB, Extra DepthWise separable convolution (ExtraDW), and ConvNext. Table 4 shows the architecture of the proposed attention-based MobileNetv4. PW and DW denote PointWise and Depthwise convolutions, respectively.

Table 4.

The architecture of the proposed attention-based MobileNetv4.

Block	Layer	Input dimension ()	DW	DW	Stride	Output dimension ()
Conv2D	Conv/BN/CSAM/ReLU		-		2
Fused IB	Conv/BN/CSAM/ReLU/PW		-		2
Fused IB	Conv/BN/CSAM/ReLU/PW		-		2
ExtraDW	DP/PW/DP/PW				2
IB	PW/DW/PW		-		1
IB	PW/DW/PW		-		1
IB	PW/DW/PW		-		1
IB	PW/DW/PW		-		1
ConvNext	DP/PW/PW			-	1
ExtraDW	DP/PW/DP/PW				2
ExtraDW	DP/PW/DP/PW				1
IB	PW/DW/PW		-		1
IB	PW/DW/PW		-		1
IB	PW/DW/PW		-		1
IB	PW/DW/PW		-		1
Conv2D	Conv/BN/CSAM/ReLU		-		1
AP	AP		-		1
Conv2D	Conv/BN/CSAM/ReLU		-		1
Conv2D	Conv/BN/CSAM/ReLU		-		1
Linear	Classifier		-	-	-	10

ResNet18 and MobileNetv4 are designed for 1000-class classifications, and we modified their output layers for implementation on the datasets. Many experiments are performed to find the best location of the suggested attention modules in the architecture of the proposed networks. The experimental results showed that embedding them after BN in the convolutional layers led to higher performance metrics. Thus, they are used in the convolution layer and the BBs of the ResNet18, and the convolution layer and Fused IB blocks of the MobileNetv4. As a result, the fine-tuned ResNet18 and MobileNetv4 and the attention-based networks (ResNet18PCSAM, ResNet18SCSAM, MobileNetv4PCSAM, and MobileNetv4SCSAM) are used to implement on the datasets.

The suggested attention modules

Fig. 1 displays the schematic diagram of the suggested attention modules in this study. We introduced two different kinds of CSAM, including the Parallel Channel-spatial Attention Module (PCSAM) and the Sequential Channel-spatial Attention Module (SCSAM).

Fig. 1.

The schematic diagram of the suggested attention modules. (a) and (b) respectively indicate the architecture of the proposed parallel and sequential channel-spatial modules, while (c) is the structure of the channel and spatial attention modules.

The architecture of the PCSAM and SCSAM is explained in the following. The PCSAM. The Channel Attention Module (CAM) and Spatial Attention Module (SAM) work in parallelly in the PCSAM. They are learn the inter-channel and inter-spatial relationships associated with the structure and location of RoI. CAM and SAM extract what and where information from feature maps, similar to the ventral and dorsal pathways in the human brain. To calculate the channel and spatial attentions, the GAP and GMP operations are taken along the spatial and channel axes, respectively. In the CAM, a Shared Multi-Layer Perceptron (Shared MLP) with one hidden layer was employed to estimate attention across channels, while in the SAM, a DC layer was embedded to extract attention in the spatial domain. We decrease the length of the hidden layer of the Shared MLP, inspired by the SE block, to improve the performance of the proposed attention modules. Assuming the input feature map has Channel, Height and Width dimensions (), the final output of the CAM is a channel attention map with only the Channel dimension (), while the final output of the SAM is a spatial attention map with the Height and Width dimensions (). The mathematical calculation of the PCSAM is shown in Eqs. (1-5).

1

2

3

4

5

where +, , and denote element-wise summation, multiplication, and the sigmoid activation function, respectively.

The SCSAM. The CAM and SAM are connected sequentially in the SCSAM. First, the channels related to the RoI are identified by CAM, and then the focus is on its location using SAM. The mathematical calculation of the SCSAM is presented in Eqs. (6-8).

6

7

8

Experiments

The experiment setup, hyperparameter tuning, training and test steps, image classification results, ablation study, statistical analysis, and visualizing feature maps and Grad-CAM are explained in this section.

Experiment setup

The proposed methodology is implemented by the Jupyter Notebook tool of the Python programming language in the Google Colaboratory environment with the Graphics Processing Unit (GPU). The experimental hardware and software environments in this study are shown in Table 5.

Table 5.

Experimental hardware and software environments.

Hardware/Software	Parameter/Version
Operating System	Windows 11-64 bit
CPU	Intel (R) Core (TM) i7-8565U
GPU	Intel (R) UHD Graphics 620
Python	3.11.13
PyTorch	2.6.0+cu124
Numpy	2.0.2
Opencv	4.12.0

Hyperparameter tuning

We first fine-tuned the networks on the datasets in the training step, and then experimentally found the optimal hyperparameter values. Table 6 shows the hyperparameter settings of the networks in the experiments.

Table 6.

Hyperparameter setting of the networks.

Hyperparameter	Variables	Selected
Batch size	4, 8, 16, 32, 64, 128	32
Epochs	30, 40, 50, 60, 70, 80, 90, 100	50
Learning rate	0.1, 0.01, 0.001, 0.0001	0.01
Momentum	0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9	0.9
Weight decay	0.5, 0.05, 0.005, 0.0005	0.0001
Loss function	-	Cross-entropy
Optimizer	-	SGD

Training and test steps

The accuracy and loss curves of the ResNet18 and MobileNetv4 models on the datasets during the training and test processes are shown in Figs. 2 and 3. As we can see, all networks reached a good convergence, and no overfitting occurred. By comparing the curves of the ResNet18 and MobileNetv4, we conclude that the MobileNetv4 models have better convergence than the ResNet18 models.

Fig. 2.

The accuracy and loss curves of the ResNet18 models. (a–c) indicate CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, whereas () are ResNet18, ResNet18PCSAM, and ResNet18SCSAM, respectively.

Fig. 3.

The accuracy and loss curves of the MobileNetv4 models. (a–c) denote CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, while () are MobileNetv4, MobileNetv4PCSAM, and MobileNetv4SCSAM, respectively.

Image classification results

For comparing the efficiency of the suggested attention modules with the previous related modules, the BAM, CBAM, VDAB, and NPVDAB were also embedded in the architecture of the ResNet18 and MobileNetv4, and they were implemented on the datasets under the same experimental conditions as other networks. The classification results of the networks on the datasets are compared to evaluate the performance advantages and disadvantages of each model. We calculated the number of parameters, model size, Floating-point operations per second (FLOPs), Multiply-accumulate operations (MACs), inference time, Top-1 and Top-5 accuracies, and standard deviation of the models that are presented in Tables 7, 8 and 9. The M, MB, GB, and ms are abbreviations of million, megabyte, gigabyte, and millisecond, respectively. The mean standard deviation values are computed for five random seeds. The best accuracy values under different architectures are shown in bold. As we can see, the MobileNetv4 models obtained higher accuracy than the ResNet18 models, while they have lower computational complexity than the ResNet18 models. The MobileNetv4SCSAM achieved better accuracy values on all datasets than other architectures. It improves Top-1 accuracy of the MobileNetv4 by 1.41%, 1.62%, and 1.04% on the CIFAR-10, CIFAR-100, Tiny-ImageNet datasets, respectively, while these improvements have been 1.31%, 1.5%, and 0.91% by the MobileNetv4PCSAM. The MobileNetv4 SCSAM increases Top-5 accuracy of the MobileNetv4 by 0.39%, 0.7%, and 0.91% on the CIFAR-10, CIFAR-100, Tiny-ImageNet datasets, respectively, whereas these increases have been 0.29%, 0.49%, and 0.63% by the MobileNetv4PCSAM. Hence, we conclude that the SCSAM has better accuracy values than the PCSAM, BAM, CBAM, VDAB, and NPVDAB. Also, the MobileNetv4SCSAM obtained the lowest standard aviation values among other models. It imposes fewer parameters, FLOPs, MACs, and inference time values on the ResNet18 and MobileNetv4 than the PCSAM, BAM, and CBAM. Therefore, comparing classification results shows that the SCSAM has established a better balance between accuracy and computational complexity.

Table 7.

Comparison between different architectures on the CIFAR-10 dataset.

Architecture	Parameters (M)	Model size (MB)	FLOPs (GB)	MACs (GB)	Inference time (sec)	Top-1 accuracy (%)	Top-5 accuracy (%)	Standard deviation
ResNet18	11.17	58.14	106.81	53.32	1.451	65.95	93.75
ResNet18BAM	11.2	61.36	107.17	66.23	1.456	66.23	93.88
ResNet18CBAM	11.19	59.17	106.86	53.33	1.452	66.18	93.81
ResNet18VDAB	11.19	59.17	106.86	53.33	1.452	66.16	93.7
ResNet18NPVDAB	11.17	59.05	106.81	53.32	1.451	66.05	93.79
ResNet18PCSAM	11.96	69.36	128.81	64.27	1.75	67.13	94.09
ResNet18SCSAM	11.18	59.17	106.84	53.32	1.452	67.41	94.15
MobileNetv4	2.51	10.64	0.97	0.48	0.013	91.37	99.61
MobileNetv4BAM	3.17	13.7	1.11	0.55	0.015	91.59	99.73
MobileNetv4CBAM	2.83	13.37	1.1	0.54	0.015	91.53	99.68
MobileNetv4VDAB	2.83	13.37	1.1	0.54	0.015	91.5	99.59
MobileNetv4NPVDAB	2.51	10.85	0.98	0.48	0.013	91.47	99.65
MobileNetv4PCSAM	6.87	26.3	3.66	1.82	0.05	92.68	99.9
MobileNetv4SCSAM	2.83	13.36	1.09	0.54	0.15	92.78	100

Table 8.

Comparison between different architectures on the CIFAR-100 dataset.

Architecture	Parameters (M)	Model size (MB)	FLOPs (GB)	MACs (GB)	Inference time (ms)	Top-1 accuracy (%)	Top-5 accuracy (%)	Standard deviation
ResNet18	11.22	58.31	106.82	53.32	1.451	43.78	84.38
ResNet18BAM	11.24	61.54	107.18	53.49	1.456	43.96	84.59
ResNet18CBAM	11.23	59.34	106.87	53.34	1.452	43.92	84.46
ResNet18VDAB	11.23	59.34	106.87	53.34	1.452	43.85	84.41
ResNet18NPVDAB	11.22	59.22	106.82	53.32	1.451	43.9	84.45
ResNet18PCSAM	12.01	69.54	128.81	64.27	1.75	44.98	84.75
ResNet18SCSAM	11.23	59.34	106.85	53.33	1.452	45.63	84.93
MobileNetv4	2.62	11.08	0.99	0.49	0.013	69.57	90.28
MobileNetv4BAM	3.29	14.14	1.14	0.56	0.015	69.89	90.64
MobileNetv4CBAM	2.94	13.81	1.13	0.56	0.015	69.81	90.5
MobileNetv4VDAB	2.94	13.81	1.13	0.56	0.015	69.77	90.49
MobileNetv4NPVDAB	2.62	11.29	1	0.49	0.013	69.63	90.41
MobileNetv4PCSAM	6.89	26.14	3.66	1.83	0.05	71.07	90.77
MobileNetv4SCSAM	2.94	13.8	1.12	0.55	0.015	71.19	90.98

Table 9.

Comparison between different architectures on the Tiny-ImageNet dataset.

Architecture	Parameters (M)	Model size (MB)	FLOPs (GB)	MACs (GB)	Inference time (ms)	Top-1 accuracy (%)	Top-5 accuracy (%)	tandard deviation
ResNet18	11.27	58.51	106.83	53.33	1.451	23.77	60.38
ResNet18BAM	11.29	61.73	107.19	53.49	1.456	23.89	60.51
ResNet18CBAM	11.28	59.54	106.88	53.34	1.452	23.84	60.43
ResNet18VDAB	11.28	59.54	106.88	53.34	1.452	23.82	60.4
ResNet18NPVDAB	11.27	59.42	106.83	53.33	1.451	23.81	60.4
ResNet18PCSAM	12.06	69.74	128.82	64.28	1.75	25.1	60.98
ResNet18SCSAM	11.28	59.54	106.86	53.33	1.452	25.32	61.15
MobileNetv4	2.75	11.57	1.01	0.5	0.014	62.13	83.05
MobileNetv4BAM	3.29	14.63	1.14	0.56	0.015	62.22	83.32
MobileNetv4CBAM	3.07	14.3	1.15	0.57	0.016	62.16	83.25
MobileNetv4VDAB	3.07	14.3	1.15	0.57	0.016	62.14	83.23
MobileNetv4NPVDAB	2.75	11.78	1.02	0.5	0.014	62.15	83.19
MobileNetv4PCSAM	6.93	26.27	3.67	1.83	0.05	63.04	83.68
MobileNetv4SCSAM	3.07	14.29	1.14	0.56	0.015	63.17	83.96

Ablation study

In this section, two sets of ablation experiments are performed to evaluate the role of GAP, GMP, and the DC layers on the effectiveness of the proposed attention modules. In the first experiments, the impact of GAP, GMP, and GAP+GMP layers on the performance improvement of the attention-based MobileNets on the Tiny-ImageNet dataset is analysed, and the classification results are shown in Table 10. It confirms that the combination of GAP and GMP in the suggested attention modules causes more improvement in the efficiency of the network.

Table 10.

Ablation experiment results with different pooling layers.

Architecture	Parameters (M)	Model size (MB)	FLOPs (GB)	MACs (GB)	Inference time (ms)	Top-1 accuracy (%)	Top-5 accuracy (%)
MobileNetv4	2.75	11.57	1.01	0.5	0.014	62.13	83.05
MobileNetv4PCSAM (GAP)	6.93	26.27	3.66	1.83	0.05	62.52	83.18
MobileNetv4PCSAM (GMP)	6.93	26.27	3.66	1.83	0.05	62.78	83.43
MobileNetv4PCSAM (GAP+GMP)	6.93	26.27	3.67	1.83	0.05	63.04	83.68
MobileNetv4SCSAM (GAP)	3.07	14.29	1.13	0.56	0.015	62.59	83.35
MobileNetv4SCSAM (GMP)	3.07	14.29	1.13	0.56	0.015	62.86	83.62
MobileNetv4SCSAM (GAP+GMP)	3.07	14.29	1.14	0.56	0.015	63.17	83.96

The effect of the standard and dilation convolution layers on the improvement of the efficiency of the attention-based MobileNets on the Tiny-ImageNet dataset is evaluated in the second set of experiments, and the classification results are presented in Table 11. It proves that the DC layer is more efficient than the standard convolution layer.

Table 11.

Ablation experiment results with different convolution layers.

Architecture	Parameters (M)	Model size (MB)	FLOPs (GB)	MACs (GB)	Inference time (ms)	Top-1 accuracy (%)	Top-5 accuracy (%)
MobileNetv4	2.75	11.57	1.01	0.5	0.014	62.13	83.05
MobileNetv4PCSAM (Standard)	6.93	26.28	3.68	1.84	0.05	62.64	83.52
MobileNetv4PCSAM (Dilation)	6.93	26.27	3.67	1.83	0.05	63.04	83.68
MobileNetv4SCSAM (Standard)	3.07	14.3	1.15	0.57	0.016	62.77	83.64
MobileNetv4SCSAM (Dilation)	3.07	14.29	1.14	0.56	0.015	63.17	83.96

Statistical analysis

To study the difference between the accuracy of the networks on the datasets and to measure the role of each proposed attention in their performance, Analysis of Variance (ANOVA) is used. Fig. 4 shows a box plot summarizing the distribution of accuracy values for each architecture across the datasets in the test step. It indicates that the accuracy values of the MobileNetv4 models are higher than all networks. Also, the attention-based models achieved higher accuracy than the baseline models. Among all networks, the MobileNetv4SCSAM obtained the highest accuracy on all datasets compared to others.

Fig. 4.

Box plot summarizing the distribution of accuracy values for each architecture across the datasets. (a–c) are the accuracy values of the networks on the CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, respectively.

A box plot summarizing the distribution of classification accuracy of the different network sets on each dataset during the test step is displayed in Fig. 5. It confirmed that the networks obtained higher accuracy values on the CIFAR-10 dataset, whereas they had the lowest accuracy on the Tiny-ImageNet dataset. The accuracy values of all networks on the CIFAR-100 dataset are higher and lower than their accuracies on the Tiny-ImageNet and CIFAR-10 datasets, respectively.

Fig. 5.

Box plot summarizing the distribution of classification accuracy of the different network sets on each dataset. (a–f) are the accuracy values of the ResNet18, ResNet18PCSAM, ResNet18SCSAM, MobileNetv4, MobileNetv4PCSAM, and MobileNetv4SCSAM, respectively on the datasets.

Visualizing feature maps and Grad-CAMs

We visualized the feature maps of an input image of each dataset in the middle convolution layers of all models to understand what happens inside each network on each of the images as it processes. The input images and their feature maps from the middle convolution layers of each architecture on the datasets are displayed in Fig. 6. It confirmed that the border and structure of desired shapes in the images were separated from the background better in the extracted feature maps by the attention-based networks. This distinction is greater in the MobileNetv4 models than in the ResNet18 models. The MobileNetv4SCSAM extracted the best feature maps from input images compared to other architectures. Also, it has better feature maps than MobileNetv4 and MobileNetv4PCSAM. Therefore, we can say that the SCSAM is more focused on the shape and texture of the RoI in the input images than the PCSAM.

Fig. 6.

Feature maps visualization results from the middle convolution layers of the networks on the datasets. (a–c) are the images and their feature maps from the CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, whereas (1) include input images, () are the output feature maps of the middle convolution layers of the ResNet18, ResNet18PCSAM, ResNet18SCSAM, MobileNetv4, MobileNetv4PCSAM, and MobileNetv4SCSAM, respectively.

The heat map produced by Grad-CAM can show the regions that the network focuses on in the input image, and in that way aids in understanding how the model makes decisions. The input images and their Grad-CAMs from the middle convolution layers of each architecture on the datasets are displayed in Fig. 7. As we can see, the MobileNetv4 models have better heat maps than the ResNet18 models. Also, the attention-based networks cover more of the RoI than the baseline models. It confirmed that when we add the PCSAM and SCSAM to the CNNs, a larger area of the RoI is covered by Grad-CAM. Among all architectures, the MobileNetv4SCSAM covers more regions of the RoI than others. The SCSAM can guide the network to focus on the more significant overall and comprehensive features while ignoring irrelevant features compared to the PCSAM.

Fig. 7.

Grad-CAMs visualization results from the middle convolution layers of the networks on the datasets. (a–c) are the images and their Grad-CAMs from the CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets, whereas (1) include input images, () are the output Grad-CAMs of the middle convolution layers of the ResNet18, ResNet18PCSAM, ResNet18SCSAM, MobileNetv4, MobileNetv4PCSAM, and MobileNetv4SCSAM, respectively. Also, the white values inside the images denote the IoU percentages.

One primary metric to quantify Grad-CAM validation is the Intersection over Union (IoU). It is used for extracting the overlap between two regions³⁸. We calculated the IoU of the Grad-CAM heatmap and RoI coverage to know how the model pays attention to the ground-truth areas of the images. The higher the IoU percentage, the more attention the model has paid to RoI. The obtained IoU values in Fig. 7 show that more parts of the RoI were influential in decision-making in attention-based models than in baseline models. Also, the models based on the SCSAM achieved the highest IoU values among other models. Therefore, comparing the results of the IoU proves that the SCSAM is more efficient than the PCSAM in serving the models’ decision-making about identifying the correct class of samples.

Discussion

The proposed attention modules in this study force the models to focus on the features associated with the structure and location of the RoI. Therefore, the network will achieve greater efficiency by improving its decision-making regarding the correct prediction of the class of the input images. A comprehensive comparison between the obtained classification results of the models indicates that the MobileNetv4SCSAM not only has a more efficient performance than other architectures but also imposes lower computational complexity. Also, comparing the classification results of MobileNetv4SCSAM and MobileNetv4PCSAM proves that SCSAM imposes less computational complexity and increases the accuracy of the models more than PCSAM. Therefore, we conclude that the effect of the structure of the proposed attention mechanism in improving the classification results in the suggested method is far greater than the number of parameters. The results of the visualization of feature maps and Grad-CAMs showed that the attention-based models not only extract feature maps related to the shape of the RoI but also focus more on its position. Such a positive impact is much greater in architectures based on SCSAM than PCSAM. Comparing subfigures (2) and (3) in Fig. 7 shows that the Grad-CAMs heatmap area for the background environment in subfigure (3) increases, while the heatmap area for the target object remains unchanged. But the Grad-CAM heatmap of the subfigures (5) covered a wider space of the target object than the subfigures (6). Such a contradiction doesn’t prove the complete ineffectiveness of the PCSAM. Comparing the obtained Grad-CAMs shows that SCSAM has a positive impact on the correct decision-making of all models, whereas PCSAM has such an effect only on models with a low number of parameters.

Conclusion

Two different kinds of channel-spatial attention modules, including the PCSAM and SCSAM, are introduced to enhance the performance of CNNs for image classification problems in this paper. Each of them has two distinctive sub-modules: the channel and spatial attention modules. The CAM extracts the feature maps related to the shape of the target objects, whereas the SAM focuses on their position. The suggested attention modules are embedded in the architecture of ResNet18 and MobileNetv4. The baseline and attention-based networks are trained and evaluated on the three standard and general image classification datasets with the same experimental conditions for 50 epochs. A comprehensive comparison of the results obtained by the models in the test step showed that SCSAM than PCSAM and previous similar modules, not only imposing less complexity on the networks but also further improving their efficiency. For future work, the proposed channel-spatial attention modules can be implemented in the architecture of other CNNs. Also, the suggested attention-based networks in this study can learn and evaluate on similar image classification datasets.

Author contributions

Data curation, software, and writing original draft: M.Z. Conceptualization, funding acquisition, investigation, and methodology: M.Z. and M.S.A. Formal analysis and visualization: M.Z. and H.S. Resources, supervision, and writing review and editing: M.S.A and H.S. All authors have read and agreed to the published version of the manuscript.

Funding

This work is based upon research funded by Iran National Science Foundation (INSF) under project No 4039017. The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data availability

The source code and datasets are available through the following link: https://github.com/mzolfagharimodares/ImageClassification-CNN-CSAM.git

Declarations

Competing interests

The authors declare no competing interests.