MGWO-CNN: hyperparameter optimization of CNN classifier for cervical cancer detection using Modified Grey Wolf Optimizer

Abstract

Every nation reports an increasing number of deaths among women due to cervical cancer each day, which has become a major global issue. Rapid diagnosis and treatment of cervical cancer can reduce mortality rates. Most studies on cervical cancer detection have made use of ensemble methods and Convolutional Neural Network (CNN) models. However, overfitting, parameter adjustment, and gradient vanishing issues affect most of these models. To address these issues, we propose a Modified Grey Wolf Optimizer-based Convolutional Neural Network (MGWO-CNN) model that incorporates the concepts of chaos theory and differential evolution mutation to detect cervical cancer. Traditionally, neural networks combined with backpropagation achieve poor convergence due to their dependence on initial values. Metaheuristic techniques offer a superior alternative to backpropagation. The proposed technique adjusts CNN hyperparameters to train the model architecture effectively. This optimized model extracts key features from cervical Pap smear images and predicts the outcomes. The results demonstrate that the MGWO-CNN model is a remarkably effective method to detect cervical cancer. We evaluated the efficiency of the model by comparing its performance on two datasets (Herlev and SIPaKMeD) using four performance measures: accuracy, sensitivity, specificity, and precision. The proposed approach outperforms existing methods in terms of accuracy, precision, sensitivity and specificity, achieving values of 99.45%, 100%, 97.96%, and 100%, respectively.

Introduction

The cervix is the bottom portion of the uterus that exists in the vagina, which is also known as the birth canal. The cells of the lower cervix of the uterus are those that form cervical cancer¹. In most cases, sexually transmitted infections caused by Human papillomavirus (HPV) are the root cause of cancer of the cervical cavity. Figure 1 shows the evolutionary progression of cervical cancer in HPV-infected patients². There has been a six-time increase in cervical cancer in women with positive Human Immunodeficiency Virus (HIV) compared to noninfected women³. Social and economic factors, lack of access to HPV vaccination, lack of cervical cancer screening, and lack of therapy services contribute to significant disparities in cervical cancer rates. When HPV infection is not treated in women, it can lead to cervical cancer in up to 95% of cases⁴.

Fig. 1.

Evolutionary progression of cervical cancer with HPV².

Periodic screenings and prompt detection of cervical cancer significantly reduce the chance of death^5,6. Screening for cervical cancer can be challenging because the early stages of the disease often have no symptoms. Routine annual investigation may reveal changes in the cervix. Screening for carcinogenic or malignant cells within the lining of the cervix is possible with the Pap smear test⁷. This method gently scrapes the cervix to collect a sample to detect expansion of cell abnormalities. Women aged 18–64 years should undergo regular screenings as cervical cancer grows slowly over time⁸. According to⁹, cervical cells undergo a series of alterations before the development of cervical cancer, and subsequently malignant cells begin to proliferate and metastasize. To diagnose cervical cancer promptly and provide the right therapies at the right time, it is important to take the test periodically¹⁰. However, cervical cancer screening requires a lot of manual labor and time, as the microscope slide contains too many cells to check all¹¹. People often receive the screening result within 2–6 weeks¹¹. As a substitute for manual diagnosis, computer-aided detection methods are gaining popularity due to their lower susceptibility to human errors^12,13. Some researchers^14–16 emphasize the need to develop intelligent tools for cervical cancer surveillance to accelerate the screening procedure and significantly reduce the burden of specialists in the field. Automated cervical cancer screening systems are based on cervical cellular classification, which is the process of predicting the kind or malignant status of cervical cells to perform their core function¹⁷. In recent years, multiple research articles have been published on the topic of early cervical cancer identification through deep learning approaches. Three significant review works can be found in^18–20.

Deep learning is an approach in which computers can learn from past experiences, which can help to predict current scenarios using complicated information related to cervical cancer detection^21,22. The deep learning approach alone is not remarkably effective in the classification of cervical cancer²³. Improving the classification accuracy of cervical cancer diagnosis is an immediate necessity, as current screening approaches have their limits²³. In this work, we examine how metaheuristic techniques could work together with deep learning approaches to provide better diagnostic tools for the detection of cervical cancer. In this work, we utilize the Modified Grey Wolf Optimizer (MGWO) and the Convolutional Neural Network (CNN) to optimize the detection of critical features associated with cervical cancer. The objective of the proposed approach is to improve the diagnostic performance of cervical cancer by detecting images of samples that may be unseen using traditional diagnostic methods. CNN is a specialized type of deep learning neural network designed to learn and identify patterns within images, such as edges, textures, and objects, by leveraging convolutional layers. The GWO algorithm is a nature-inspired optimization method that can be effectively used to optimize a diverse set of challenges. Nevertheless, the GWO algorithm is not flawless and may occasionally become trapped in local optima during finding the global optimal solution or the solution which is closer to global optimal. Thus, it is essential to combine the GWO algorithm with additional methods to enhance its effectiveness in classifying cervical cancer. In this work, we are integrating chaos theory and differential evolution mutation with the GWO algorithm. This hybrid optimization algorithm gains enhanced exploration capabilities and reduces the likelihood of getting stuck in local optima. In addition, it has the potential to improve the performance of cancer diagnosis, resulting in better treatment planning and better patient outcomes.

The article organizes its subsequent sections as follows. Section 2 discusses various models for predicting cervical cancer based on deep learning approaches. Section 3 describes the modified algorithm based on the grey wolf optimizer. Section 4 describes the dataset and the techniques to develop the MGWO-CNN classification model. Subsequently, Section 5 illustrates the experimental procedure and the performance measures used to assess the proposed model. The last Section 6 concludes the work with some avenues for future research.

Related work

This section summarizes recent significant studies that have been done to improve cervical cancer diagnostic accuracy.

• Tan et al.²⁴ utilizes the transfer learning and pre-trained CNN model to arrange Pap smear images into seven groups. The authors applied, evaluated, and compared thirteen pre-trained CNN models on the publicly accessible dataset. Of 13 models, the authors claim that the performance of DenseNet-201 is best, achieving 87.02% accuracy on the Herlev dataset.

• Chankong et al.²⁵ proposed an automatic cervical cancer cell segmentation and classification method. They have tested their artificial neural network method on the Herlev dataset, achieved an accuracy of 93.78%.

• Nair and Amala²⁶ proposed an optimized deep neural network model that efficiently classifies cervical cancer. The authors tested the performance of their proposed model on the Herlev dataset. According to Nair and Amala the optimized deep neural network model achieves a multiclass average accuracy of 96.49%.

• Bora et al.²⁷ used the ensemble method to improve classification accuracy. They have also tested their method on the Herlev dataset and achieved an accuracy of 96.51%.

• Wankhade et al.²⁸ proposed a collective voting stacking approach that achieves an accuracy rate of 98.30%.

• Abd-alhalem et al.²⁹ addressed the classification problem by combining bilinear pooling with CNNs. They have resolved the computational complexity issue of bilinear features by strengthening the bilinear CNN with a random projection paradigm, 98.8% accuracy is achieved by their method.

• Plissiti et al.³⁰ proposed the SIPaKMeD dataset and achieved 93.35% accuracy using deep CNN and SVM (Support Vector Machine).

• Win et al.³¹ used the bagging ensemble classifier for classification. They have tested their method on the SIPaKMeD dataset and achieved an accuracy of 94.09%.

• Mahajan and Kaur³² proposed cervical cancer classification in Pap smear images with enhanced segmentation and deep progressive learning-based techniques. They have tested their method on the SIPaKMeD dataset and achieved an accuracy of 97.4%.

• Mathivanan et al.³³ classified cervical cancer through a fusion of deep learning models. They have tested their method on the SIPaKMeD dataset and achieved an accuracy of 98.08%.

Many studies on Pap smear image analysis have shown several common limitations. A major concern about the Pap smear Herlev and SIPaKMeD datasets is that these datasets are small and unbalanced. This raises concern about the effectiveness of the models in the real world situation. Another concern is the lack of external validation. Most studies do not test their models on independent datasets. It makes it difficult to trust their robustness or clinical reliability. Some models use pre-trained networks like DenseNet-201 without adapting them to the Pap smear image domain. These models may not capture notable cytological features, as they are initially trained in natural images. The features that the model learned to find during the training processes of ImageNet (e.g., edges, textures, objects shapes) are not adapted to find all the details on the cells that include nucleus shape, cytoplasm texture, or chromatin patterns. Many studies focus only on accuracy as a performance metric. Although accuracy is useful, it does not reflect how well the model handles class imbalances or critical misclassifications. Clinical metrics such as sensitivity, specificity, F1 score, or ROC-AUC are rarely reported, which can hide important flaws. In addition, data augmentation and regularization strategies are not discussed in much detail, and they are crucial techniques to avoid overfitting, especially when working with small data sets. Lastly, there is a lack of proper handling of class imbalance. These limitations underline the need for models that are more robust, clinically applicable, and generalizable. Feature-level fusion and ensemble predictions increase model size, training time, and memory requirements. Often, only accuracy is reported, while sensitivity, specificity, ROC AUC, and class-wise metrics are frequently omitted, which may obscure poor performance in less frequent but clinically important classes.

In the above-mentioned studies, primarily deep learning approaches have been utilized to diagnose cervical cancer; however, there are still some challenges to overcome that we tried to solve in this research work.

MGWO algorithm

Here, we first discuss the GWO proposed by Mirjalili et al.³⁴. Next, we introduce modifications to the Mirjalili’s GWO method by integrating chaotic maps and differential evolution mutation.

GWO algorithm

Mirjalili et al.³⁴ proposed the GWO technique, a population-based metaheuristic optimization technique, in 2014, inspired by the behavior of grey wolves in nature, including their hunting strategies and leadership hierarchy. Grey wolves typically live in groups, with the number of wolves varying, but often ranging between 5 and 12 individuals. The pack is structured with four main types of wolves: alpha, beta, delta, and omega. The alpha wolf is the leader of the group, making all critical decisions and guiding the pack. The beta wolf, acting as a subordinate, advises the alpha when necessary. The delta wolf focuses on supporting the pack’s welfare and ensuring safety. Omega wolves hold the lowest rank in the hierarchy, helping maintain social balance and sometimes fulfilling roles such as ensuring the group’s cohesion.

Based on the social structure of the grey wolves, the optimizer designates the three fit solutions as alpha, beta, and delta. The remaining search agents then organize themselves and adapt their parameters based on the alpha, beta, and delta wolves. The omega wolves trail behind these three wolves. The mathematical model details of the GWO algorithm are as follows:

Encircling prey (optimal solution)

Let, represent the prey’s vector position and x represent the wolf’s vector position, we can mathematically depict the surroundings of the target using Eqs. (1) and (2)^34,35 where t is the current iteration, and and are the coefficient vectors

1

2

Using Eqs. (3) and (4)³⁴ and , respectively, are calculated, where and are random vectors with values between 0 and 1 inclusive, and the value of decreases linearly from 2 to 0 as the iteration increases. Between the two positions, a grey wolf agent can go to any position using the two random variables and . Agents in the grey wolf algorithm can move to near the optimal solution by modifying the values of and .

3

4

Reaching the optimal solution

Finding the best answer in nature, alpha, beta, and delta wolves lead the group as they seek food, and the rest of the pack follows. To replicate this hunting principle, we introduce agents with random values and assess their fitness. The three most precise outcomes are designated as alpha, beta and delta, given that the positions of the agents and the prey are unknown in the abstract search space. The remaining agents, such as the omega wolves, will adjust their places and orientations based on the three most superior wolf agents.

5

6

7

8

9

10

11

Using Eqs. (5)–(11)³⁴, search agents can modify their positions; nevertheless, their final positions are not known in advance. Instead, they are picked at random by the alpha, beta, and delta agents, places inside a circle whose radius is chosen by the three best-fit options³⁵.

Prey exploitation

As part of their hunting strategy, grey wolves encircle their victims until they freeze, this situation is called exploitation. In the mathematical model, when the agents come closer to the prey, the value of decreases, which further changes the value of . The variation of comes to a halt when its value shifts from to . As the iteration increases, the value of goes from 2 to 0. While the alpha, beta, and delta wolves hunt, the search agents can go to any spot between themselves and the prey³⁵.

Exploration for the prey

The grey wolf agents form a small group to find objects, and then they all come together to attack. This occurrence is controlled in mathematical models by the value of . In cases where is either more than 1 or less than , the grey wolf agents separate and seek out better prey. Nevertheless, the agents will converge towards the prey if the value of falls within the range of to +1³⁵. For more details on GWO, the interested reader can refer to^34,35.

Hybrid GWO, chaotic map and differential evolution mutation

The local optima are rapidly approached by the GWO algorithm. The risk of becoming caught in the local optima increases as the problem’s complexity or dimensions rise. The cause of this diminished performance is the limited exploration capacity of GWO³⁴. Therefore, enhancing the algorithm’s exploration capacity may yield an improved variant of the GWO algorithm. Nonetheless, augmenting the exploration capacity directly impacts the exploitation capacity of the GWO, potentially leading to an elevated inaccuracy in the derived solution. Therefore, to enhance the overall efficacy of GWO, it is essential to strengthen both its exploration and exploitation capabilities. Regarding this, we are integrating GWO, chaos theory³⁶ and differential evolution mutation³⁷.

First, we incorporate the chaotic theory concepts in GWO. This is achieved by applying logistic chaotic maps to adjust GWO coefficient vectors and using Eqs. (12) and (13). These equations inject deterministic yet unpredictable behavior into the search process, leading to potentially better optimization results. The logistic map is a regularly used chaotic map that is determined using Eq. (14)³⁶.

12

13

14

where is a chaotic variable at iteration t, , and r is the control parameter, which is set typically 4 for ensuring maximum chaos. Typically, is initialized by random value in the range 0.0001 and 0.9999. Also, we avoid taking the value of as 0.25, 0.5 and 0.75 because any of these values can produce a non-chaotic pattern when combined with r.

Note that as the iteration increases the value of decreases, therefore, we should also adapt the dynamic behavior of the chaos for a better impact. That is, there should be high chaos for high (use chaos heavily in the early stages for exploration), and moderate chaos for low (gradually reduce the influence of chaos as a approaches to 0 to focus more on exploitation). We can achieve this by scaling the chaotic influence with using Eq. (15).

15

We use the in Eqs. (12) and (13) to evaluate and with an increase in iteration.

Now, we incorporate the differential evolution mutation strategy to enhance the position update in GWO. A common DE mutation strategy is given by Eq. (16)³⁸.

16

where are the random vectors of the population, and is a factor that controls the differential variation ³⁷. We have set = 0.8 based on extensive experiments. To generate the mutant vector many mutation strategies have been reported in the literature³⁹. Here, we adopt the DE/best/1 strategy as shown by Eq. (17) to generate the mutant vector (wolf)³⁷.

17

where, is the best wolf who has best fitness, and and are the random wolves of the pack. The flowchart of MGWO is shown in Fig. 2.

Fig. 2.

Flowchart of MGWO.

The proposed MGWO-CNN model

Timely identification and categorization of cervical cancer is crucial, although it presents a challenging endeavor within the imaging system. In this work, the proposed MGWO-CNN model is utilized to classify pap smear images into two specific categories: benign and malignant. Figure 3 shows the proposed framework for effectively classifying cervical cancer images. This framework consists of collecting the Pap smear dataset, preprocessing, data augmentation, CNN hyperparameter optimization, feature extraction, classification, and performance evaluation. In this paper, first we use the Pap smear dataset in the form of images that contain a collection of cervical cells. Basically, the medical Pap test images have more noise, background content, imbalanced datasets, and low contrast. To address these issues, scale adjustment, noise removal, and image contrast enhancement techniques are utilized in the preprocessing phase. After preprocessing, to solve the class imbalance problem, data augmentation techniques were applied on the images. Afterward, the proposed MGWO algorithm is utilized to tune the hyperparameters to optimize the CNN. The optimized CNN then extracts features from the enhanced images and based on the learned features, categorizes the Pap test images into two different classes. Note that the training of CNN is carried out with the help of a proposed MGWO algorithm. Before discussing each phase in detail, we describe the datasets that are used in this work.

Fig. 3.

Proposed MGWO-CNN framework for cervical cancer image classification.

Smear image datasets

In this study, we used the following two publicly available datasets for testing the proposed MGWO-CNN model: Herlev and SIPaKMeD.

Herlev dataset

This dataset was generated at Herlev university hospital by collecting a vast collection of glass slides containing single Pap-smear cell images⁴⁰. Proficient cyto-technicians employ a microscope having a spatial resolution of 0.201 micrometers per pixel to capture digital photographs of individual cells. Subsequently, every single cell image is categorized into seven distinct cell types. Basically, Herlev dataset is categorized into two classes: normal (see Fig. 4a) and malignant (see Fig. 4b). The normal class consists of 242 images of three cell types (normal-columnar, normal-intermediate, normal-superficial) and the malignant class consists of 675 images of four cell types (light-dysplastic, moderate-dysplastic, severe-dysplastic, and carcinoma-in-situ).

Fig. 4.

Herlev dataset: (a) Normal class, (b) Malignant class.

SIPaKMeD dataset

This dataset contains 4049 Pap smear images of cervical cells³⁰. Basically, SIPaKMeD dataset (see Fig. 5 is also categorized into two classes: normal and malignant). Normal sets include 2411 images and fall into the parabasal, superficial intermediate, or metaplastic categories. The malignant sets include 1638 images and fall into the dyskeratotic and koilocytotic categories.

Fig. 5.

SIPaKMeD dataset.

Note that, in this work, we use 80% of the data for training purposes and 20% data for testing purposes.

Preprocessing phase

The Pap smear images that are captured from distinct sources may have the following inconsistencies: background noise, poor visibility of cellular structures (low contrast), overlapping cells, staining variability, and high dimensionality. To remove these inconsistencies, the preprocessing of images is essential. Through preprocessing, we get high-quality images that ensure accurate and reliable cervical cancer classification. In this study, we have applied the following techniques to get enhanced high-quality images: min-max normalization, Gaussian filtering, and piecewise contrast enhancement.

Augmentation

To fix the class imbalance problem in the Herlev and SIPaKMeD datasets, this study applies oversampling data augmentation technique only on the training set images of both classes to avoid the problem of data leakage in the test set. A horizontal flip and a random 0.2-inch-wide and 0.2-inch-high zoom are applied to the training sets of the two datasets. The rotation procedure can be executed by rotating the images in both clockwise and counterclockwise directions by 5 and 10 degrees. Table 1 displays the data samples of both datasets before and after applying the augmentation techniques on training set of images.

Table 1.

Details of training and test set images.

	SIPaKMeD			Herlev
Type	Total images (per class)	Train/augmented	Test	Total images (per class)	Train/augmented	Test
Normal	2411	1929/1929	482	242	194/540	48
Malignant	1638	1310/1929	328	675	540/540	135

¹SIPaKMeD dataset contains 4049 total images, split into Normal and Malignant categories.

²Herlev dataset contains 917 total images, categorized similarly

Convolutional neural network

This work utilizes the convolutional neural network (CNN) model, which is a combination of three convolutional layers, three pooling layers, two fully connected layers, and one flattened layer. After each convolution layer, there is a batch normalization layer that normalizes the output of the convolutional layer, and then the ReLU activation function provides the nonlinear relation to its input and is helpful for CNN to learn faster and perform better. Finally, a max-pooling layer down-sampled the data. To increase the performance of the model and prevent overfitting, we have given an initial dropout to the fully connected layer. The reason behind choosing the architecture with 3 convolutional layers and 3 pooling layers is that it offers a strong balance between feature extraction capability and computational efficiency. It is also deep enough to capture the hierarchical structure of Pap smear images without being prone to overfitting. The use of 2 fully connected layers allows robust classification while keeping the model interpretable and compact.

Here, the values of the pixels of a smear image are sent to the network as input, and the network produces the probability that the image is part of a given class. In this model, the smear images are fed layer by layer for feature extraction and classification purposes. The values of the previous layer’s output are used as input for each subsequent layer. To further facilitate prediction by fully connected layers, the flattened layer converts the spatially ordered feature maps into a one-dimensional vector. Lastly, the fully connected layer employs the sigmoid activation function to provide predictions based on the features learned in the convolutional and pooling layers⁴¹. The training configuration used to train the model is shown in Table 2.

Table 2.

Training protocol block.

Protocol	Value
Optimizer	Adam
Learning rate	0.001
Learning rate schedule	Adaptive: ReduceLROnPlateau (patience = 5, factor = 0.5)
Loss function	Binary cross-entropy
Number of epochs	100
Batch size	32
Regularization	Dropout rate = 0.5 in FC layers
Data augmentation	Yes: rotation, zoom, flip
Weight initialization	He Normal
Early stopping	Yes (patience = 10, monitor = validation loss)
Checkpointing	Best model saved based on lowest validation loss
Activation function	Sigmoid

Tuning of hyperparameters of CNN using MGWO algorithm

Optimizing hyperparameters is an essential stage in the learning process to enhance the model performance. The MGWO algorithm was implemented in the CNN architectural model to optimize various parameters such as dropout rate, learning rate, number of filters and kernel size. Subsequently, the hyperparameters are optimized according to the positions of the wolves that exhibit the highest fitness. Table 3 shows the initial search space ranges of CNN hyperparameters. Whereas the optimized hyperparameters are shown in Table 4. Ultimately, evaluate the efficacy of the proposed MGWO-CNN system on the test data set using the selected performance metric. Figure 6 shows the architecture of the hyperparameter-optimized CNN model.

Table 3.

CNN hyperparameters initial search space range.

Parameter	Range
Learning rate	[0.0001, 0.1]
Number of filters	[16, 256]
Kernel size	[3, 5]
Dropout rate	[0.0, 0.5]

Table 4.

Optimized CNN hyperparameters for different metaheuristic techniques.

Model	Filters	Kernel	Dropout	Learning rate
MVO-CNN	32, 64	(3 3)	0.3	0.001
GWO-CNN	32, 64	(3 3)	0.4	0.0005
MFO-CNN	64, 128	(5 5)	0.3	0.0007
WOA-CNN	32, 64	(3 3)	0.2	0.0008
MGWO-CNN	64, 128	(3 3)	0.3	0.0005

Fig. 6.

Architecture of hyperparameter optimized CNN using MGWO.

Experimental results and analysis

Experimental setup

The presented MGWO-CNN model has been built using Python’s TensorFlow library. A personal computer with a 3.4 GHz Intel(R) Core(TM) i7-6500U CPU, 16 GB of RAM, and an NVIDIA RTX 3060 GPU is used for training. The model is executed using an open-source platform known as Google Colab notebook.

To divide a dataset into training and test sets, this study split 20% of the images into the test set and the remaining 80% of the images into the training set. The next step is to train the model on the training set with a mini-batch size of 32, considering the dataset and the number of target classes (n = 2). The initial parameters for the MGWO algorithm are as follows: Population size = 10, number of dimensions = 4, maximum iteration = 30, and we used the sphere minimization function as a fitness function for this algorithm. For every training cycle, we calculated the loss between our model’s projected results, which are based on the probability distribution P generated by the fully connected layer, and the ground truth G, using the Eq. (18)⁴¹:

18

We utilized the MGWO as the optimizer in our CNN model, and this served as a roadmap for updating the weights. To evaluate the present model’s quality, we computed the accuracy of the updated model on the test set. We ran our model through 30 iterations of training, and the one that performed best on the test set was chosen for finalization. To test our methods, we employed a 10-fold cross-validation procedure. For each stage, we randomly divided the images into 10 folds; 8 of these folds were utilized for training the model, while 2 were utilized for testing. We repeated the entire procedure ten times. Records were made on the average performance, and we evaluated the efficacy of our CNN model using accuracy, sensitivity, specificity, and precision. Figure 7 shows the graph between accuracy versus epochs and loss versus epochs. The relationship between accuracy and epochs shows how well the model performs in identifying data from training and validation. When the accuracy continuously increases as the number of epochs increases, it shows a good sign of model generalizability. The relationship between loss and epochs measures the model’s prediction error by applying the loss function. It is a sign of effective training optimization and convergence when the loss value decreases as the number of epochs increases.

Fig. 7.

Performance graphs showing (a) Epoch vs. Accuracy and (b) Epoch vs. Loss.

Evaluation metrices and efficiency evaluation

This study tests two datasets using the suggested framework to see how well they perform with binary classifications and evaluates the efficacy of the suggested method by using commonly used metrics for classifying features: accuracy, sensitivity, specificity, precision, and F1-score. Based on the prediction values, a confusion matrix is generated for the Herlev dataset called dataset-1, and the SIPaKMeD dataset called dataset-2, shown in Figs. 8 and 9 respectively. Appraisal of the performance of the proposed model with both datasets is shown in Table 5. Figure 10 shows the ROC curve for both datasets. This metric illustrates the model’s performance in terms of the false positive rate (FPR) and the true positive rate (TPR) outcomes. A value close to one indicates that the model is robust and efficient.

Fig. 8.

Confusion matrix for Dataset-1.

Fig. 9.

Confusion matrix for Dataset-2.

Table 5.

Performance evaluation on Herlev and SIPaKMeD datasets.

Dataset	Train/test ratio	Accu. (%)	Prec. (%)	Sens. (%)	Spec. (%)	F1-score (%)
Herlev	80/20	99.45	100	97.96	100	98.97
	70/30	92.31	93.49	91.33	93.33	92.40
SIPaKMeD	80/20	99.38	99.59	99.38	99.39	99.48
	70/30	93.29	94.16	92.54	94.06	93.30

Fig. 10.

(a) ROC curve for Dataset-1 and (b) ROC curve for Dataset-2.

The exceptionally high values of accuracy, precision, and specificity that the proposed MGWO-CNN model attained might make one worried about the possibility of overfitting. Overfitting typically occurs when models learn not only the underlying patterns but also the noise and specific characteristics of the training data. This results in poor performance with unseen data. This is a legitimate concern, especially when working with relatively small and class-imbalanced datasets, which do not fully reflect the complexity and variability of real-world clinical data.

To address overfitting, several strategies were implemented in the proposed work. First, data augmentation techniques: rotation, flipping, zooming, etc. were applied on the dataset to increase the variability of training set. Second, CNN architecture was regularized with dropout and batch normalization layers to avoid over-dependence of the model on particular features. Third, the Modified Grey Wolf Optimization (MGWO) was employed for hyperparameter tuning, which enhances model performance by avoiding local optima and reducing the risk of overfitting to non-discriminative features. In the last, evaluate model robustness across different splits of the dataset.

Comparative evaluation

Performance enhancement of the proposed methodology is assessed by employing various existing metaheuristic methodologies on CNN: Mean Variance Optimization (MVO)⁴², Grey Wolf Optimizer (GWO)³⁴, Moth Flame Optimization (MFO)⁴³ and Whale Optimization Algorithm (WOA)⁴⁴.

The following part provides an account of the comparative assessment conducted by the developed model on two distinct datasets, denoted as dataset-1 and dataset-2.

Dataset-1 evaluation

Figure 11 displays the evaluation of the proposed method using dataset-1. Figure 11a presents the accuracy assessment of the training dataset. The accuracy of MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN is 0.93, 0.94, 0.94, and 0.92 respectively for 80% of the data. In comparison, the proposed approach produced an excellent accuracy of 0.9945, indicating higher accuracy as compared to MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN. Figure 11b illustrates the evaluation of the precision parameter by altering the data used for training. The training data was divided into 80%, and the precision values for MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN were found to be 0.94, 0.95, 0.94, and 0.93 respectively. However, the proposed approach attained a precision of 1.00, indicating a significant performance improvement as compared to other models.

Fig. 11.

Evaluation of the proposed method using Dataset-1 with four optimization techniques—MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN. The figure compares their performance in terms of (a) accuracy, (b) precision, (c) sensitivity, and (d) specificity, highlighting the strengths and limitations of each optimizer in improving CNN model performance.

Figure 11c shows the results of the sensitivity evaluation. The sensitivity of the proposed method is 0.9796 for 80% of the data used for training, which is greater than the current approaches; as a result, the proposed approach achieves a higher percentage of improvement when compared to MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN. The study of the developed model’s specificity is shown in Fig. 11d. For 80% of the data, the proposed model obtained a specificity of 1.00, while the others MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN compute specificities of 0.94, 0.95, 0.94, and 0.93, respectively. By extracting critical properties from the cells, the proposed technique achieved effective performance in cervical cancer detection.

Dataset-2 evaluation

Figure 12 displays the evaluation of the proposed method using dataset-2. Figure 12a presents the accuracy assessment of the training dataset. The accuracy of MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN is 0.92, 0.93, 0.93, and 0.92 respectively for 80% of the data. In comparison, the proposed approach produces an excellent accuracy of 0.9938, indicating higher accuracy as compared to MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN. Figure 12b illustrates the evaluation of the precision parameter through the manipulation of the training data. The precision values for MVO-CNN, GWO-CNN, MFO-CNN and WOA-CNN were found to be 0.92, 0.93, 0.92, and 0.94 respectively. However, the proposed approach attained a precision of 0.9959, indicating a significant performance improvement compared to other models.

Fig. 12.

Evaluation of the proposed method using Dataset-2 with four optimization techniques—MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN. The figure compares their performance in terms of (a) accuracy, (b) precision, (c) sensitivity, and (d) specificity, highlighting the strengths and limitations of each optimizer in improving CNN model performance.

Figure 12c shows the results of the sensitivity evaluation. While the suggested method achieved a sensitivity of 0.9938 with 80% training data, MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN had sensitivity values of 0.92, 0.93, 0.94, and 0.93, respectively. The study of the developed model’s specificity is shown in Fig. 12d. For 80% of the data, the proposed model obtained a specificity of 0.9939, while the others, MVO-CNN, GWO-CNN, MFO-CNN, and WOA-CNN, compute specificities of 0.92, 0.94, 0.92, and 0.93, respectively. By extracting critical properties from the cells, the proposed technique achieved effective performance in cervical cancer detection. Figure 12d shows how changing the training data affects the specificity metric. With an MVO-CNN of 6%, GWO-CNN of 5%, MFO-CNN of 6%, andWOA-CNN of 5%, the performance improvement reported by the proposed approach at 80% data is 0.9939 specificity. This research accomplished remarkable results in the detection of cervical cancer by improving the classifier with the optimization method that was designed.

When comparing MGWO-CNN’s performance to that of other published literature, the findings are represented in Table 6. The findings revealed a robust and exact model for classifying cervical cancer, as well as important insights into the analytical approach used.

Table 6.

Comparison of the proposed approach with existing methods.

Dataset	References	Results
Herlev	Tan et al.24	Accuracy: 87.02%
	Chankong et al.25	Accuracy: 93.78%
	Nair and Amala26	Accuracy: 96.41%
	Bora et al.27	Accuracy: 96.51%
	Wankhade et al.28	Accuracy: 98.30%
	Abd-Alhalem et al.29	Accuracy: 98.80%
	Proposed approach	Accuracy: 99.45%
		Sensitivity: 97.96%
		Specificity: 100%
		Precision: 100%
SIPaKMeD	Plissiti et al.30	Accuracy: 93.35%
	Win et al.31	Accuracy: 94.00%
	Mahajan and Kaur32	Accuracy: 97.40%
	Mathivanan et al.33	Accuracy: 98.08%
	Proposed approach	Accuracy: 99.38%
		Sensitivity: 99.38%
		Specificity: 99.39%
		Precision: 99.59%

Conclusion

The purpose of this research is to develop a framework for accurately and efficiently classifying cervical cytology images into two classes by utilizing the proposed MGWO-CNN model. To achieve the objective of this work, two publicly available datasets, namely Herlev and SIPaKMeD datasets, are used. After acquiring the images from datasets, some preprocessing techniques were applied to these images to enhance the quality of the images. Then, the enhanced images are passed to CNN for the feature extraction process. During the training of CNN, hyperparameter adjustment is essential for better results. In this work, the proposed MGWO algorithm is used to fine-tune CNN, which in turn accomplishes the cancer classification process based on the feature vector. The proposed MGWO algorithm combines the chaotic map and differential evaluation mutation with Mirjalili’s GWO to enhance the exploration and exploitation capabilities of the search space. The experimental results show that combining the CNN with the proposed technique produces better outcomes than existing methods. The accuracy, precision, sensitivity, and specificity values obtained by the suggested approach utilizing the cervical cytology image dataset are 0.9945, 1.00, 0.9796, and 1.00, respectively. Based on these findings, this hybrid approach holds great promise for improving cervical cancer diagnosis. Although the MGWO-CNN model was trained in and tuned on microscopic images of cervical cancer (e.g., from Herlev or SIPaKMeD datasets), simply utilizing it to classify microscopic cells of another cancer swab (e.g., oral, breast, or prostate cytology) might not provide robust results. This is because there exists considerable domain specific differences in cellular morphology, staining properties, background noise, and feature distributions between different types of cancer tissues. In addition to this, the hyperparameters were optimized using the MGWO algorithm for the classification of the cervical cancer dataset. This potentially causes domain overfitting and decreases generalization to different cancer types. Future research will investigate extending this model using transfer learning or domain adaptation methods to allow cross-cancer generalizability while maintaining diagnostic accuracy.

Author contributions

Conceptualization, S.J. and A.J.; methodology, S.J.; development, S.J. and A.J.; validation, M.J.; formal analysis, S.J., A.J. and M.J.; writing–original draft preparation, S.J.; writing–draft revision, A.J. and M.J.; visualization, S.J. AND M.J.; supervision, A.J. and M.J.;

Funding

Open access funding provided by Manipal University Jaipur. Open access funding will be provided by Manipal University Jaipur.

Data availability

The datasets (Herlev and SIPaKMeD) analysed during the current study are available in two different repositories that can be accessed through the following links: Herlev dataset - https://mde-lab.aegean.gr/index.php/downloads/, SIPaKMeD dataset -https://www.cs.uoi.gr/ marina/sipakmed.html

Declarations

Competing interests

The authors declare no competing interests.