Hybrid Transfer Learning for Classification of Uterine Cervix Images for Cervical Cancer Screening

Abstract

Transfer learning using deep pre-trained convolutional neural networks is increasingly used to solve a large number of problems in the medical field. In spite of being trained using images with entirely different domain, these networks are flexible to adapt to solve a problem in a different domain too. Transfer learning involves fine-tuning a pre-trained network with optimal values of hyperparameters such as learning rate, batch size, and number of training epochs. The process of training the network identifies the relevant features for solving a specific problem. Adapting the pre-trained network to solve a different problem requires fine-tuning until relevant features are obtained. This is facilitated through the use of large number of filters present in the convolutional layers of pre-trained network. A very few features out of these features are useful for solving the problem in a different domain, while others are irrelevant, use of which may only reduce the efficacy of the network. However, by minimizing the number of filters required to solve the problem, the efficiency of the training the network can be improved. In this study, we consider identification of relevant filters using the pre-trained networks namely AlexNet and VGG-16 net to detect cervical cancer from cervix images. This paper presents a novel hybrid transfer learning technique, in which a CNN is built and trained from scratch, with initial weights of only those filters which were identified as relevant using AlexNet and VGG-16 net. This study used 2198 cervix images with 1090 belonging to negative class and 1108 to positive class. Our experiment using hybrid transfer learning achieved an accuracy of 91.46%.

Introduction

Cervical cancer is one of the major reasons of illness and mortality among women worldwide [1]. It contributes to 12% of all cancers and is the second most common cause of death among women worldwide [1]. According to WHO report [2], almost 80% of deaths due to cervical cancer occur in developing countries. Pap smear is the most commonly used method for screening of cervical cancer. But it requires efficient networking between smear collection and cytology laboratories [3]. Also, it suffers from low sensitivity of 52% [4]. In addition to being an expensive test, results of this test are available approximately after 2 weeks. Hence, this approach cannot facilitate the screening and treatment in a single visit. VIA is demonstrated as a simple, cost-effective test that facilitates screen and treat approach [5, 6]. Application of 3–5% acetic acid turns the precancerous lesions white, called acetowhite (AW) regions. Typical cervix images prior to and after the application of acetic acid are shown in Fig. 1. Precancerous lesions, which have turned white, are marked as AW in Fig. 1b.

Fig. 1.

Typical cervix images. a Prior to the application of acetic acid. b After the application of acetic acid

These lesions can be destroyed instantly using cryotherapy [7]. This screen and treat approach can be performed in a single visit to the hospitals, which suppresses further development of cervical cancer.

Considerable training is required to discriminate between actual cancer lesions and lesions that are benign. The most important deciding features for VIA-positive lesions are intensity of acetowhitening, margin of AW lesions, and the rate at which AW lesions appear and disappear. Identification of these features requires considerable amount of skill. Hence, accuracy of this test depends on the skill level of the person who performs the test [8]. Blumenthal et al. [9] reported that health workers need to be trained well for successful implementation of VIA. This problem can be considered as a challenge in computer vision. Images of cervix during VIA examination can be acquired and analyzed using image processing algorithms. Deploying these algorithms in a cell phone or a comparable device with camera can be used as decision support system for the health workers during cervical cancer screening.

Related Work

State-of-the-art algorithms developed by various researchers used traditional machine learning method, which are based on the selection of hand-crafted features and classifying the images based on these features using a suitable classifier. These researchers tried to classify the images by extracting the features related to intensity of acetowhitening, margin of AW lesions, and the texture with in AW lesions. A few researchers [10–14] considered acetowhite feature for the classification of cervix images into VIA-positive and VIA-negative classes. Another group [15, 16] used texture within the aceotowhite lesion as feature. Claude et al. [17] and Raad et al. [18] used lesion margin feature. Identification of relevant features for detection of cervical cancer is a complex task. This problem can be overcome by using deep learning.

Deep learning is a rapidly growing field in machine learning for the analysis of images. It can be used for wide variety of image analysis tasks such as image classification, image segmentation, object detection, and image registration. Deep learning can be used for an image analysis task in two ways as shown in Fig. 2.

Fig. 2.

Deep learning approaches

CNN is a deep learning algorithm used extensively to analyze images. They are mainly composed of different layers, namely convolutional layer, activation layer, pooling layer, and fully connected layer. CNNs can be employed for medical image classification in two ways. The first approach is to train the CNN from scratch which is called full training. Depending on the number of layers in the architectures, CNNs can be classified as deep layer CNN and shallow layer CNN. Full training of CNNs with deep architecture requires large amount of annotated data which are usually not available in the medical domain. Hence, the most popular deep CNNs [19–24] were trained with images of non-medical domain. Deep CNNs are capable of obtaining full representation of the training data. But requirement of large number of annotated data hinders its applicability in the analysis of images in medical domain. Hence, full training using shallow layered CNNs [25–29] was considered for image analysis in medical domain. To exploit the advantages of deep architecture of CNN on the limited dataset, another approach known as transfer learning was utilized. This approach facilitates the use of the knowledge acquired by pre-trained CNNs which were trained using large number of non-medical data. Transfer learning can be performed in two ways. First approach extracts the features from different layers after retraining them with small dataset [30–33]. The features extracted are classified using different classifiers. Second approach of transfer learning involves fine-tuning various layers of pre-trained CNNs to perform a different task [34–37]. It was demonstrated by these researchers that transfer learning is the best choice for the analysis of medical images, where large amount of annotated data is not available. A large number of filters present in the convolutional layers of pre-trained network facilitates extraction of wide variety of features. A very few features out of these features are useful for solving the problem in a different domain, while others are irrelevant, use of which reduces the efficacy of the network. However, by minimizing the number of filters required to solve the problem, the efficiency of the training the network can be improved. In this study, we consider identification of relevant filters using the pre-trained networks, namely AlexNet and VGG-16 net to detect cervical cancer from cervix images. This paper presents a novel hybrid transfer learning technique, in which a CNN is built and trained from scratch, with initial weights of only those filters which were identified as relevant using AlexNet and VGG-16 net.

Method

Data Collection and Ground Truth

The cervix images were collected from the Kasturba Medical College, Manipal, India. Ground truth for the study was provided by a Gynaecologic Oncology expert. A total of 231 images were collected of which 31 images were VIA positive. We also collected 1413 cervix images from the National Cancer Institute (NCI) and the National Institute of Health (NIH) archive. This consisted of 890 VIA-negative and 523 VIA-positive images. Combining both the datasets, we had 1644 images in total with 1090 VIA negative and 554 VIA positive. To increase the size of the dataset in VIA-positive class, we used data augmentation. We rotated the VIA-positive images by an angle of 90°. With data augmentation, we have 1108 VIA-positive images in this dataset consisting of 2198 images.

Hybrid Transfer Learning

We designed two shallow layer CNN architectures from scratch. We name them as CNN-1 and CNN-2 in this manuscript.

CNN-1

Architecture of CNN-1 is listed in Table 1. Number of filters in the convolutional layers of the CNN-1 was selected based on how many filters in the pre-trained networks highlight the specific features required.

Table 1.

Architecture of the CNN-1

Layer type	Input size	Filter size	Number of filters	Stride	Pad	Output size
Data input	227X227X3	NA	NA	NA	NA	227X227X3
Convolution 1	227X227X3	5X5X3	23	4	0	57X57X23
ReLU 1	57X57X23	NA	NA	NA	NA	57X57X23
Max pool 1	57X57X23	3X3	1	2	0	28X28X23
Convolution 2	28X28X23	5X5X23	28	1	2	28X28X28
ReLU 2	28X28X28	NA	NA	NA	NA	28X28X28
Max pool 2	28X28X28	3X3	1	2	0	13X13X28
Fully connected 1	13X13X28	NA	NA	NA	NA	1024X1
Fully connected 2	1024X1	NA	NA	NA	NA	512X1
Fully connected 3	512X1	NA	NA	NA	NA	2X1

NA not applicable

Two pre-trained networks AlexNet and VGG-16 net were considered for visualizing the responses of different filters in convolutional layers. The main feature used for identification of cervical cancer is acetowhite feature. Further, margin of acetowhite region and texture within the acetowhite region are used to distinguish between various kinds of possible acetowhite regions. Filter responses were visualized for a few sample cervix images including VIA-positive and VIA-negative category. Responses of filter in the first and second convolutional layers of AlexNet and VGG-16 net were visualized. Responses of a few filters in the first convolutional layer of AlexNet for VIA-positive and VIA-negative images is shown in Fig. 3.

Fig. 3.

Responses of a few filters in the first convolutional layer of AlexNet for VIA-positive and VIA-negative images

Out of the 96 filters in the first convolutional layer of AlexNet, filters 10, 23, 32, 39, 40, 41, 77, 84, and 95 highlighted the whiteness features distinctly. Similarly, responses of a few filters in the first convolutional layer of VGG-16 net are shown in Fig. 4.

Fig. 4.

Responses of a few filters in the first convolutional layer of VGG-16 net for VIA-positive and VIA-negative images

Out of the 64 filters in the first convolutional layer of VGG-16 net, filters 2, 3, 9, 10, 13, 14, 16, 17, 20, 23, 24, 26, 30, and 53 distinctly highlighted the acetowhite feature in the image. We identified 23 filters which distinctly highlighted the acetowhite information in the image. These 23 filters were adapted into first convolutional layer of the proposed CNN-1. Size of filters in the first convolutional layer of AlexNet and VGG-16 net is 11X11X3 and 3X3X3 respectively. We resized the size of selected filters to 5X5X3, which is the size of the filters in the first convolutional layer of the proposed CNN. We also extracted the filter bias weights of the visualized filters and used them as the initial filter bias weights of the proposed CNN-1.

In the similar manner, we visualized responses of the filter in the second convolutional layer of AlexNet and VGG-16 net. Filters 15, 30, 48, 69, 112, 120, 140, 184, 232, and 248 of AlexNet and filters 2, 3, 9, 10, 11, 13, 16, 17, 19, 20, 23, 25, 26, 30, 42, 53, 55, and 60 of VGG-16 net distinctly highlighted the acetowhite features. Hence, 28 filters from second convolutional layers of pre-trained networks highlighted acetowhite feature. Thus, the second convolutional layer of the proposed CNN-1 has 28 filters. Filters were resized to 5X5X23. Since depth of the filters in the second convolutional layer is required to be 23, we randomly selected 23 slices of 64 slices from VGG-16 net. Similarly, from AlexNet, we randomly picked up 23 slices out of 48 slices. Bias weights of relevant filters in the second convolutional layer of pre-trained networks were used as the initial bias weights of the filters in the second convolutional layer of the proposed CNN-1. Two max pooling layers were used to reduce the dimensionality of the features. Finally, three fully connected layers were added whose weights were initialized randomly.

The CNN-1 was trained using cervix images. Dataset was divided as 85% for training, 7.5% for validation, and 7.5% for testing. Two hyperparameters, namely learning rate (LR) and batch size (BS), were varied and performance of CNN was observed. The parameter which provided the best performance was selected. The CNN was trained for 100 epochs.

We compared the performance of the proposed hybrid CNN-1 with the performance of proposed CNN-1 architecture by initializing the filter weights in the convolutional layers to random values.

CNN-2

Architecture of CNN-2 is shown in Fig. 5. As in the CNN-1 architecture, first convolutional layer (conv1 of Fig. 5) contains 23 filters weights of which were adapted from identified filters from AlexNet and VGG-16 net which highlighted acetowhite features. In the second convolutional layer of CNN-1, depth of the adapted filters was scaled to 23 by random selection of 23 slices. But in the second convolutional layer of CNN-2 (conv2a and conv2b of Fig. 5), the entire set of depths (48 in AlexNet and 64 inVGG-16 net) of adapted filters from the second convolutional layers of pre-trained networks was used. This is facilitated through the use of 1X1 convolutional layers (conv01 and conv02 of Fig. 5). Outputs of conv2a and conv2b layers are concatenated along the depth. The rest of the layers in CNN-2 after concatenation layer is the same as that of layers after conv2 layer of CNN-1. CNN-2 is trained in a similar manner as that of CNN-1. Initial learning rate was set to 0.001 and dropped by a factor of 0.1 in every 50 epochs to avoid overfitting.

Fig. 5.

Architecture of CNN-2

We compared the performance of the proposed hybrid transfer learning approach with traditional transfer learning approach. We fine-tuned pre-trained AlexNet and VGG-16 net using cervix images after replacing the final fully connected layers with a new fully connected layer with two neurons to suit the proposed two class problem.

Results

All the experiments were performed using system with Tesla K20Xm graphics card of 6GB memory. MATLAB R2019a was used to implement the network.

Performance of CNN-1 for different learning rates and batch sizes is shown in Fig. 6.

Fig. 6.

Performance of CNN-1 for different learning rates and batch sizes

It can be observed from Fig. 6 that for learning rate of 0.001 and batch size of 50, CNN-1 provides the best accuracy of 91.46%. Hence, we selected the learning rate as 0.001 and batch size as 50.

Accuracy, sensitivity, specificity, and AUC values for training, testing, and validation steps of CNN-1 for LR = 0.001 and batch size = 50 are listed in Table 2.

Table 2.

Performance of CNN-1 for LR = 0.001 and BS = 50

	Accuracy (%)	Sensitivity (%)	Specificity (%)	AUC
Training	97.16	95.75	98.6	0.97
Testing	91.46	89.16	93.83	0.92
Validation	84.85	77.11	92.68	0.85

A plot of training accuracy and validation accuracy for LR = 0.001 and BS = 50 is shown in Fig. 7.

Fig. 7.

Plot of training accuracy and validation accuracy for LR = 0.001 and BS = 50 for CNN-1

A plot of training loss and validation loss for CNN-1 for LR = 0.001 and BS = 50 is shown in Fig. 8.

Fig. 8.

Plot of training loss and validation loss for CNN-1 for LR = 0.001 and BS = 50

From Figs. 7 and 8, it can be observed that training and validation accuracy and loss curves are close to each other which indicate that the proposed CNN-1 is trained properly.

We also observed the performance of CNN-1 by initializing the weights of the filters in convolutional layers to random values. Figure 9 depicts the performance of CNN-1 by initializing the filter weights to random values.

Fig. 9.

Performance of CNN-1 by initializing the filter weights to random values

From Fig. 9, it can be observed that the best accuracy of 84.15% was achieved for LR = 0.004 and BS = 250.

A plot of training and validation accuracy for LR = 0.004 and BS = 250 is shown in Fig. 10.

Fig. 10.

A plot of training and validation accuracy for LR = 0.004 and BS = 250 for CNN-1 when filter weights are initialized with random values

A plot of training and validation loss for CNN-1 when filter weights are initialized with random values is shown in Fig. 11.

Fig. 11.

Plot of training and validation loss for CNN-1 when filter weights are initialized with random values

By initializing the filter weights to random values, we obtained an accuracy of 84.15%, sensitivity of 83.13%, specificity of 85.19%, and AUC of 0.84.

To understand if the initialized filters from AlexNet and VGG-16 have improved after training, we visualized the filter outputs of CNN-1 and compared with the corresponding filter responses of AlexNet and VGG-16 net which were identified as relevant. Reponses of filters 10, 23, 32, 39, 40, 41, 77, 84, and 95 in the first convolutional layer of AlexNet for VIA-positive image (original image) depicted in Fig. 3 before training is shown in Fig. 12.

Fig. 12.

Responses of filters 10, 23, 32, 39, 40, 41, 77, 84, and 95 in the first convolutional layer of AlexNet before training

Responses of corresponding filters in CNN-1 after training are shown in Fig. 13.

Fig. 13.

Responses of corresponding filters in CNN-1 after training

It can be observed from Figs. 12 and 13 that filters 10, 23, and 32 do not highlight the acetowhite region distinctly after training, while in other filters, the acetowhite regions are more distinct after training when compared to their responses before training.

Reponses of filters 2, 3, 9, 10. 13, 14. 16, 17, 20, 23, 24, 26, 30, and 53 in the first convolutional layer of VGG-16 net for VIA-positive image before training is shown in Fig. 14.

Fig. 14.

Reponses of filters 2, 3, 9, 10. 13, 14, 16, 17, 20, 23, 24, 26, 30, and 53 in the first convolutional layer of VGG-16 net before training

Responses of corresponding filters in CNN-1 after training are shown in Fig. 15.

Fig. 15.

Responses of corresponding filters of VGG-16 net after training

From Figs. 14 and 15, it can be observed that filters 3, 16, 20, 30, and 53 do not highlight the acetowhite region distinctly after training, while in other filters, the acetowhite regions are more distinct or remained same after training when compared to their responses before training. In total, 5 filters out of 23 filters did not highlight the acetowhite feature after training.

Performance of CNN-2 for different batch sizes is shown in Fig. 16.

Fig. 16.

Performance of CNN-2 for different batch sizes

At batch size of 150 CNN performance was better in terms of accuracy and sensitivity. Accuracy, sensitivity, specificity, and AUC values for training, testing, and validation steps of CNN-2 for batch size = 150 are listed in Table 3.

Table 3.

Performance of CNN-2 for BS = 150

	Accuracy (%)	Sensitivity (%)	Specificity (%)	AUC
Training	86.41	79.72	93.20	0.86
Testing	85.37	86.75	83.95	0.85
Validation	78.18	75.90	80.49	0.78

A plot of training and validation accuracy and loss for BS = 150 for CNN-2 is shown in Fig. 17.

Fig. 17.

Plot of training and validation accuracy and loss for BS = 150 for CNN-2

Performance of traditional transfer learning approach using fine-tuning pre-trained AlexNet, VGG-16 net, and performance of proposed CNNs is listed in Table 4.

Table 4.

Performance of fine-tuning AlexNet and VGG-16 net

	Accuracy (%)	Sensitivity (%)	Specificity (%)	Training time (s)	Learnable parameters (millions)
AlexNet	84.31	93.50	75.00	5079.32	61
VGG-16	84.15	83.13	85.18	62,479.52	138
CNN-1	91.46	89.16	93.83	1506.08	5.4
CNN-2	85.37	86.75	83.95	661.84	5.4

From Table 4, it can be observed that proposed CNN-1 and CNN-2 perform equally well when compared to the performance of traditional transfer learning with lesser training time and learnable parameters.

Discussions

One of the principal challenges in deep learning is overfitting, as an overfit model is not generalizable to unseen data. Overfitting of a CNN on training data is usually identified by monitoring the loss and accuracy on the training and validation datasets. In the case if the model performs well only on the training set compared to the validation set, then the model has likely been overfit to the training data. As per statistical learning theory [38], large number of learnable parameters in a predictor make the model hard to train in the case of limited data samples. As per Zhang et al. [39], with 2n + d number of parameters, a simple neural network with two-layers is capable of perfectly fitting any dataset of n samples of dimension d. However, the pre-trained networks used in transfer learning have much more than 2n + d parameters which may lead to overfitting. Also, it requires more computational resources and training time as shown in Table 4. There have been a few techniques proposed to limit overfitting, for example, regularization with dropout, batch normalization, data augmentation, and early stopping. In spite of these efforts, there is still a concern of overfitting. The best solution for reducing overfitting is to obtain more training data. A model trained on a larger dataset typically generalizes better, though that is not always attainable in medical imaging. Wu et al. [40] in their experimentation of classifying Tiny ImageNet data using deep and shallow pre-trained CNNs concluded that benefits of going deeper in the case of small data is limited. Hence, solution is reducing architectural complexity. This is achieved in the proposed hybrid transfer learning approach. Along with having simple architecture, since weights of filters in the convolutional layers have been adapted from learnt filters, this provides comparable performance to the state-of-the-art transfer learning method with lesser training time, learning parameters, and memory requirements. This type of transfer learning can be used for classification when number of available data are small as in healthcare. Drawback of this type of hybrid transfer learning is that it can be performed only on color images as the ImageNet dataset consists of color images having red, green, and blue channels. It cannot be applied to healthcare aspects which has gray scale images as in the case of radiology. A comparison of performance of proposed method with that of state-of-the-art traditional methods for classification of cervix images as listed in Table 5 reveals that proposed method performs equally well and in some cases even better than state-of-the-art methods in spite of larger data compared to the number of data used by state-of-the-art methods.

Table 5.

Comparison of performance of proposed method with that of state-of-the-art traditional methods

Authors	Features	Classifier	Number of images	Performance
Park et al. [10]	RGB intensity values, ratios, gray scale values	KNN, SVM	29	Sensitivity, 79% Specificity, 88%
Li et al. [11]	G component of RGB color space, contrast feature	K-means	99	Sensitivity, 94% Specificity,87%
Kim et al. [12]	Pyramid color histogram in CIE Lab space (PLAB), pyramid histogram of oriented gradients (PHOG)	SVM	2000	Sensitivity, 73% Specificity, 77%
Xu et al. [13]	PHOG, PLAB, and pyramid histogram of local binary pattern (PLBP)	AdaBoost	345	Sensitivity, 86.4% Specificity,74.2% Accuracy, 80.3%
Kudva et al. [14]	Color, SGLDM, NGTDM NGLDM, wavelet, LBP	SVM	102	Sensitivity, 99.05% Specificity, 97.16% Accuracy, 97.94%
Ji et al. [15]	24 texture features such as joint entropy, angle entropy, energy, contrast, etc.	Minimum-distance classifier	300	Accuracy, 87.03%
Claude et al. [17]	Contour features	Multilayer perceptron	30	Accuracy, 95.8%
Proposed method	Hybrid transfer learning	CNN	2198	Sensitivity, 91.46% Specificity, 89.16% Accuracy, 93.83%

Considering only acetowhite feature of images in the filters, we were able to achieve a classification accuracy of 91.46%. Performance of the network may be improved by having more filters in the convolutional layers which highlight the margin of acetowhite region and texture within the acetowhite region.

Conclusion

A novel hybrid transfer learning approach for cervix image classification was reported in this paper. Two shallow layer CNNs were designed and initial weights of the filters were adapted from pre-trained AlexNet and VGG-16 net which highlight the acetowhite features. These networks were trained to classify cervix images. It provided a classification accuracy of 91.46%. Performance of the network may be improved by having more filters in the convolutional layers which highlight the margin of acetowhite regions and texture within the acetowhite region.

Compliance with Ethical Standards

Disclaimer

The contents are the responsibility of Manipal Academy of Higher Education and do not necessarily reflect the views of Massachusetts General Hospital, USAID or the United States Government.