Binary Classification of Laryngeal Images Utilising ResNet-50 CNN Architecture

Abstract

In India, laryngeal cancer is a significant health concern, underlining the critical need for early detection methods. This study introduces a novel approach to classify laryngeal lesions into nine morphological categories; due to data scarcity for all the nine classes, the data is divided into cancer and non-cancer classes, including both non-cancerous and Squamous Cell Carcinoma (SCC), by analysing endoscopy images with advanced convolutional neural networks, deep learning, and image processing techniques. A dataset of 1978 endoscopy images from 960 patients at a tertiary care center in Lucknow, between May 2015 and December 2023, was utilised for this purpose. These images, captured using an Olympus CV-170 processor and annotated via the CVAT tool, were processed to highlight Regions of Interest (ROI) for detailed examination. The dataset was split, with 90% for training/validation and 10% for testing. A total of 197 images out of 1978 were selected for testing, which included 43 cancerous and 154 non-cancerous images. For the feature extraction, ResNet50 was utilised. The model’s evaluation through the Receiver Operating Characteristic (ROC) curve demonstrated high effectiveness, with areas of 0.95, 0.98, and 0.93 for combined, NBI-only, and WL-only datasets, respectively. The accuracy rates were notably high across all datasets, highlighting the potential of this model to significantly aid in the early detection and classification of laryngeal cancer. In India, where the incidence of head and neck cancer is high and there is a lack of both advanced instruments and expertise in Narrow Band Imaging (NBI), this model could be instrumental in the early detection of laryngopharyngeal cancer. Level of Evidence: 2 C.

Introduction

Laryngeal cancer (LCA) ranks among the top ten cancer types in Indian men, accounting for 3–6% of all male cancer cases in the country [1]. The incidence of laryngeal cancer varies across India, with rates ranging from 1.26 to 8.18 per 100,000 population in different regions. Notably, the 5-year survival rate for laryngeal cancer in India stands at approximately 28%, lower than rates observed in China and Korea [2].

Early diagnosis and comprehensive preoperative assessment are crucial for curing the disease and preserving function. Narrow Band Imaging (NBI), an advanced optical image enhancement technology, is instrumental in the early detection and surveillance of head and neck cancers, including LCA. It demonstrates a high sensitivity (88.9–97.0%) and specificity (84.6–96%) in identifying early laryngeal cancers [3].

Machine learning and convolutional neural networks (CNNs) are utilized to analyse medical images across various specialities, such as ophthalmology, radiology, gastroenterology, and dermatology. Recent advancements in computer vision (CV) and deep learning are increasingly applied in analysing images of upper aerodigestive tract lesions. CV techniques are employed in classifying, detecting, and segmenting these images. The image diagnosis is based on gross morphological features supported by histopathology helps to make inferences and predictions. These algorithms have significant potential in both the diagnosis and management of diseases.

This study aims to develop a CNN algorithm for classifying laryngeal lesions into benign and malignant. Such a tool could be particularly valuable in India, where there is a notable shortage of trained, experienced endoscopists and limited access to telemedicine facilities. Implementing this technology could facilitate early diagnosis of laryngeal cancer in a large and diverse population.

Materials and Methods

Experiment Setup

The experiment was carried out using an 11th-generation Intel(R) Core (TM) i7 processor and Nvidia(R) GeForce (TM) RTX(TM) 3060 GPU. All the code was written in Python using the Pytorch framework. The Current study experimented with ResNet50 architecture as a feature extractor. The learning rate was 0.001, the optimiser was ADAM, and a Batch Size of 32 was used. The image augmentation was done by patching-based sampling. The laryngeal lesion classification is based on vascular pattern characteristics in white light and NBI images (Ni et al.,2011). The ENT expert (RS) annotated the images with more than eight years of experience in WL/NBI images, confirming it with histopathological diagnosis. The computer vision annotation tool (Cvat- v2.11.1) was used to annotate the images.

Ni classification (Fig-1).

Fig. 1.

Diagrams of microvasculature and endoscopic views of vocal folds, illustrating classification of intraepithelial papillary capillary loop features using narrow band imaging [4]. Type I (A1, A2 and A3): thin, oblique and arborescent vessels are interconnected and intraepithelial papillary capillary loops are almost invisible. Type II (B1, B2 and B3): diameter of oblique and arborescent vessels is enlarged, and intraepithelial papillary capillary loops are almost invisible. Type III (C1, C2 and C3): intraepithelial papillary capillary loops are obscured by white mucosa. Type IV (D1, D2 and D3): intraepithelial papillary capillary loops can be recognised as small dots. Type Va (E1, E2 and E3): intraepithelial papillary capillary loops appear as solid or hollow, with a brownish, speckled pattern and various shapes. Type Vb (F1, F2 and F3): intraepithelial papillary capillary loops appear as irregular, tortuous, line-like shapes. Type Vc (G1, G2 and G3): intraepithelial papillary capillary loops appear as brownish speckles or tortuous, line-like shapes with irregular distribution, scattered on the tumour surface

Data Collection

A total of 1978 laryngeal images using white light and Narrow Band Imaging (NBI) were collected from 960 untreated patients who presented with symptoms of hoarseness or difficulty in swallowing. These images were gathered at the Raj ENT Centre & Voice Clinic in Lucknow, India, from May 2015 to December 2023. An Olympus CV-170 processor paired with a flexible fiberoptic bronchoscope (BF-Q170) was used to capture these images. All images were taken by the author (RS) at a close range of 8–20 mm from the lesion, or the region of interest (ROI). In most cases, 4% lidocaine was applied to anesthetize the larynx for the patient’s comfort. Patients who had undergone treatment via surgery or radiotherapy were not included in this collection. The study did not include images with haziness, less brightness, white-outs and reflection due to saliva or secretions. Each captured image boasts a resolution of 1066 × 1066 and is stored in JPEG format. Some of the sample images from the dataset are presented in Fig-2. It can be seen that the sample sizes for all classes are different; also, within the class itself, the ROI size may vary. Patching-based sampling is applied to all the images before training the classification model.

Fig. 2.

Samples of the data shown for all the 9 classes after taking the ROI (labelled by laryngologist) images from original image

Datasets and Training Procedure

In this study, 1978 images from 960 patients were analysed. Labelling of these images was performed by the surgeon (RS) using Cvat. The images were categorised into nine morphological subclasses: Benign, Nodule, Polyp, Papilloma, Cyst, Reinke’s Edema, Granuloma, Leukoplakia, and Squamous Cell Carcinoma (SCC). All lesion types, except SCC, were further grouped into a sub-category labelled ‘Benign (Non Cancer)’.

The images were divided into two sets: 90% for training and validation, and 10% for testing. These images consisted of a combination of White Light (WL) and Narrow Band Imaging (NBI) types. Specifically, there were 998 NBI images and 980 WL images. The size of the original image is 1066 × 1066. If we directly use large-size images for training and testing, both time and space complexity will become high, and the model will also not learn the feature correctly, so we proposed patching-based sampling for both training and validation. After getting the ROI images from the original 1066 × 1066, the training and validation data was further divided into patches of 224 × 224 with a stride of 60. Number of patches in each image varies from 1 to 77.

After the sampling process, the total number of images for training and validation becomes 12,153 of both the categories Benign and SCC, having WL-5192 and NBI-6959. From sampled data, 85% were used for training and 15% for Validation. For testing purposes, one hundred ninety-seven (197) original images were used (SCC-43, Benign-154).

The classification model used throughout the process is ResNet50 [5]. The model was trained for 25 and 50 epochs, and the results for both are reported in the result section.

Testing has been done Using Two Procedures

Testing Procedure-(a)

From the original image of 1066 × 1066, I first cropped the ROI region suggested by the laryngologist and resized the image to 256, then took the 224 × 224 image from the centre as the input for the classification model. Deep learning model ResNet50 used as classification model (Fig-3).

Fig. 3.

After getting the cancerous region from original image, resizing the image to 256 and then taking 224 from centre and sending it to classification model

Testing Procedure-(b)

From the original image of 1066 × 1066, first cropped the ROI region suggested by the laryngologist and then used the sliding window size of 224 × 224 with stride 60 to get the all-possible patches. A number of patches varied from 1 to 77. All patches are sent for the classification to the classification model, and at the end, voting process is used for the final prediction of the Cancer or Benign (Non-Cancer) class. Deep learning model ResNet50 used as the classification model.

Results

Out of the total 1978 images (WL + NBI), White Light has 980, and NBI has 998 images. In the White light category, cancer is 210, and non-cancer is 770 images. In the NBI category, cancer is 220, and non-cancer is 778 images. Total cancer (C) images are 430, and non-cancer (NC) are 1548 (See Fig. 4).

Fig. 4.

After getting the cancerous region from original image, finding all the possible patches of 224 and sending them to the classification model and then doing voting process for final class

The data distribution is 90% train and validation images and 10% are test images. Sampled data: Size: 224 × 224, stride:60 for Both (NBI and White Light): Cancer-6281 and Non Cancer-5872. For NBI: Cancer- 3609 and Non Cancer- 3350. For White Light: Cancer- 2674 and Non Cancer-2518.

The Data Distribution in sampled data is 85% for training and 15% for validation. Testing Data include Both Data: C-43 and NC-154. For NBI-test data, C-22 and NC-77. For White Light-Test Data, C-21 and NC-77.

Results for both testing procedures reported below:

This paper’s standard evaluation metrics for classification include Accuracy, Precision, Recall (Sensitivity), Specificity and F1-Score [6].

For Procedure-(a)

Testing result using resizing the ROI to 256 and taking an image having size 224 from the Centre. Best performing model is the model that trained on NBI data for 25 epochs (See Table 1 and 2).

Table 1.

Testing done by directly resizing the ROI image to 356 and taking image size 224 from center. The best performing model is model trained on NBI data for 25 epochs

Training Data	Test Data	Test Accuracy(model trained for 25epochs)	Test Accuracy(model trained for 50 epochs)
Both	Both	65.48	71.07
Both	NBI	78,79	76.77
Both	White light	69.36	73.47
NBI	Both	83.25	78.68
NBI	NBI	77.87	75.76
NBI	White light	80.61	78.57
White light	Both	64.97	64.97
White light	NBI	55.56	62.63
White light	White light	69.39	69.39

Table 2.

Testing done by taking all possible patches of image having size 224 with stride 60. The best performing model is model trained on Both data for 50 epochs

Training Data	Test Data	Test Accuracy(model trained for 25 epochs)	Test Accuracy(model trained for 50 epochs)
Both	Both	88.32	90.86
Both	NBI	91.91	94.94
Both	White light	91.83	95.91
NBI	Both	86.80	86.80
NBI	NBI	85.85	81.81
NBI	White light	77.51	79.59
White light	Both	89.34	78.68
White light	NBI	74.74	62.62
White light	White light	90.81	90.81

For Procedure-(b)

Testing the result by taking the all-possible patches of 224 and then voting on the classification results on those patches. The best performing model for this case is the model trained on Both data for 50 epochs.

Analysis for best Performing Model (Test Data-197)

Confusion Matrix on both data (WL/NBI)-The total number of testing images is 197, consisting of 43 Cancer and 154 Cancer images. For test data including both NBI and White Light, evaluation metrics are P: 73.58, R: 90.70, F1-Score: 81.25, Accuracy: 90.86, Sen: 90.70, Spec: 90.91. For NBI test data, evaluation metrics are P: 84.00, R: 95.45, F1-Score: 89.36, Accuracy: 94.95, Sen: 95.45, Spec: 94.81. For White Light test data evaluation metrics are P: 94.74, R: 85.71, F1-Score: 90.00, Accuracy: 95.92, Sen: 85.71, Spec: 98.70. Here, P-Precision, R- Recall (Sensitivity), Sen- Sensitivity, Spec- Specificity. Confusion metrics for all the tree cases are presented below in Fig-5.

Fig. 5.

Confusion metrics for the test data (a) Both NBI and White Light, (b) NBI and (c) White Light

The ROC (Receiver operating characteristic curve) [7] for the test images is shown in Fig-6. It shows how well our classifier classifies Cancer and Non-Cancer classes. The area of the ROC curve for Both (NBI + WL), NBI, and White Light has ROC areas of 0.95, 0.98 and 0.93, respectively. The accuracy on both data (WL/NBI) is 90.86%, on NBI data 94.95% and on White light data is 95.92%.

Fig. 6.

ROC curve for Both (NBI + WL), NBI and White Light having ROC area of 0.95, 0.98 and 0.93 respectively

Discussion

CNN-based tools are used efficiently to diagnose various medical conditions like skin cancer, gastric cancer, breast cancer metastasis and diabetic retinopathy. Gulshan and his colleagues reported a sensitivity of 97.5% and a specificity of 93.4% for detecting referable diabetic retinopathy [8]. Hirasawa et al. (2018) developed a CNN model that required 47 s to analyse 2296 test images with correct diagnosis in 71 of 77 gastric cancer lesions with an overall sensitivity of 92.2% [9]. Early applications of machine learning in the field of otolaryngology and head and neck surgery have encompassed the automated identification of auditory brainstem response patterns and genomic forecasting of oral squamous cell carcinoma [10, 11].

Deep learning (DL) approaches, particularly those utilising convolutional neural networks (CNNs), are becoming more prevalent in the analysis of Upper Aerodigestive Tract (UADT) video endoscopy for the automatic identification, categorization, and segmentation of diseases. Image segmentation serves the purpose of identifying objects within images through the delineation of their boundaries and the areas they encompass. In the realm of computer vision, semantic segmentation is defined as the process of allocating each pixel in an image to a specific class from a predefined list. Among various tasks in computer vision, semantic segmentation holds significant interest for UADT endoscopy due to its utility [12].

In this study single operator diagnosed it clinically, collaborated it by histopathology report and labelled the region of interest in these images. ResNet-50 CNN model used as classifier. It is learned via specific optimisation algorithms and later automatically predicts the category label of the test image. The sensitivity (recall) on test images on NBI is higher as compare to WL. In contrast, the Precision is higher with WL images.

All images for training and testing in this study are HD and were directly taken from the pen drive of the same Olympus processor. The images were captured at a close distance, excluding any with blurring, whiteouts, secretions, and specular or saliva reflections. Privacy concerns were addressed, and labour-intensive annotation was performed by the same operator (RS). Due to the use of the same operator and machine, this study has no inter or intra-observer variability. The interpretability of the learning model is straightforward. However, the current model might not perform well with standard definition or HD images from far from the region of interest. The current model can be tested on images taken with an HD camera by other manufacturers, images captured from a greater distance, mucosa treated with chemo-radiotherapy, and other granulomatous laryngeal lesions specific to Southeast Asia and Asia-Pacific.

Yao et al. implemented a DL model on 22,132 image frames derived from 137 flexible laryngostroboscopic videos, including 55 from healthy subjects and 82 from those with diseases. This model was designed to select pertinent frames from the videos autonomously. In categorising frames as informative versus uninformative, the model demonstrated high accuracy, achieving a precision of 94.4%, a recall of 90.2%, and an F1-score of 92.3% [13].

Numerous studies have indicated that DL models perform well in binary classification scenarios. However, when it comes to multiclass classification, there is a noticeable decline in the performance metrics [14–16]. In the study by Xiong et al., which analysed 14,897 images, the accuracy for four-class classification stood at 77.3%, whereas the accuracy for binary classification reached 89.7%. The primary focus of the current study was on binary classification despite the identification of various subclasses within the benign category. Future research could explore these subclasses further.

Our algorithm can significantly enhance the real-time delineation of cancer margins during transoral laser surgeries using NBI technology. We can train the model effectively by semantic segmentation of WL/NBI images and annotating video frames. This model can then be utilised for image segmentation and surgical margin sampling (17).

In the study by Claudio Sampieri et al. (2024), the SegMENT plus segmentation model was applied to 3,933 images, demonstrating high Dice Similarity Coefficient (DSC), Intersection over Union (IoU), accuracy, and fast inference speed. Additionally, the model’s performance was robust on an external testing set of 356 images, yielding positive outcomes during the analysis of intraoperative laryngoscopic videos (17). Furthermore, Nazila Esmaeili and colleagues (2021) employed a Deep Convolutional Neural Network (DCNN) to differentiate between benign and malignant groups using Contact Endoscopy-Narrow Band Imaging (CE-NBI). Combining both technologies and utilising the ResNet50 model, it achieved an accuracy of 0.835 on previously unseen data.

Our model is designed to facilitate the early detection of benign and malignant lesions in the larynx. In Southeast Asia, there is a high occurrence of laryngeal cancer, often diagnosed at later stages of the disease. Using magnification and image enhancement technologies like narrow-band imaging (NBI) is expensive and demands additional training and expertise. This model can be used in remote clinics equipped with a WL/NBI flexible scope and by trained paramedics and speech-language pathologists for cancer screening. They will be trained in flexible laryngoscopy, identifying various laryngeal lesions and advanced screening techniques of image capture from the region of interest.

It has been observed that the interpretation of vascular patterns is subjective and depends on the clinician’s experience [18–21]. By analysing the texture of laryngeal images, this algorithm minimises reliance on the operator’s skill. It reduces variations in observations between different clinicians, thereby notifying healthcare providers of potentially premalignant conditions.

Conclusion

The integration of AI in UADT endoscopy is quickly becoming a pivotal area with the potential to achieve levels of clinically practical performance, setting the stage for radical changes in otolaryngology practices in the foreseeable future. In this scenario, creating models intended for immediate use in office and surgical environments becomes critical. In India, where the incidence of head and neck cancer is high and there is a lack of both advanced instruments and expertise in Narrow Band Imaging (NBI), this model could be instrumental in the early detection of laryngopharyngeal cancer by deploying the model using Hugging Face Space and Gradio. The primary challenge lies in gathering high-quality image data. Moreover, validating this AI model with external datasets is crucial to ensure its accuracy and reliability for clinical use.

Declarations

Ethical Approval

High standard of Ethical consideration taken.

Research Involving Human Participants and/or Animals

No animal.

Informed Consent

Informed consent was taken for the procedure.

Conflicts of Interest

No conflict of interest.