Improving Laryngoscopy Image Analysis Through Integration of Global Information and Local Features in VoFoCD Dataset

Abstract

The diagnosis and treatment of vocal fold disorders heavily rely on the use of laryngoscopy. A comprehensive vocal fold diagnosis requires accurate identification of crucial anatomical structures and potential lesions during laryngoscopy observation. However, existing approaches have yet to explore the joint optimization of the decision-making process, including object detection and image classification tasks simultaneously. In this study, we provide a new dataset, VoFoCD, with 1724 laryngology images designed explicitly for object detection and image classification in laryngoscopy images. Images in the VoFoCD dataset are categorized into four classes and comprise six glottic object types. Moreover, we propose a novel Multitask Efficient trAnsformer network for Laryngoscopy (MEAL) to classify vocal fold images and detect glottic landmarks and lesions. To further facilitate interpretability for clinicians, MEAL provides attention maps to visualize important learned regions for explainable artificial intelligence results toward supporting clinical decision-making. We also analyze our model’s effectiveness in simulated clinical scenarios where shaking of the laryngoscopy process occurs. The proposed model demonstrates outstanding performance on our VoFoCD dataset. The accuracy for image classification and mean average precision at an intersection over a union threshold of 0.5 (mAP50) for object detection are 0.951 and 0.874, respectively. Our MEAL method integrates global knowledge, encompassing general laryngoscopy image classification, into local features, which refer to distinct anatomical regions of the vocal fold, particularly abnormal regions, including benign and malignant lesions. Our contribution can effectively aid laryngologists in identifying benign or malignant lesions of vocal folds and classifying images in the laryngeal endoscopy process visually.

Introduction

Laryngoscopy is one of the most common subclinical techniques in otolaryngology, which is used to diagnose and treat laryngeal diseases, especially ones relating to vocal folds [1]. Vocal folds and their lesions in the glottis can be conveniently observed and evaluated by clinical doctors with the assistance of laryngoscope systems. However, lesions spreading across the anterior commissure can challenge doctors in making precise judgments, leading to prolonged treatment times [2]. Consequently, automatically localizing vocal fold lesions can potentially assist doctors with diagnosis, prognosis, and minimizing interventions, ultimately facilitating surgical approaches.

On the other hand, deep learning techniques have become increasingly prevalent in medical imaging and promoted computer-assisted clinical systems for laryngoscopy diagnosis. Learning large amounts of complex medical imaging data enables neural networks to identify complex patterns and features that may not be visible to the human eye [3–7], leading to more accurate diagnoses and improved treatment outcomes [8–10]. Leveraging neural networks in laryngoscopy image analysis has demonstrated the potential to enhance the diagnosis and treatment of larynx-related issues [11, 12]. These techniques can provide real-time assistance during the procedure [13], enabling surgeons to obtain precise estimations of lesion spread, edge of lesions, or typical tissue damage without needing immediate biopsy [14–16], enhancing clinical decision-making and patient care.

Nonetheless, deep learning applications in laryngoscopy analysis have not been thoroughly investigated. Previous work on laryngoscopy image classification typically applied machine learning techniques to categorize extracted vocal fold features, such as color, texture, geometry, and shape [17–20]. Meanwhile, early studies in detecting vocal folds only used convolutional neural networks (CNNs) to obtain initial feasible results [21–24]. Recent studies [24–28] in laryngeal endoscopy analysis can identify the normality of the vocal folds or diseases, such as benign, precancerous, and cancerous, potentially supporting the diagnosis of laryngeal diseases. However, the obtained results still had limitations, such as one-task datasets, no explainable mechanism decided on the pixel level, and no evaluation of the model’s effectiveness through clinical image simulation before practical usage.

Concretely, one of the challenges is the need for multitask annotated data. Most of the available training datasets focus on a certain task, either image classification or object detection, as the related work section. Furthermore, approaches in previous research have yet to be proposed for simulating multiple tasks using laryngoscopy images. It can only perform object detection and image classification individually. Their final results have yet to supply dual tasks, such as the content of an overall image as well as object locations and categories shown in images. These pieces of information are crucial to supplying detailed post-procedure reports that ENT clinicians need for general evaluation and interested localization for the proper treatment and care of patients. In addition, the presence of artifacts (e.g., shadows, blurs, and specular highlights) in laryngoscopy images can affect the application of deep learning models in real-world clinical facilities. To reduce these effects, initial simulated experiments for training cutting-edge deep learning methods are essential to be established.

To this end, we first introduce a new dataset, named VoFoCD, the first multitask laryngology image dataset, to encourage more studies in this field. It consists of 1724 laryngology images, manually annotated by medical experts at both image and object-wise levels. Unlike existing datasets, ours support multiple tasks in laryngology analysis with four image-level vocal fold condition categories (i.e., normal vocal folds, vocal folds with benign tumors, vocal folds with malignant tumors, and no vocal fold findings) and six glottic object-level bounding box ground-truth (i.e., left vocal fold, right vocal fold, left arytenoid cartilage, right arytenoid cartilage, benign lesion, malignant lesion). The dataset can thus support not only vocal fold classification tasks but also detecting vocal fold landmarks and lesions. Furthermore, we present a benchmark suite to facilitate evaluating and advancing the simultaneous detection and classification of the vocal folds, their relevant landmarks, and lesions in laryngoscopy images. We conducted an extensive evaluation and in-depth analysis of state-of-the-art classification and detection methods to show their potential applications in laryngoscopy analysis.

Additionally, we develop a novel multitask network for both vocal fold classification and glottis detection, dubbed Multitask Efficient trAnsformer for Laryngoscopy (MEAL). Inspired by the work of Le et al. [29], which aims to enhance the performance of the downstream task (i.e., camouflage segmentation, which can be applied to segment hidden tumors) by fusing global information (i.e., classification results), our proposed MEAL network leverages global features to improve glottis detection. Nonetheless, they directly multiplied the same global classification probability value to all regional segmentation features, leading to incorrect results when multiple objects appear, which will not be helpful in our dataset. To overcome this limitation, we embed global representations (i.e., the overall context of medical images extracted by the general classification model) to enrich local features (i.e., details identified by the detection network, focusing on lesions or other areas of interest within the laryngoscopy images) and enhance the regularization effect, improving the identification of larynx anatomical landmarks and lesions.

Last but not least, we demonstrate the interpretability of our proposed MEAL network (i.e., explainable artificial intelligence—XAI) by integrating a masked attention mechanism [30] into our model to visualize relevant regions that the model focuses on. Our XAI solution can give clinical domain experts convincing evidence of the output results.

Our key contributions are summarized as follows:

• We present a new laryngology image dataset, namely VoFoCD, that supports both vocal fold conditions’ identification and lesion localization.

• We propose a novel method for multiple tasks, called Multitask Efficient trAnsformer for Laryngoscopy (MEAL). Our network leverages both local and global embedding techniques to improve the performance of detecting lesion categories.

• We introduce an XAI solution for aided diagnostics for doctors. Utilizing a masked attention mechanism, our proposed network can visualize useful regions in vocal fold images.

• We insightfully analyze and discuss identifying conditions and localizing lesions in simulated laryngoscopy images of the laryngoscopy process.

Related Work

Deep Learning in Laryngoscopy Analysis

Before the deep learning era occurred, laryngoscopy research was mainly performed using conventional machine learning algorithms to analyze laryngoscopy images. Such works extracted some features of the vocal folds, such as color, texture, geometry, image intensity, gradient direction, frequency content, and vascular structure. Afterward, various algorithms such as Naive Bayes, Multilayer Perceptron, K-Nearest Neighbors, Support Vector Machine, and Random Forest were utilized to evaluate image classification and achieved reasonable accuracy [17–20].

Until recently, the application of deep learning, specifically CNNs, in supporting the diagnosis of laryngeal problems, particularly vocal folds, through laryngeal endoscopy images has continuously been applied and developed. Xiong et al. [28] adopted the GoogLeNet InceptionV3 network to automatically classify laryngeal cancer, precancerous laryngeal lesions, benign laryngeal tumors, and normal tissues in laryngoscopic images. Meanwhile, ResNet-101 was utilized as a lesion classifier, providing a valuable reference for the screening of laryngeal neoplasms during laryngoscopy [27]. Furthermore, it supplied distinguishing benign, precancerous, and cancer in a real-world condition and compared against the clinical visual assessments by 12 otolaryngologists. Cho et al. [25] compared the performance of different CNNs, including VGG16, InceptionV3, and Xception, in determining the normality of vocal folds through laryngeal endoscopic images. These authors also expanded their datasets to analyze further normal vocal folds and eight possible common diseases of vocal folds [26]. The model based on the U-Net architecture of Yousef et al. [24] automatically detected vocal fold obstructions in HSV data of vocally normal and AdSD instances with challenging conditions in running speech, achieving an impressive average overall accuracy of 94.81% on a massive number of HSV frames. Meanwhile, Yao et al. [31] developed and validated a pre-trained ResNet18 model for distinguishing healthy vocal folds and vocal fold polyps on laryngoscopy videos while demonstrating the ability of a previously developed informative frame classifier to facilitate deep learning development [32]. The polyp classifier trained on machine-labeled frames achieved an accuracy of 85%, higher than that of human-labeled frames, which achieved an accuracy of 69%.

Automated detection of anatomical structures in medical image analysis is a prerequisite for autonomous diagnosis as well as various computer- and robot-aided interventions. Applying various CNNs, such as ResNet, Inception, and MobileNet, to recognize two categories (i.e., vocal folds and tracheal rings) during the laryngoscopy or bronchoscopy processes helped improve airway management by helping novices or infrequent incubators and bronchoscopists identify key anatomy in real-time [21]. Adamian et al.[33] described a user-friendly computer vision tool based on CNNs for automated quantitative tracking of vocal fold motion from video laryngoscopy. They trained a deep learning model for vocal fold localization from video endoscopy for automated frame-wise estimation of glottic opening angles to find vocal fold movement disorders. Bur et al. [34] localized structural laryngeal lesions and classified them at the object level as benign or suspicious for malignancy within digital flexible laryngoscopic images using state-of-the-art computer vision detection models. Four available detection models, such as VFNet, GFLV2, PAA, and ATSS, were employed to localize and classify structural laryngeal lesions in laryngoscopic images simultaneously. Generally, classification accuracy was 88.5% for laryngeal images, and mAP50 across all four detection models was 50.1%.

To the best of our knowledge, no dataset has focused on classifying images and detecting objects of white-light laryngoscopy images at once to date. Unlike previous approaches, our work introduces a two-stage algorithm that mirrors the natural process medical experts follow when screening laryngoscopy images. This simple and intuitive two-stage process enhances our algorithm’s performance, surpassing traditional methods. Moreover, most previous research did not take advantage of local and global conceivability as a two-stage process, in which the object and image classification followed the vital anatomical landmarks attention. During the object detection stage, our algorithm prioritizes local features, scrutinizing various vocal fold regions, particularly abnormal areas like benign and malignant lesions. Subsequently, for a more nuanced classification of lesions, we harness the global features obtained from the preceding laryngoscopy classification stage, incorporating this additional but crucial information. This holistic strategy enhances diagnostic accuracy while adhering to the established workflow of medical professionals, ensuring our method is a valuable and user-friendly aid for clinicians.

Object Detection in Medical Images

We provide a concise overview of recent deep learning object detection methods in medical imaging, informing the architectural decisions behind our proposed model. The field of medical imaging detection has witnessed notable advancements with the emergence of several sophisticated object detection models. One of the most popular object detection frameworks is Faster Region-based Convolutional Neural Network (RCNN) [35], which has shown promising results in various medical imaging applications [36–38]. Another widely used object detection model is EfficientDet [39], demonstrating state-of-the-art benchmark dataset performance.

Since Vision Transformers (ViT) [40] emerged as a promising approach for image classification tasks, several recent works have extended ViT to the object detection task. Detection Transformer (DETR) [41] has gained significant attention, including medical imaging [42, 43], due to its capability to detect objects in a single shot without requiring a region proposal network.

In addition to these models, recent works have also focused on incorporating multiscale features and attention mechanisms to improve detection performance in medical imaging. For example, Multi-Scale Attention with Dense-U-Net (MSA-DUNet) [44] utilized a dense U-Net architecture with multi-scale attention mechanisms to improve the detection accuracy of liver lesions.

YoloV5 model [45] is a versatile object detection algorithm using a single-stage object detection pipeline that combines anchor-based techniques with other improvements to achieve high accuracy and fast inference times on GPUs. The model has gained popularity in the object detection community due to its beginner-friendly engineering. Recently, there has been a growing trend among researchers in the medical imaging field to adopt the YOLOv5 model [46, 47].

Our proposed model utilizes YOLOv5 as the glottic object detection backbone and Efficient Transformer as the enhanced lesion classifier, leveraging merits from both CNN and Transformer, thus achieving promising results in the evaluation stage.

Materials and Method

VoFoCD Dataset

Data Collection

We present a dataset of laryngeal images collected from 304 outpatients and inpatients at the Department of Otorhinolaryngology, Cho Ray Hospital in Ho Chi Minh City, Vietnam, between 2019 and 2021. A total of 1724 laryngeal images were acquired using a flexible endoscopic system (Olympus Medical Systems Corp., Tokyo, Japan) connected to a camera, with a resolution of 480 × 360 pixels.

The research has gained approval from the ethics committee at Cho Ray Hospital in Ho Chi Minh City, Vietnam. It adheres to the ethical standards of the Declaration of Helsinki. As this is a retrospective study, the requirement to obtain informed consent from patients is waived.

Data Annotation

The annotated dataset creation unfolded in the following manner: Laryngoscopy images were gathered in Cho Ray Hospital, and then the ground truth was initially established independently by two junior ENT doctors. Their annotation criteria were rooted in pathological diagnoses and their clinical experiences. The classification task involved labeling four distinct vocal fold conditions, while the detection task encompassed generating bounding boxes for six objects within the glottis-related area. Subsequently, the doctors cross-evaluated each other’s results to achieve consensus on the labeled data. When disagreements arose and decisions could not be reached, these images were given for expert consultation in the subsequent step. Finally, a senior expert conducted a manual review and revision of the dataset.

Collected images were classified into four categories based on the availability and condition of the vocal folds: no vocal fold findings (444 images), norma vocal folds (379 images), vocal folds with benign tumors (491 images), and vocal folds with malignant tumors (410 images). Additionally, we localized six objects for detection tasks, including right and left vocal folds, right and left arytenoid cartilages, and abnormalities of the vocal folds with benign or malignant lesions, as in Table 1.

Table 1.

Overall laryngology image dataset

Task	Categories/objects	Train	Test	Total
Image classification	No vocal fold Normal vocal folds Vocal folds with benign lesion Vocal folds with malignant lesion All categories	434 301 394 328 1457	10 78 97 82 267	444 379 491 410 1724
Object detection	Left vocal fold Left arytenoid cartilage Right vocal fold Right arytenoid cartilage Benign lesion	955 634 976 652 523	247 158 244 168 129	1202 792 1220 820 652
	Malignant lesion	385	103	488
	All glottic objects	4125	1049	5174

We remark that patient data were pseudonymized, and biopsy results were used to label the images into benign and malignant lesions based on the WHO classification, such as the benign tumor class including nodules (n = 152), cyst (n = 14), polyp (n = 175), papillomatosis (n = 19), hyperplasia (n = 27), hyperkeratosis (n = 46), and mild dysplasia (n = 58); and the malignant group including moderate dysplasia (n = 21), severe dysplasia (n = 36), carcinoma in situ (n = 17), and squamous cell carcinoma (n = 336). Examples of our VoFoCD dataset are illustrated in Fig. 1.

Fig. 1.

Examples of our VoFoCD dataset with four categories for vocal folds image classification and six glottic object types for vocal fold landmark and lesion detection

Data Splitting

Due to the rarity of lesions, particularly malignant ones, in the dataset, it is crucial to ensure fair data splitting that preserves the distribution of objects between the train set and test set. To achieve this objective, we propose our method that takes into account the varying number of objects in each image.

Initially, we assign a fabricated label to each image using one-hot encoding based on the ground truth objects. This process generates a vector for every image, with each value indicating the presence or absence of a specific object within that image.

Subsequently, we conduct a statistical analysis of all the generated vectors and distribute them fairly between the training and testing sets, maintaining an 80/20 ratio. This approach guarantees that both sets exhibit a similar distribution of objects. Detailed statistics can be found in Table 1, showcasing the efficacy of our splitting method in achieving a balanced distribution.

Proposed Method

Overview of Network Architecture

Our proposed methodology, Multitask Efficient trAnsformer for Laryngoscopy (MEAL), comprises three principal modules: Global Laryngoscopy Classifier, Glottic Object Detector, and Reinforced Lesion Classifier.

Figure 2 depicts the key constituents of our approach. The workflow is structured as follows: an input image is fed into Global Laryngoscopy Classifier and Glottic Object Detector to generate global and local embedding, respectively. Afterward, the global embedding can be utilized for image-level classification directly, while the glottic detector localizes glottic objects along with their corresponding labels. If the obtained object labels consist of lesion categories, the Reinforced Lesion Classifier utilizes the corresponding local and global features to improve the final predictions. The following sections thoroughly explain these principal modules.

Fig. 2.

Overview architecture of our proposed Multitask Efficient trAnsformer for Laryngoscopy (MEAL). Global Laryngoscopy Classifier extracts global embedding of the laryngoscopy image, while the Glottic Object Detector focuses on localizing different local embedding of anatomical regions of the image. If the former detects a lesion from the image, Reinforced Lesion Classifier equipped with Efficient Transformer [54] fuses both local and global embedding of previous components to give the final enhanced prediction. FFN, feed-forward neural network; ROI, Region-of-Interest; GAP, Global Average Pooling; Conv, Convolution; GT BBox, Ground truth bounding box

Global Laryngoscopy Classifier

Many convolutional neural networks with superior performance in pattern recognition have been proposed recently. However, our goal is to achieve both effectiveness (i.e., good performance) and efficiency in processing (i.e., reasonable inference time). We emphatically conducted experiments and found that EfficientNet [48], well-known for its good performance in image classification, can achieve both our requirements (see Table 2). Therefore, EfficientNet is employed as the Global Laryngoscopy Classifier.

Table 2.

Benchmark results of vocal folds classification

Model	No. parameters	Frames per second (FPS)	Accuracy	F1-score	Precision	Recall
ConvNeXt pico [53]	8,534,724	271	0.839 (0.039)	0.841 (0.041)	0.848 (0.039)	0.839 (0.039)
ResNet50 [54]	23,516,228	213	0.948 (0.009)	0.949 (0.009)	0.951 (0.009)	0.948 (0.009)
MobileNetV3 [55]	4,207,156	352	0.936 (0.027)	0.937 (0.027)	0.938 (0.026)	0.936 (0.027)
InceptionV3 [56]	21,793,764	247	0.930 (0.018)	0.934 (0.015)	0.937 (0.017)	0.933 (0.015)
EfficientNet-B0 [42]	4,012,672	280	0.951 (0.015)	0.951 (0.015)	0.950 (0.015)	0.948 (0.015)

The best (in bold) and second-best (underlined) results are highlighted

Glottic Object Detector

In recent years, the object detection community has shifted away from two-staged detectors, such as Fast-RCNN [49] and Faster-RCNN [35], towards fast and lightweight single-stage detectors, such as SSD [50] and the YOLO family [51–54]. In this work, we leverage the well-engineered YoloV5 [45] as the Glottic Object Detector due to its fast processing and high accuracy.

Reinforced Lesion Classifier

Although YoloV5 has demonstrated superior performance, we found that YoloV5-based Glottic Object Detector exhibits exceptional glottic object detection capability but unsatisfactory classification performance, especially in discriminating between benign and malignant lesions. This is potential because one-stage detectors attempt to achieve multiple objectives using a single-loss function. To address these limitations, inspired by a few studies [29, 55, 56], which utilized multiple feature learning, we propose a Reinforced Lesion Classifier to leverage both global and local embeddings, building on the rationale that an abnormal laryngoscopy image inherently contains crucial details about local abnormalities. To this end, we aim to utilize the intrinsic value of this mutual information, particularly from both local and global embeddings—extracted by the Global Laryngoscopy Classifier and Glottic Object Detector. Therefore, we introduce the Efficient Transformer [57], the core component of Reinforced Lesion Classifier, which is capable of capitalizing on the mutual information encoded in both embeddings. Our lesion classifier targets only differentiating between the malignant lesion and benign lesion since accurate lesion classification is essential for proper diagnosis and treatment in medical image analysis.

To extract local features, our solution is to utilize the Region-of-Interest (ROI) pooling technique [49], followed by Convolutions 1 × 1 to extract features accurately from desired regions of the input image. This approach has two main advantages: first, it allows for convenient extraction and scaling of features of different scales from the Glottic Object Detector’s feature pyramid into the same spatial dimension; second, it enables the concatenation of all pyramid local features together into an enhanced local embedding (see Fig. 2).

To leverage the benefits of global embedding, the features extracted from the Global Laryngoscopy Classifier are going over a Convolutions 1 × 1 before concatenating with the enhanced local embedding, resulting in a fusion of global and local embedding. This approach enables our model to effectively capture both local and global information, improving its ability to detect objects accurately.

After that, the Reinforced Lesion Classifier is employed to learn attention maps, leveraging the self-attention mechanism [58] to capture long-range dependencies and incorporate global context information from the resulting global–local fusion embedding while reducing computational complexity through depthwise convolutions [59]. We refer the readers to the original paper for more information about Efficient Transformer and depthwise convolutions. The fusion of global and local embeddings acts as query, key, and value for the attention mechanism, with query and key undergoing regularization via element-wise multiplication with the original global embedding. In line with established practices in deep learning, we apply Global Average Pooling (GAP) to condense CNN embeddings before passing them through the subsequent Feed Forward Network (FFN) and the final softmax Lesion Classifier.

In conclusion, by leveraging the classification result and embedding of the Global Laryngoscopy Classifier as prior knowledge, the Reinforced Lesion Classifier produces more accurate results, especially for benign and malignant regions. The results of both these classifiers are valuable references that can aid the workflow of otolaryngologists. The right half of Fig. 2 details the architecture of our proposed Reinforced Lesion Classifier.

Implementation Details

In our study, all the experimented CNN architectures of the Global Laryngoscopy Classifier were trained using Cross Entropy Loss over 100 epochs, a batch size of 32, and an image size of 512 on an NVIDIA A100 GPU. We utilized the AdamW optimizer [64] with a starting learning rate of 0.001 and L2 regularization of 0.01. The exponential decay rates for gradient averages (β1) and squared gradient averages (β2) were established as 0.937 and 0.999, respectively. The learning rate is halved every 10 iterations if there is no improvement in validation loss. Additionally, to replicate diverse data qualities and real-world noise scenarios, we incorporated various data augmentation techniques, including random cropping, geometric affine transformations, and color adjustments, to produce more data samples.

The Glottic Object Detector was trained using the default YoloV5s settings with low data augmentation due to the relatively small size of our dataset to avoid overfitting. The loss function was employed as in their respective original papers.

The Reinforced Lesion Classifier was trained using Cross Entropy Loss. The training process was augmented with the same method as in Global Laryngoscopy Classifier but with a lower probability of 0.3.

Evaluation Metrics

We evaluated the image classification task and key anatomy detection task using different evaluation metrics. For image classification, we reported several important parameters, inference time, accuracy, and F1-Score performed by classification models. For object detection, mAP50, the average precision at the IoU threshold of 50%, was mainly used for comparison.

Results

Classification Benchmarking

Table 2 shows the comparison between various popular CNN models, such as ConvNeXt pico [60], ResNet50 [61], MobileNetV3 [62], InceptionV3 [63], and EfficientNetB0 [48], for vocal folds classification task in the VoFoCD dataset. These networks were trained in the same setting as the Global Laryngoscopy Classifier. Overall, EfficientNet-B0 outperforms others with an F1 score and accuracy of 0.951 while having the second least number of parameters. Such results demonstrate the remarkable efficiency of convolution neural network models for classifying global images.

Detection Benchmarking

We compared our proposed method (MEAL) with other state-of-the-art models, including Faster-RCNN [35], EfficientDet [39], and DETR [41], as in Table 3. Compared methods utilize Resnet50, whereas our MEAL adopts CSP-Darknet53 as the backbone. These methods were trained on the same settings as the Glottic Object Detector.

Table 3.

Benchmark results of glottic object detection

Model	No. parameters	mAP50
FasterRCNN [35]	43,056,434	0.840 (0.022)
EfficientDet [39]	24,553,443	0.820 (0.041)
DETR [41]	41,271,307	0.822 (0.024)
MEAL (Our)	12,214,610	0.874 (0.004)

The best results are in bold

Table 3 shows that our MEAL outperforms existing methods by a large margin while having the smallest number of parameters. Table 4 indicates the detection performance of MEAL on each glottic category.

Table 4.

Performance of MEAL in detecting each glottic object category

Category	AP50	Precision	Recall	F1
Left vocal fold	0.959 (0.009)	0.956 (0.009)	0.943 (0.011)	0.949 (0.009)
Left arytenoid cartilage	0.885 (0.025)	0.910 (0.021)	0.845 (0.025)	0.876 (0.017)
Right vocal fold	0.966 (0.010)	0.957 (0.013)	0.956 (0.011)	0.957 (0.011)
Right arytenoid cartilage	0.898 (0.013)	0.914 (0.019)	0.858 (0.016)	0.885 (0.014)
Benign lesion	0.714 (0.035)	0.827 (0.037)	0.684 (0.036)	0.748 (0.030)
Malignant lesion	0.802 (0.042)	0.885 (0.035)	0.751 (0.046)	0.811 (0.029)
All	0.871 (0.009)	0.908 (0.012)	0.839 (0.013)	0.871 (0.010)

A visualized comparison between our proposed methods and Faster R-CNN is illustrated in Fig. 3. The visual representation showcases the considerable enhancement our MEAL provides in accurately classifying vocal fold lesions compared to Faster RCNN as well as not using Reinforced Lesion Classifier. This improvement is of great importance in identifying such lesions during clinical laryngoscopy.

Fig. 3.

Visual comparison of our method against previous methods. From left to right: Faster R-CNN [35], our YoloV5-based Glottic Object Detector [45], our MEAL, ground truth, and original image

Effectiveness of Reinforced Lesion Classification

To evaluate the performance of the proposed MEAL on detecting lesions, we compared the results of using and not using the proposed Reinforced Lesion Classifier module. Table 5 reveals that integrating Reinforced Lesion Classifier surpasses its non-use counterpart, exhibiting performance gains of 0.4%, 0.8%, and 0.2% across all glottic object categories, malignant lesions, and benign lesions, respectively. Our proposed Reinforced Lesion Classifier demonstrates its efficacy, as evidenced by the observed improvements. While these results represent a promising initial attempt, we acknowledge that there is still room for improvement, particularly in addressing the challenges posed by benign lesions.

Table 5.

Performance of lesion detection of MEAL in terms of mAP50

Method	Benign	Malignant	All categories
Without reinforced Lesion classifier	0.716 (0.062)	0.815 (0.006)	0.870 (0.008)
With reinforced Lesion classifier	0.718 (0.054)	0.823 (0.009)	0.874 (0.004)

The best results are in bold

Insightful Analysis

XAI-based Interpretability Toward Aided Diagnostics

Unlike previous models that lacked an explainable mechanism to justify their results, our method enables us to incorporate this ability effortlessly by integrating an attention map. Particularly, we propose replacing the ROI pooling in MEAL with the more advanced masked attention mechanism recently introduced by [30]. This modification resulted in only minor changes in model performance but significantly improved our ability to understand how the model combines global and local features. Specifically, by leveraging the attention map generated by the softmax output of the query-key matrix multiplication, we can visualize the essential regions that each learnable query attends to. This allows us to perform explainable artificial intelligence (XAI) and to provide more convincing evidence to clinical domain experts that our model is learning valuable regions.

The learnable regions of each query are illustrated in Fig. 4. These attention maps are presented as heat maps overlaid upon the original images, with warmer colors indicating higher saliency, thereby significantly contributing to object categorization. By providing a visual representation of the key areas of focus for the deep neural network, these attention maps can facilitate the interpretation of the model’s decision-making processes.

Fig. 4.

Attention maps in our MEAL model. From left to right and top to bottom: original image, ground truth, global embedding, small features, medium features, and large features

Real-World Scenario Simulation

To investigate the robustness of our proposed model in a real-world scenario, we generated subsets of contextual images with varying levels of opacity, ranging from 10 to 40%, which correspond to the likelihood of clinical laryngoscopy. Particularly, we applied the Gaussian blur operations to the original dataset to create blurred images with different parameters. We aimed to create scenarios that mimic the clinical laryngoscopy process by simulating different opacity levels. These syntheses can mimic the conditions physicians encounter during such procedures. Hence, by subjecting our model to these contextual subsets, we can assess its performance in a manner that closely resembles its intended use in the practice.

Table 6 unequivocally demonstrates the unwavering performance of our proposed MEAL in the face of blur conditions up to 30%. This blur level has been shown to yield consistent accuracy of classification of 0.929 and detection of 0.773, a testament to the robustness of our model. The performance of our model drops only 2.2% in classification and 10.1% in detection compared with the performance on the standard condition (i.e., original images). Only when the opacity of the laryngoscopy dataset increases to 40% does the performance of MEAL start to exhibit a gradual decrease. These results showcase the impressive capabilities of our proposed MEAL in simulated clinical scenarios of the laryngoscopy process. Our model has the potential to be deployed in practical clinics.

Table 6.

Results of MEAL on synthetically blurred images on different levels

Contextual images	Classification (accuracy)	Detection (mAP)
Original	0.951 (0.015)	0.874 (0.004)
Gaussian blur 10%	0.946 (0.012)	0.811 (0.008)
Gaussian blur 20%	0.935 (0.012)	0.796 (0.007)
Gaussian blur 30%	0.929 (0.012)	0.773 (0.008)
Gaussian blur 40%	0.900 (0.016)	0.737 (0.010)

Figure 5 visualizes an example of our detection results on different blur levels. Our MEAL model proved to be a robust performer in detecting tumors in laryngoscopy images that were blurred between 0 and 30%. However, a stark misalignment is evident, running from a malignant object to a benign one in the 40%-blurred image, setting it apart from the others. Such misalignment could significantly impact clinical diagnosis and increase the risk of missing cases of malignancy. Therefore, when implementing tumor detection in a clinical setting, prioritizing the avoidance of false negatives takes precedence over false positives. Consequently, minimizing the risk of missing tumor detection is essential, even at the cost of detecting more tumors with lower accuracy.

Fig. 5.

Visualized detection results on blurred images with various levels. From left to right and top to bottom: ground truth, results of MEAL on the original image, 10% blurred image, 20% blurred image, and 30% blurred image, respectively

Furthermore, Fig. 6 demonstrates the performance of our proposed MEAL in terms of the Precision-Recall Curve (PRC), which offers a nuanced perspective on the model’s accuracy, prioritizing the false positive rate and thereby highlighting misclassifications in the negative class. Our model performed exceptionally well on the original image dataset, with high precision and recall scores. Remarkably, the model’s performance remained stable and robust even in the presence of blurring, ranging from 10 to 30%. On the other hand, at the blur level of 40%, the precision and recall scores began to decline, marking a noticeable departure from the different levels. These findings lend valuable insights into the strengths and limitations of our model, highlighting its impressive capacity to handle blurring while signaling the challenges posed by more extreme conditions.

Fig. 6.

Performance of MEAL in terms of Precision-Recall Curves (PRCs) on various levels of blurred laryngology images

Discussion

The prior studies achieved very encouraging results in generating laryngoscopy datasets for classification or detection tasks. In the classification task, Cho and Choi [25] and Cho et al. [26] compared the performance of different CNNs, determining the normality of vocal folds through laryngeal endoscopic images and expanding their datasets to classify further normal vocal folds and eight possible common vocal fold diseases. Meanwhile, Ren et al. [27] utilized ResNet-101 to validate five vocal fold lesion classifiers. Xiong et al. [28] adopted the GoogLeNet InceptionV3 network to automatically classify laryngeal cancer, precancerous laryngeal lesions, benign laryngeal tumors, and normal tissues in laryngoscopy images. The model of Yousef et al. [24] automatically detects vocal fold obstructions in HSV data of vocally normal and AdSD instances. Moreover, Yao et al. [32] developed and validated a pre-trained ResNet18 model for distinguishing healthy vocal folds and vocal fold polyps on laryngoscopy videos. In the detection task, Bur et al. [34] utilized available detection models to determine objects in images on a laryngeal image dataset, including two classifiers (benign or suspicious for malignancy). These studies serve specific tasks such as classification or detection. They do not take advantage of the potential of local or global information in laryngoscopy images. It is necessary to have a dataset for both the tasks of localizing and classifying objects in images as well as classifying whole images. The goal of our work is to leverage the local and global knowledge of images to train deep learning models as seen by a clinician. It helps the model train like doctors making clinical decisions to enhance model results.

In this study, we introduce the VoFoCD dataset, which is a multitask laryngology image dataset for both image-wise and object-wise levels. It includes four vocal fold conditions in image-level categories and six glottic bounding boxes in object-level categories. Consequently, the dataset supports vocal fold image classification tasks and the classification and localization of vocal fold landmarks and lesions. Simultaneously, we propose the MEAL model, designed for both vocal fold image classification and glottis landmark classification and detection in the VoFoCD dataset. We integrate the overall context of medical images extracted by the classification part to local features identified by the detection part. This enhancement improves the identification of larynx anatomical landmarks and lesions. Thanks to comprehensive integrated learning from images and object localities, our study exhibits a mean accuracy of 0.951, with a standard deviation of 0.015 in the four-class classification task. It is higher than the models of the studies [28] (four-class classification with 0.867), [24] (two classifiers with 0.942), and [32] (two-class classification with 0.850). Our method slightly surpasses the performance of Cho et al. [26] in their nine-class classification task, where they achieved a mean accuracy of 0.945 with a standard deviation of 0.028. Importantly, our approach demonstrates an advantage in both accuracy (higher mean accuracy) and stability (smaller standard deviation) compared to Cho et al.’s method. Additionally, our model showcases its potential in the laryngoscopy process by concurrently performing multiple tasks, including classification and detection, a capability absence in their approach. However, our result is lower than the study [25] (two-class classification with 0.997); this could be because the classifiers in [25] were simple. The mAP50 result of the six-class object detection in laryngoscopy images is 0.869, higher than that of Bur et al. [34], with 0.501.

Deep neural networks are frequently referred to as “black boxes” due to a lack of insight into their internal decision-making processes despite their demonstrated success across diverse domains. To gain the confidence of otolaryngologists and patients, it is imperative that the deep learning model developed for laryngoscopy images not only exhibits high reliability but also furnishes transparency regarding the rationale behind its decision-making mechanisms. Previous research has shown the capability of computer vision to effectively discriminate various types of vocal fold lesions in laryngoscopic images. Nevertheless, these methodologies encounter a transparency deficiency, as they fail to offer clinicians insights into the decision-making process at the object level underlying the model’s diagnosis [21, 28, 31, 33, 34]. Some previous studies [11, 25, 26, 28] initially used gradient-weighted class activation mapping to visualize results. It allows us to explain how deep learning models extract features and focus on areas important to decision-making at the image level. In our study, we underscore the interpretability of our MEAL network by integrating a masked attention mechanism into the model, aligning with the principles of explainable artificial intelligence. The attention maps can provide a visual explanation of how the model makes predictions at the pixel level and what regions of the vocal fold images are most relevant for diagnosis. By inspecting the attention maps, clinicians can verify the model’s reasoning and identify potential errors or inconsistencies. Moreover, attention maps can also help clinicians discover new patterns or features that might be overlooked by human eyes, such as subtle differences between benign and malignant lesions. Therefore, the attention maps can enhance the interpretability and trustworthiness of the model, as well as facilitate the communication and collaboration between computer scientists and clinicians.

Furthermore, when the laryngoscopy images in the acquired dataset have consistent quality and optimal conditions, the model yields highly accurate results. Nevertheless, numerous challenges may arise during the laryngoscopy procedure in practical clinical settings, including blurring due to motion-induced shaking and masking caused by secretions. Therefore, before implementation in clinical practice, it is necessary to simulate real-world images to comprehensively assess the model’s effectiveness across potential scenarios. Our endeavor involves analyzing and evaluating the model’s ability to identify conditions and localize lesions within the opacity settings of the laryngoscopy dataset, thus gauging its efficacy before embarking on clinical evaluations. Notably, our approach introduces a novel aspect by generating a set of simulation images featuring varied blur levels for model evaluation, addressing the challenges of time-consuming and expensive processes associated with collecting medical image datasets.

Since it is limited by the time and expertise required to label bounding boxes and images manually, our dataset merely has more than 1700 laryngeal images. Moreover, the number of pathology genres in the VoFoCD dataset is insufficient to analyze particular lesions. Consequently, a more extensive laryngoscopy image set of the types of vocal fold lesions with associated pathologic diagnoses is required. In addition, our MEAL model only evaluates the opacity image levels for simulated real-world scenarios. In the future, our model needs to be performed and investigated in different clinical laryngoscopy settings obtained in various real-world scenarios instead of simulation to evaluate its performance comprehensively. In this study, such a dataset is annotated for both classification and detection tasks. We will continuously annotate the dataset to serve as the segmentation task for vocal fold lesions toward support clinical practice.

Conclusion

This paper has provided a new dataset and demonstrated comprehensive experiments for our novel approach to support task parallelism. The newly introduced VoFoCD dataset comprises vocal folds image classification and anatomical landmark detection of the glottis. The classification task involves four classes, while the detection task involves identifying six object categories simultaneously.

Furthermore, we proposed an integrated narrative network, namely MEAL, that utilizes a specialized global-local fusion strategy to provide conditioned attention information for localizing and classifying glottis landmarks and lesions. The proposed method achieved state-of-the-art performance on the VoFoCD dataset. This will facilitate further research on the automatic identification of vocal folds and lesions.

In future work, we plan to add segmentation masks into the dataset along with the expansion in size as well as the diversity of pathologies of vocal folds. Finally, we will apply these deep learning models to clinical practice for testing and to support disease diagnosis and treatment.

Author Contribution

Thao Thi Phuong Dao and Tuan-Luc Huynh contributed to the study conception and design. Thao Thi Phuong Dao, Ngoc Boi Van, Chanh Cong Ha, and Bich Anh Tran prepared material, collected data, and labeled data. Thao Thi Phuong Dao, Tuan-Luc Huynh, Minh-Khoi Pham, Trung-Nghia Le, Tan-Cong Nguyen, Quang-Thuc Nguyen, and Minh-Triet Tran trained backbones, created novel models, compared to evaluate models, conducted experiments, and visualized results. The first draft of the manuscript was written by Thao Thi Phuong Dao, Tuan-Luc Huynh, Minh-Khoi Pham, Trung-Nghia Le, Tan-Cong Nguyen, and Minh-Triet Tran. All authors commented on versions of the manuscript and revised them. All authors read and approved the final manuscript.

Funding

This work is funded by Vietnam National University Ho Chi Minh City (VNU-HCM) under grant no DS2020-42-01). Thao Thi Phuong Dao and Quang-Thuc Nguyen are supported by the scholarships of Vingroup Innovation Foundation (VinIF) for Masters students (grant no. VINIF.2022.ThS.JVN.08 and VINIF.2022.ThS.JVN.10, respectively).

Data Availability

All laryngoscopy images in this study are owned by Cho Ray Hospital, Ho Chi Minh City, Vietnam, for research purposes and cannot be made publicly available owing to patient privacy and ethical concerns.

Declarations

Ethics Approval

This study was performed in line with the principles of the Declaration of Helsinki. Approval was granted by the Independent Ethics Committee of Cho Ray Hospital (Date: March 9th, 2022/No.1280/GCN-HĐĐĐ).

Consent to Participate

Informed consent was waived in this study. Because the study only uses the historical data of laryngoscopy images, there will be no interaction with or impact on patients.

Competing Interests

The authors declare no competing interests.