Weakly Supervised Learning of Oral Cavity Images for Separating Precancer from Cancer

Abstract

Early detection is crucial for reducing the morbidity and mortality associated with oral cancer. Visual examination of the oral cavity is the first step in screening for lesions that are cancer or precancer (i.e., the direct precursor to cancer). In contrast to existing work that uses image or lesion labels, we explore a weakly supervised method to address the situation where only patient-level labels are available. Specifically, we use multiple-instance learning (MIL) to classify patients with oral precancer versus cancer. Our approach involves three steps: image filtering, mouth detection, and patient-level classification. The method performs well on filtering and mouth detection and shows promising results in distinguishing precancer from cancer on an independent test set, demonstrating its potential as a proof-of-concept (PoC). We also discuss current limitations and future directions.

I. Introduction

Medical image analysis plays a critical role in the screening, diagnosis, staging, and treatment of many diseases. With the rapid development of artificial intelligence (AI) and machine learning (ML) algorithms, especially deep learning (DL) in recent years, computerized methods using these techniques have been actively studied and applied to assist physicians in analyzing different types of images across various medical applications. Although many studies have demonstrated promising results, computer-aided analysis of medical images faces unique challenges compared to natural images due to the specific characteristics of medical data [1]. One crucial challenge is that there may be a lack of sufficient label or ground truth data because of the high cost, long time, and special expertise needed for collecting annotations. Weakly supervised learning is one general category of ML techniques that aim to deal with or mitigate such a challenge. It covers three types of weak supervision: 1) incomplete supervision (only a subset of training data has labels), 2) inexact supervision (only coarse-level labels are available), and 3) inaccurate supervision (labels may be noisy or incorrect) [2]. In this study, we focus on the second type of weak supervision for our application of using clinical photographic images for oral cancer detection.

Oral cavity and oropharyngeal cancers are among the most common cancers in the world, and estimated numbers of new cases and deaths in the United States for 2025 are 59,660 and 12,770, respectively [3]. Oral cancer has one of the lowest five-year survival rates, and its advanced stage is associated with a high mortality rate [4]. Hence, early detection through screening are critical for reducing mortality and morbidity rates. Visual inspection of the mouth cavity region is usually the first step for detecting oral cancer and precancer. Suspected precancerous or cancerous lesions identified by visual evaluation are biopsied, and their histopathological analysis is the diagnostic gold standard. However, visual evaluation requires considerable experience and training, and suffers from significant inter- and intra-observer variance [5].

For helping improve the performance of visual evaluation of the mouth, AI and ML-based and other automatic image analysis techniques have been explored and developed [6–15]. For example, [8] trained a ResNet-101 model for oral image classification and a Faster R-CNN model for oral lesion detection. It also proposed a strategy to combine annotations from multiple clinicians to generate the ground truth for class label and lesion location. Both binary-class (‘lesion’ vs ‘no lesion’, and ‘referral’ vs ‘non-referral’) and multi-class (five classes based on referral decisions and disease types) models were trained and evaluated in their experiments. [10] developed a mobile App and collected an oral image dataset using smartphones. They trained MobileNet variant models and MobileViT models using their dataset for classifying normal images from cancer images and tested and compared their models on a very small subset of a Kaggle dataset. Another study [11] combined MobileNet with a Bayesian DL model to classify oral photographic images into two classes: suspicious or not suspicious. The models were evaluated on a private smartphone image dataset and a small public dataset modified and reduced from a Kaggle dataset. The authors of [13] evaluated the clinical efficacy of their dual-mode (white-light and auto-fluorescence) imaging device for oral cancer screening. The green and red channels of white-light image and the normalized ratio of red and green channels of the auto-fluorescence image were used as the inputs to two models, MobileNet on the smartphone and VGG19-based Bayesian DL network on the cloud, to classify the lesion region as suspicious or not suspicious. The model performances were compared with the diagnoses of both onsite specialists and remote specialists. In [15], suspicious regions in oral photographs were identified by the detection network of Single Shot MultiBox Detector (SSD) first and then classified as having oral cavity squamous cell carcinoma (OCSCC) or not using a DenseNet121 model. The models were trained using images of OCSCC and normal oral mucosa obtained from eleven hospitals and evaluated on the internal dataset as well as an external dataset consisting of images of OCSCC and normal oral mucosa collected from medical journals. They also compared the model classification performance with medical specialists of different levels of experience on oral cancer diagnosis using a clinical validation set that contains images of OCSCC, non-OCSCC oral diseases, and normal oral mucosa.

Our prior work along this line included several initial preprocessing tasks done using a small pilot set of oral cavity photographic images [16–19]. For example, we developed a classifier to automatically separate the images into different categories based on the main captured anatomical location sites [16] and a classifier to identify images containing a ruler [17]. We proposed an approach to automatically extract the measurement information from a ruler placed in oral cavity images [18] and developed an iterative human-in-the-loop method to expand the training set for improving the segmentation performance of the buccal mucosa region in the mouth [19]. We also reported on our efforts in cleaning and harmonizing a large multi-center oral cavity image dataset [20].

In this study, we focus on classifying patients with precancerous oral lesions from those who have cancerous oral lesions based on the characteristics of the clinical photographic images of their oral cavity. Each patient has a varied number of images. The labels provided are at the patient level and not the image level. That is, for a patient with the label cancer (or precancer), at least one of the patient’s images contains cancerous (or precancerous) lesion(s), but the information on which image has it (them) and the lesion location is not available.

For this, we explore weakly supervised learning, specifically multiple-instance learning (MIL), with a goal to advance the state of the art in research for this topic. MIL addresses scenarios in which labels are available only at the bag level rather than for individual instances, representing an inexact form of weak supervision since these labels are coarse-grained. In the binary setting, a bag is defined as positive if it contains at least one positive instance and negative if all instances are negative. Numerous DL-based MIL methods [21] have been proposed, differing in architecture, feature representation, training strategy, and loss function. Most existing medical MIL applications focused on histology, CT, or MRI images [22]. To our knowledge, this is the first study to apply and evaluate MIL for oral photographic image analysis where no image- or lesion-level annotation is available. In our formulation, each patient serves as a bag and each image as an instance. The following sections describe the dataset, preprocessing steps, MIL algorithm, experimental results, limitations, conclusions and future work.

II. Methods

A. Oral Cavity Image Dataset

The images used in this work are a subset sourced through a special collaborative research agreement from a large oral cancer study conducted at four medical centers in Taiwan by the National Cancer Institute (NCI), at the National Institutes of Health (NIH) [23]. In this study, participants with clinical precursors were followed up every six months for a period of five years, while controls and invasive cancer cases had only one visit. Since the NIH does not have direct participant contact, per the Common Rule, the study is exempt from Institutional Review Board (IRB) review. Local IRBs at the sites in Taiwan obtained necessary IRB approvals for the study protocols.

In our data, each patient has a ground truth (either precancer or cancer) determined from the biopsy result. In the experiments, we only consider the images taken at the baseline visit. Variable number of images were taken for each patient. The anatomical site captured include: 1) right buccal (RB) mucosa; 2) left buccal (LB) mucosa; 3) top of the mouth (TM); 4) floor of the mouth (FM); 5) dorsal tongue (DT); 6) ventral tongue (VT). Besides images containing the above six main anatomical fields, the data collecting staff also took another type of images, i.e., the so-called OSF images. Oral submucous fibrosis (OSF) is a chronic precancerous condition of the oral cavity that results in reduced mouth opening, among other deficits. OSF images containing the participant’s fingers/hand placed next to the opening mouth (such as the last two images shown in the bottom row of Fig.1) were taken to assess the extent of mouth opening.

Fig. 1.

Multifield oral images.

B. Image Preprocessing

The image preprocessing steps consist of two main components: OSF image filtering and mouth detection.

• OSF image filtering

  Since OSF images generally do not contain much of the oral cavity (such as examples shown in Fig. 1), we filter out those images for this task. They are identified by two approaches. One is to directly use the multifield annotation information (so-called “Location ID”) entered by the staff in the original image file name (i.e., containing “OSF”) and extracted during the stage of data cleaning and harmonization [20]. Most of the OSF images can be found this way. For the images without the multifield annotation information, we apply a classifier that we developed for categorizing images into 7 classes: RB, LB, DT, VT, FM, TM, and OSF, and enhanced from prior work reported in [16]. The classifier is based on the ResNeSt [24] network.

• Mouth region detection

  We developed a mouth detector which can help identify non-oral images as well as remove the irrelevant/background area outside the mouth. This reduces the search space for the subsequent MIL algorithm and helps improve classification performance. We trained a YOLOv8 [25] model for mouth detection using a pilot dataset in which mouth regions were manually marked. We apply the mouth detector to all the images after filtering out the OSF images. The detected mouth regions are then cropped out and resized to size 224 × 224 (pixels) to be used for the following cancer-precancer classification.

C. Oral Precancer and Cancer Classification using MIL

As previously mentioned, only the coarse-grained annotations (i.e., patient-level class label) are provided for the images (and lesions). The fine-grained annotations (i.e., image-level class label, lesion-level class label, and lesion boundary marking) are unavailable. We apply the MIL network that is adapted from [26]. It consists of four main components: 1) the pretrained image encoder for extracting features from each image of the patient; 2) the classifier for predicting the class probability of each image; 3) the attention network for outputting the attention weight for each image; and, 4) the predictor for aggregating the predictions of all images and generating the final prediction score and class label for the patient. The image encoder uses an ImageNet pretrained model to output a feature vector and an embedding layer to reduce the feature vector to a lower dimension. The image (instance) classifier generates a prediction probability for each image embedding feature. The attention network applies the attention mechanism in [27] to all the image embedding features and learns the weights for images. The final predictor computes the weighted average of image prediction probabilities using the learned weights outputted from the attention network to obtain the patient-level prediction. The network is trained end-to-end. We use this relatively simple MIL network as a starting point for the presented proof-of-concept (PoC) work, and other architectures may be explored and studied in the future.

We employ a balanced sampling approach via class-wise data augmentation in each batch since the dataset is unbalanced with the number of precancer samples significantly higher than that of cancer samples. Conventional image augmentation methods are applied for increasing the number of training images and data balancing. More details about the MIL algorithm, such as architecture, hyperparameter setting, augmentation methods, and training strategies, are discussed in Section III.

D. Evaluation Metrics

The metrics we use for assessing the performance of classification (multiclass or binary) include sensitivity (recall), specificity, precision, Youden’s index (J), balanced accuracy, F1 score (F1), and Matthews correlation coefficient (MCC) ((1)-(7)). The metrics used to evaluate the performance of mouth region detection are mean average precision (mAP) calculated at an intersection over union (IoU) threshold of 0.50 (mAP50) and at a series of IoU thresholds ranging from 0.50 to 0.95 (mAP50–95).

$$\mathit{\text{Sensitivity}}=\mathit{\text{recall}}=\frac{\mathit{\text{TP}}}{\mathit{\text{TP}}+\mathit{\text{FN}}}$$ (1)

$$\mathit{\text{Specificity}}=\frac{\mathit{\text{TN}}}{\mathit{\text{TN}}+\mathit{\text{FP}}}$$ (2)

$$\mathit{\text{Precision}}=\frac{\mathit{\text{TP}}}{\mathit{\text{TP}}+\mathit{\text{FP}}}$$ (3)

$$J=\mathit{\text{sensitivity}}+\mathit{\text{specificity}}-1$$ (4)

$$\mathit{\text{Balanced}}\mathit{\text{Accuracy}}=\frac{\mathit{\text{senstivity}}+\mathit{\text{specificity}}}{2}$$ (5)

$$F_1=2\times \frac{\mathit{\text{precision}}\times \mathit{\text{recall}}}{\mathit{\text{precision}}+\mathit{\text{recall}}}$$ (6)

$$MCC=\frac{TP\times TN-FP\times FN}{\sqrt{\left(TP+FP\right)\times \left(TP+FN\right)\times \left(TN+FP\right)\times (TN+FN)}}$$ (7)

Here, TP, TN, FP, and FN denote true positives, true negatives, false positives, and false negatives, respectively.

III. Experimental Results and Discussions

A. Evaluations of the Multifield Classifier and the Mouth Detector

The oral images are preprocessed to exclude OSF images and non-mouth areas before being fed to the MIL classifier. Since we have not reported our work on the 7-class multifield classification and mouth detection before, for the completeness of the current study, we briefly present the experimental evaluation results of these two preprocessing steps. They were conducted on two sets of images that are different from the dataset used for this work on precancer and cancer classification, respectively.

Table I lists the number of images used to train, validate, and test the 7-class ResNeSt model for multifield classification (the data is split at the patient level). The multiclass classification performance of the model on the test set is shown in Table II. The average sensitivity, specificity, precision, F1 score, and MCC of all classes are 0.879, 0.981, 0.880, 0.879, and 0.860, respectively. We also calculate the metrics for separating OSF images from non-OSF images since in the current study, we use the classifier for identifying OSF image only. The binary classification achieves a very high performance (i.e., F1 score and MCC values both are 0.997). Fig. 2 shows the corresponding confusion matrix, which indicates that only two images in the test set are misclassified with respect to the OSF class.

TABLE I.

Number of images in each set for 7-class multifield classifior development

	RB	LB	TM	DT	VT	FM	OSF
Train	6106	6134	4849	6657	3645	3712	1601
Val.	1254	1314	1049	1479	850	785	313
Test	1255	1338	1016	1463	761	817	388

TABLE II.

Multifield classification performance on the test set

	Sens.	Spec.	Precision	F1	MCC
7-class	0.879	0.981	0.880	0.879	0.860
OSF binary	1.000	0.9997	0.995	0.997	0.997

Fig. 2.

Multifield classification confusion matrix on the test set.

Table III lists the number of patients/images that were used to develop the mouth detector. They were collected from one hospital in a small pilot study. Fig. 3 displays the loss values and performance metric values on the training and validation sets with respect to the increase in the epoch number during the training of the detection model (YOLOv8), respectively. The detection performance of the model on the test set is evaluated using mAP50 and mAP50–95, and their values are 0.995 and 0.959, respectively. Several examples of mouth region detection results are shown in Fig. 4.

TABLE III.

Image dataset for mouth detector development

	Train	Val	Test
# of patients	57	8	16
# of images	945	86	265

Fig. 3.

The training and validation performance of the mouth detection model with respect to the increase of training epoch number.

Fig 4.

Examples of mouth detection results.

B. Dataset for Precancer-Cancer Classification

Our main goal is to separate cancer from precancer based on the visual characteristics of lesions shown in oral images using only patient-level class labels. The dataset used for the experimental tests consists of 1385 patients who each have only one lesion. Among them, 317 have oral cancer, and 1068 have oral precancer. We randomly split the dataset at patient level into training, validation, and test sets with the ratio of 70:15:15, respectively. The split is stratified within each class. The number of patients in each class and in each set is shown in Table V, respectively. All images are pre-processed for OSF image filtering and mouth detection.

TABLE V.

Number of patients in each set

	Train	Val	Test	All
Precancer	748	160	160	1068
Cancer	222	48	47	317
All	970	208	207	1385

C. Experimental Setup for Precancer-Cancer MIL classifier

The backbone of the MIL network is an ImageNet-pretrained ResNet18 model, which is used to extract the CNN feature of an input image. The extracted ResNet18 feature is fed into an embedding layer to reduce its size to 128. The activation function for the embedding layer is a ReLU. Between the embedding layer and the image classifier, a dropout layer (dropout rate of 0.2) is added. The image classifier consists of a fully connected layer followed by a SoftMax layer to output the prediction probability. The attention network pools all the embedding features and learns the weight for each image. The patient-level prediction is obtained by the weighted sum of the prediction probabilities of all the images. During training, the CNN backbone is frozen while the top-layers are fine-tuned. We train the model using binary cross-entropy loss and the Adam optimizer with a batch size of 32 for 20 epochs. Both the learning rate and weight decay are set to 0.0001. We apply standard normalization and augmentations, including random horizontal and vertical flips and rotations.

D. Results of Precancer-Cancer Classification with MIL

The confusion matrix for the test set is illustrated in Fig. 5. The values of sensitivity, specificity, J, balanced accuracy, F1, and MCC obtained are 0.660, 0.931, 0.591, 0.795, 0.697, and 0.616, respectively, as shown in Table V. The corresponding ROC curve with an AUC value of 0.91 is shown in Fig. 6.

Fig. 5.

Test set confusion matrix.

Fig. 6.

ROC curve on the test set

As a PoC work, the MIL classification performance demonstrated by Fig.5, Fig. 6, and Table V, is promising. Although it is lower than what has been reported by some other computer-aided oral cancer visual evaluation studies in the literature [6,7], we hypothesize that it may be due to the following differences between our work and the existing studies. Different from the studies that are to separate cancerous oral images from normal oral mucosa images, we conduct experiments for the case of separating patients with oral precancer from patients with oral cancer. Majority of the existing studies use visual evaluation decisions provided by medical specialists as ground truth, while in our study, the ground truth is based on biopsy results. In addition, existing studies have fine-level (lesion-level or image-level) labeling information and utilized strongly supervised learning, while we explore weakly supervised learning in which only coarse-level (patient-level) labels are available.

E. Limitations

In the future, we will aim to address the following aspects to further evaluate and improve the proposed method. The value of sensitivity achieved by our MIL classifier on the test set is significantly lower than that of the specificity (0.660 vs 0.931). Dataset imbalance could be a contributing factor, although we have applied a balanced data sampling method and conventional data augmentation. We will explore image synthesis to generate more images of cancer and use synthesized images to balance the dataset. Another limitation of our work is that the patient-level categories do not include control (normal and benign). While some of the images of these patients of oral precancer and oral cancer may not contain precancerous or cancerous lesions, these images are forced to be predicted with respect to the two classes of precancer and cancer at the instance level in MIL, although the corresponding prediction score should indicate high uncertainty of the instance level classification. In the future, we will examine three-class classification by adding control samples. In addition, we need to examine model explainability and compare with strongly supervised learning after collecting lesion annotations. Nonetheless, based on our knowledge, our study is the first attempt to explore weakly supervised learning for oral photograph evaluation, and the experimental results demonstrate the feasibility and promise of the proposed approach.

IV. Conclusions

Oral cancer is a common cancer affecting many people worldwide. It is often detected late, resulting in low survival rate. In this study, we present our effort towards utilizing AI/ML methods for assisting and improving visual evaluation in oral cancer screening that could lead to early detection. Specifically, we aim to explore weakly supervised learning to classify and analyze images of the oral cavity. Our strategy consider the scenario that the ground truth label is available only at the patient level but not at the image and lesion levels. The proposed method consists of three main steps: 1) identifying and removing the type of images containing fingers next to the mouth region, which usually occlude the mouth; 2) detecting the mouth region in the oral images and cropping it out; 3) classifying patients of precancer and cancer using MIL on their mouth region images. We evaluate and report the performance at each step of our systematic analysis. The image filtering and mouth detection steps achieve high performance. Promising results on the precancer-cancer classification task are also observed. We discuss the limitations of this work, and in the future, we will focus on four research tasks: 1) adding normal class and conducting additional experiments with more data; 2) collecting annotations on lesions and developing algorithms on lesion detection and lesion classification; 3) exploring generative AI for synthesizing oral lesions or oral images to balance the dataset; and 4) integrating information from both images and non-image sources and investigating multimodal classification.

TABLE IV.

Mouth detection performance on the test set

mAP50	mAP50-95
0.995	0.959

TABLE VI.

MIL classification performance on the patients of the test set

Sens.	Spec.	Youden	BAcc	F1	MCC
0.660	0.931	0.591	0.795	0.697	0.616