Deep learning model for the automated evaluation of contact between the lower third molar and inferior alveolar nerve on panoramic radiography

Abstract

Background/purpose

In lower third molar (LM3) surgery, panoramic radiography (PAN) is important for the initial assessment of the anatomical association between LM3 and the inferior alveolar nerve (IAN). This study aimed to develop a deep learning model for the automated evaluation of the LM3–IAN association on PAN. Further, its performance was compared with that of oral surgeons using original and external datasets.

Materials and methods

In total, 579 panoramic images of LM3 from 384 patients in the original dataset were utilized. The images were divided into 483 images for the training dataset and 96 for the testing dataset at a ratio of 83:17. The external dataset comprising 58 images from an independent institution was used for testing only. The LM3–IAN associations on PAN were categorized into direct or indirect contact based on cone-beam computed tomography (CBCT). The You Only Look Once (YOLO) version 3 algorithm, a fast object detection system, was applied. To increase the amount of training data for deep learning, PAN images were augmented using the rotation and flip techniques.

Results

The final YOLO model had high accuracy (0.894 in the original dataset and 0.927 in the external dataset), recall (0.925, 0.919), precision (0.891, 0.971), and f1-score (0.908, 0.944). Meanwhile, oral surgeons had lower accuracy (0.628, 0.615), recall (0.821, 0.497), precision (0.607, 0.876), and f1-score (0.698, 0.634).

Conclusion

The YOLO-driven deep learning model can help oral surgeons in the decision-making process of applying additional CBCT to confirm the LM3–IAN association based on PAN images.

Introduction

Lower third molar (LM3) surgery can carry intractable complications, especially neurosensory disturbance caused by inferior alveolar nerve injury (IANI), with an incidence of <8.4%.1, 2, 3 This sequela often compromises quality of life^2,4 and has possible medicolegal ramifications. To predict the risk of IANI, clinical studies have investigated its risk factors mainly via imaging.⁵ The proximity of LM3 to the inferior alveolar canal (IAC) is often evaluated via panoramic radiography (PAN), which is a conventional modality,⁶ with or without cone-beam computed tomography (CBCT). The removal of LM3 with dental roots that are in direct contact with the inferior alveolar nerve (IAN) (i.e., without bony cortex between LM3 and IAN) is associated with IANI due to the increased probability of compressive load on the IAN during manipulation using an elevator.⁷

PAN is useful in preoperative anatomic and morphologic assessment in most cases of LM3 surgery. However, this modality cannot accurately differentiate direct contact between the LM3 roots and IAN (direct LM3–IAN contact) due to anatomical noise and geometric distortion effect in two-dimensional imaging. CBCT is commonly applied to visualize three-dimensional associations based on high spatial resolution and low radiation dosage. In addition to the cortication defect of the IAC on LM3s,⁸ direct contact with multiple roots,⁹ longer contact length,¹⁰ and dumbbell-shaped compression of the decorticated IAC¹¹ are reliable predictors of IANI. However, the routine use of CBCT is not recommended except in cases with clear clinical reservations after PAN.¹² On PAN images, clinicians should determine the direct LM3–IAN contact that requires the application of additional CBCT.

Deep learning architectures based on artificial intelligence (AI) have been reported in a diversity of healthcare systems, including the dentistry field.^13,14 AI has potential and helpful qualities that can improve the interpretation of imaging datasets for diagnosing dental conditions, such as dental caries,¹⁵ periodontal disease,¹⁶ mandibular fractures,¹⁷ and maxillary sinusitis.¹⁸ Regarding LM3 on PAN images, deep learning models have been applied for its detection,¹⁹ development staging,²⁰ prediction of eruption,²¹ evaluation of extraction difficulties,²² risk of IANI,²³ and the anatomical LM3–IAN association.24, 25, 26, 27

Unlike the conventional deep learning methods with two-stage object detection, such as region-based convolution neural network (R-CNN), You Only Look Once (YOLO) can simultaneously handle the detection and classification of target images in one framework.²⁸ Although YOLO is useful in the identification of radiolucent lesions of the jaw on PAN images,^29,30 studies on its application in imaging studies prior to LM3 surgery are limited. Recently, Zhu et al.²⁶ showed the automated detection of contact between LM3s and IAN. Nevertheless, an external validation was not performed. The current study developed a deep learning model using YOLO for automatically evaluating the associations between the LM3 roots and IAN. Further, its performance was compared with that of oral surgeons using original and external datasets to enhance generalizability.

Materials and methods

Data collection

This study was approved by the internal review boards of our institution and was conducted according to the principles of the Declaration of Helsinki. We collected PAN and CBCT images from 384 patients who underwent LM3 surgery from October 2010 to May 2020 at Osaka University Dental Hospital as the original dataset and from 58 patients from January 2015 to March 2021 at Ikeda City Hospital as the external dataset. PAN images were acquired from the former institution using a Hyper-X unit (Asahi Roentgen Ind. Co., Ltd., Kyoto, Japan) with exposure parameters of 64 kV and 8 mA, exposure time of 12 s, and pixel size of 96 mm, and from the latter institution using AUGE SOLIO Z CM (Asahi Roentgen Ind. Co., Ltd.) with exposure parameters of 72 kV and 12 mA, exposure time of 12 s, and pixel size of 120 mm.³¹

Patients whose LM3 were localized with proximity to IAC on PAN and those with abnormal root shape features were candidates for preoperative CBCT. The associations between LM3s and IAC were categorized into the contact and intact groups based on the absence and presence of a cortical wall of the IAC on the LM3 on coronal CBCT images, respectively.⁹

Sample preparation

In total, 579 images from 384 patients in the original dataset were randomly separated into the training (83%, 483 images with 266 contact and 217 intact LM3s) and testing (17%, 96 images with 54 contact and 42 intact LM3s) datasets (Fig. 1). Further, 58 images from the external dataset included 39 contact and 19 intact LM3s. All images were JPEG files that were resized to 320 × 320 pixels. To augment the amount of training data for deep learning, PAN images were flipped horizontally and rotated by 5° and −5°, resulting in four types of AIs: type I, trained with the original set with 483 images; type II, trained with the dataset for type I and the 5° rotation image augmentation set; type III, trained with the dataset for type II and the flip-image augmentation set; and type IV, trained with the dataset for type III and the −5° rotation image augmentation set. The best model was used as the final AI model for testing.

Figure 1.

Sequence of steps in the deep learning models. Of 579 lower third molars in the original dataset, 483 and 96 were used for the AI model training and testing, respectively. The type I image set comprised the original images only. Meanwhile, the type II, III, and IV image sets were added to augment images with 5° rotated images, 5° rotated and horizontally flipped images, and ±5° rotated and flipped images, respectively. After cropping images and annotating based on CBCT data, the AI models were constructed using YOLO version 3 (YOLOv3).²⁸ In total, 94 (97.9%) of 96 images in the original dataset and 55 (94.8%) of 58 images in the external dataset were detected using the final AI model. These images were used for performance evaluation.

Each image was labeled manually by a drawing rectangular bounding box focusing around the LM3 using labeling and annotated to contact or intact. We used the YOLO version 3 (YOLOv3) algorithm,²⁸ an object detection application system, with Darknet53.cov74 feature extractor, which comprises 53 convolutional layers (Fig. 2). Down-sampling the input images at different levels and extracting their features can predict bounding boxes at three scales. Up-sampled layers have fine characteristics, thereby improving small object detection. Instead, of softmax function, cross-entropy loss and logistic regression were used for class predictions in each bounding box.

Figure 2.

YOLOv3 architecture. YOLOv3 predicts three three-dimensional tensors corresponding to different scales for each training data at the predicted time.

Evaluation of performance

The performance of the best AI model was compared with that of five oral surgeons from our department who have at least one year of experience in LM3 surgery. After excluding undetectable images using the AI model, the evaluation of LM3 images for testing was performed in the original and external datasets. If the LM3–IAN association was classified accurately, it was assigned as true positive (TP) or true negative (TN). When an evaluation result incorrectly indicated the LM3–IAN contact, it was labeled as false positive (FP). The opposite error was labeled as false negative (FN). Classification performance was evaluated based on the following metrics: accuracy, recall (sensitivity), precision, and f1-score.

Accuracy = (TP + TN) / (TP + TN + FP + FN).

Recall (sensitivity) = TP / (TP + FN).

Precision = TP / (TP + FP).

F1-score = 2 × precision × recall / (precision + recall).

Results

The performance of four AIs based on the learning steps from type I to type IV was evaluated (Table 1). With the learning steps of data augmentation, the detection rate, accuracy, precision, and f1-score, but not recall, gradually improved. We used type IV as the final model, which detected 94 (97.9%) of 96 images with 53 contact and 41 intact LM3s. The accuracy, recall, precision, and f1-score were 0.894, 0.925, 0.891, and 0.908, respectively (Table 2). The average evaluation metrics of the oral surgeons were as follows: accuracy, 0.628; recall, 0.821; precision, 0.607; and f1-score, 0.698.

Table 1.

Performance of deep learning models with or without data augmentation.

Deep learning model	Data augmentation	Detection rate	Accuracy	Recall	Precision	F1-score
Type I	None	0.656	0.469	0.972	0.673	0.795
Type II	+5° rotation	0.917	0.761	0.833	0.755	0.792
Type III	+5° rotation, horizontal flip	0.896	0.779	0.851	0.769	0.808
Type IV (final)	±5° rotation, horizontal flip	0.979	0.894	0.925	0.891	0.908

Type I, trained with the original set with 483 images of the lower third molars; Type II, trained with the dataset for Type I and the 5° rotation image augmentation set; Type III, trained with the dataset for Type II and the flip-image augmentation set; Type IV, trained with the dataset for type III and the −5° rotation image augmentation set.

Table 2.

Comparison of performance between the final deep learning model and oral surgeons.

	94 images of the original dataset Osaka University				55 images of the external dataset Ikeda City Hospital
	Accuracy	Recall	Precision	F1-score	Accuracy	Recall	Precision	F1-score
OS. A	0.606	0.771	0.587	0.667	0.527	0.486	0.720	0.580
OS. B	0.745	0.878	0.705	0.782	0.618	0.486	0.900	0.631
OS. C	0.564	0.880	0.557	0.682	0.600	0.459	0.895	0.607
OS. D	0.511	0.735	0.522	0.610	0.600	0.432	0.941	0.592
OS. E	0.713	0.840	0.689	0.757	0.727	0.622	0.958	0.754
OS. average	0.628	0.821	0.607	0.698	0.615	0.497	0.876	0.634
Final deep learning model	0.894	0.925	0.891	0.908	0.927	0.919	0.971	0.944

OS, oral surgeon.

Accuracy = [true positive (TP) + true negative (TN)] / [TP + TN + false positive (FP) + false negative (FN)].

Recall (sensitivity) = TP / (TP + FN).

Precision = TP / (TP + FP).

F1-score = 2 × precision × recall / (precision + recall).

In the external dataset, the final AI model detected 55 (94.8%) of 58 images with 37 contact and 18 intact LM3s. Its accuracy, recall, precision, and f1-score were 0.927, 0.919, 0.971, and 0.944, respectively. The accuracy, recall, precision, and f1-score of oral surgeons were 0.615, 0.497, 0.876, and 0.634, respectively. Hence, the metrics of the model were higher than those of oral surgeons.

Discussion

The development of IANI during LM3 surgery is primarily dependent on the anatomical LM3–IAN associations.⁵ IAN in close contact with LM3s can be at risk of intraoperative exposure; thus, IANI can more likely to develop,³² with approximately 20% increase in probability.^33,34 Based on the current research, a successful improvement in the model's f1-score via data augmentation showed that it can be used to brush up the models with the highest stability. The final model performed better than oral surgeons. In the independent dataset on a different PAN system with different exposure conditions, the outperforming performance indicated the generalization ability and robustness of the YOLO model. With the clinical use of the deep learning model, unnecessary radiation exposure of CBCT could be prevented if it is applied to determine direct LM3–IAC contact on PAN.

The cortication defect of IAC on the LM3 root is a reliable predictor of IANI.⁸ Rood and Shehab⁶ presented seven PAN signs, including four root-related and three IAC-related signs to assess the LM3–IAN association correlated with IANI during LM3 surgery. Of those signs, the interruption of the white line^35,36 and canal diversion³⁷ are predictors of direct LM3–IAC contact. The concomitant presence of two or more of these signs were highly predictive of direct LM3–IAN contact.³⁸ However, evaluating the contact on two-dimensional imaging based on human eyes can have an uncertain performance. In our previous study on IANI risk in PAN-featured models constructed via conventional logistic regression analysis,³¹ the model that includes variables associated with the Rood's signs had unreliable metric values in the external validation. However, it has an acceptable performance in the original sample. This instability was attributed to difficulties in the identification of Rood's signs on PAN images under different settings using the human eyes.

The performance of oral surgeons had relatively high recall and low precision in the original dataset and low recall and high precision in the external dataset. Considering that the oral surgeons’ criteria for determining the LM3–IAN contact on PAN images were invariant between the two tests, this trade-off status of recall and precision may be explained by the difference in the proportion of contact LM3 (positive cases) between the two datasets. The higher proportion of contact cases led to increased precision score (positive predictive value) in the external dataset from Ikeda City Hospital (37 contact of 55 LM3s, 67%), where additional CBCT after PAN might have been more strictly applied than in Osaka University (53 contact of 94 LM3s, 56%). Nevertheless, the AI model showed highly stable performance in both tests using the original and external datasets; however, there were a few undetectable cases. Taken together, the deep learning system could accelerate the initial investigation by helping in decision-making to additional CBCT based on PAN images; however, it cannot completely replace the human eyes.

The deep learning systems of the ResNet50 have been shown to determine the contact between LM3s and IAC.²⁴ Among these models, ResNet50v2 with SAM optimization was the best, with an accuracy of 0.866, precision of 0.816, recall of 0.791, and f1-score of 0.800. Because the image detection-based ResNet50 cannot display the bounding box, it is unclear whether the LM3s are recognizable visually. Meanwhile, the YOLOv3, an object detection system, had better performance in all the evaluation metrics of this study. Zhu et al.²⁶ has recently shown that the performance of YOLOv4 was similar to that of our AI model in determining the LM3–IAC contact with a recall of 0.917, precision of 0.887, and f1-score of 0.902. It is more advantageous to visualize the recognition of LM3s and evaluation status of the LM3–IAC associations on the monitor screen in the clinical settings, thereby indicating the feasibility of the YOLO-driven system for clinical use.

The current study had some limitations. First, we only utilized YOLO, and it was not compared with other algorithms, such as Faster R-CNN, two-stage detection system. However, Celik¹⁹ has shown that YOLOv3 has an excellent performance in LM3 detection on PAN (0.96 of mAP@0.5), with metric values superior to those of ResNet50 (0.91), VGG16 (0.87), and AlexNet (0.86). Second, the current model classified the presence or absence of direct LM3–IAN contact alone. Further analysis of combined IANI predictors, such as bucco-lingual positioning and degree of IAC compression, based on optimized data should be performed to automatically evaluate the risk of LM3 surgery with the YOLO-driven model in real-world clinical settings.

In conclusion, the YOLO-based AI model with data augmentation for determining the anatomical LM3–IAN association of direct contact on PAN images had a predominantly superior performance to that of oral surgeons in the original and external datasets. Hence, in the future, the deep learning technology can be utilized as a clinical decision-making tool to help oral surgeons perform additional CBCT to assess direct LM3–IAN contact in cases of LM3 roots overlapping the IAC on PAN images.

Declaration of competing interest

The authors have no conflicts of interest relevant to this article.