Deep learning for automated detection and numbering of permanent teeth on panoramic images

Abstract

Objective:

This study aimed to evaluate an automated detection system to detect and classify permanent teeth on orthopantomogram (OPG) images using convolutional neural networks (CNNs).

Methods:

In total, 591 digital OPGs were collected from patients older than 18 years. Three qualified dentists performed individual teeth labelling on images to generate the ground truth annotations. A three-step procedure, relying upon CNNs, was proposed for automated detection and classification of teeth. Firstly, U-Net, a type of CNN, performed preliminary segmentation of tooth regions or detecting regions of interest (ROIs) on panoramic images. Secondly, the Faster R-CNN, an advanced object detection architecture, identified each tooth within the ROI determined by the U-Net. Thirdly, VGG-16 architecture classified each tooth into 32 categories, and a tooth number was assigned. A total of 17,135 teeth cropped from 591 radiographs were used to train and validate the tooth detection and tooth numbering modules. 90% of OPG images were used for training, and the remaining 10% were used for validation. 10-folds cross-validation was performed for measuring the performance. The intersection over union (IoU), F1 score, precision, and recall (i.e. sensitivity) were used as metrics to evaluate the performance of resultant CNNs.

Results:

The ROI detection module had an IoU of 0.70. The tooth detection module achieved a recall of 0.99 and a precision of 0.99. The tooth numbering module had a recall, precision and F1 score of 0.98.

Conclusion:

The resultant automated method achieved high performance for automated tooth detection and numbering from OPG images. Deep learning can be helpful in the automatic filing of dental charts in general dentistry and forensic medicine.

Introduction

Orthopantomogram (OPG) is the most commonly used imaging modality in dentistry. OPG has several advantages over other radiographic modalities as it illustrates the maxillofacial bones and teeth, uses low radiation, and is inexpensive and fast.¹ OPG is routinely used to evaluate all stages of dentition and occlusion, diagnosis and treatment planning, evaluation of dentoalveolar trauma, and dental age estimation.^2,3 To achieve this, teeth must be correctly identified and numbered per the Fédération Dentaire Internationale (FDI) tooth numbering system.⁴ Dental charting or tooth numbering is an integral part of routine clinical practice and has implications in forensic investigations.⁵ Although all dentists are well-trained in reviewing OPGs, some factors, such as 2D visualisation, variations in contrast, noise, and the proximity of adjacent tooth structures, can influence the dentists’ interpretations.³ This is coupled with the substantial time needed for the recording of information and filling out dental charts. Thus, the current practice faces several shortcomings impacting clinical service efficiency and the daily practice workflow.

Recently, there has been an increased interest in using machine learning, particularly deep learning to accomplish various tasks such as image classification, object detection and segmentation.⁶ This method outperforms other mathematical approaches used for automated segmentation. Deep learning has recently been used in medical imaging to assist clinicians in detecting and localising diseases such as breast cancer, colon cancer, lung diseases, eye diseases, trauma and orthopaedics.^7–11 Deep learning holds great promise in dentistry as it enables timely and accurate extraction of clinical data from a large data set of images that is not possible by human eyes. At the same time, it improves reproducibility through objective assessments. This would directly enhance the efficiency of clinical services by supporting dentists’ clinical decisions and reducing the time allocated for interpreting radiographs.

Compared to traditional rule-based machine learning methods that still rely on handcrafted feature engineering to analyse images, deep learning can learn those features directly from raw data input (e.g. pixels of images). Recently, convolutional neural networks (CNNs), a special type of deep learning, has outperformed traditional rule-based machine learning and has gained significant attention to solve several computer vision tasks, including object detection and segmentation in medical images.^6,12 In particular, the Faster region-based CNNs (Faster R-CNNs) offer a robust framework for detecting regions of interest (ROIs).¹³ The Faster R-CNN is an improvement over the Fast R-CNN architecture, originally evolved from the Region-based CNN (R-CNN) method. The Faster R-CNN improves the R-CNN performance by simplifying the pipeline and optimising computation. To date, a handful of studies have explored the capability of the Faster R-CNNs for object detection in dentistry, in particular automated tooth detection.^14,15 For instance, Tuzoff’s et al yielded a sensitivity (i.e. recall) of 0.99 for permanent tooth numbering in a set of 1352 images.¹⁴ Another study by Kilic et al reported a sensitivity of 0.98 in deciduous tooth numbering in a set of 421 images.¹⁵

The Faster R-CNN is excellent for object detection, but some teeth may be missed if used alone without detecting ROIs. Against this background, in the present study, we proposed a three-step procedure whereby U-Net based segmentation of tooth regions or ROIs was applied prior to using the Faster R-CNN for detecting individual teeth on images.^13,16 This approach would allow for much better tooth detection performance. The current study evaluated a refined, fully deep learning algorithm that uses CNN architectures for automated tooth numbering on OPG images for adult individuals.

Methods

The present study developed and evaluated a deep learning system to detect teeth and tooth numbers on digital OPG images. This study was reported as per the Standard for Reporting of Diagnostic Accuracy Studies (STARD) guideline¹⁷ and the Checklist for Artificial Intelligence in Medical Imaging.¹⁸ Human research ethics approval was sought from the CSIRO HREC (Ref no. 2020–044-LR).

Radiographic data set

A total of 591 OPG radiographs for adult individuals were collected from collaborating dental clinics in Perth, Australia, between June 2020 and October 2020. All images were anonymous and imported in PNG format. All images were screened for quality, contrast, and orientation. Those images with very poor quality or demonstrating mixed dentition, heavy restorations, bridges, implants, or many metal crowns were excluded from the data set.

Data set’s annotation and labelling

An in-house image annotation platform (SIANNO) was used for creating ground truth annotations for images. Three qualified dentists with over 10 years of clinical experience participated in the image review. Two dentists independently reviewed the radiographs and asked to create a bounding box for each tooth and assigned a tooth number as per the FDI tooth numbering system (11-12-13-14-15-16-17-18-21-22-23-24-25-26-27-28-31-32-33-34-35-36-37-38-41-42-43-44-45-46-47-48).⁴ The third dentist independently reviewed the images (46 out of 591 images), where discrepancies between the two dentists existed, and a final decision was made. The ground truth annotations for each tooth, which included coordinates of the ROIs, were obtained in an excel sheet for each item of the imaging data.

Out of 591, 248 OPGs (42%) had all 32 teeth were present, while 343 OPGs (58%) had at least one tooth missing. The most frequently missing teeth were as the following: tooth no 28 (n = 418); tooth no 18 (n = 419); tooth no 38 (n = 428); and tooth no 48 (n = 437).

Proposed deep CNN architecture

The proposed system was inspired by Tuzoff et al work.¹⁴ However, we proposed performing a U-Net-based segmentation of teeth regions before performing the Faster R-CNN-based detection of individual teeth different from Tuzoff’s approach.^13,16 In addition to that, we eliminated the heuristic method following VGG-16-based tooth classification.¹⁹ OPGs were used as input to the system. The ROI detection module processed the image to detect the teeth regions. We experimentally found the teeth regions are about one-third of the image. Thus, processing the teeth regions saved not only unnecessary computation but also improved the tooth detection performance. The tooth detection module identified individual teeth on the image and placed a bounding box around each tooth. The region inside each bounding box was cropped and then passed to the classification module. Finally, the classification module classified each cropped region, and a tooth number was assigned. We acknowledge that the proposed approach is not novel from the perspective of the used deep learning models, since our approach relied on readily available CNNs, namely U-Net, Faster R-CNN and VGG-16 networks. However, what made the proposed approach novel was how these three CNN networks were combined to better address the challenges of our particular research problem.

Both training and validation data sets included OPG images with and without missing teeth. 90% of the data set was used for training, and the remaining 10% were used for validation. Because the data set was limited, 10-folds cross-validation was performed, which is a common technique used in deep learning and measuring performance.²⁰ The whole data set (n = 591) was divided into 10 equal subsets. One subset was served as the validation set while the others remained as the training set. Then the model was learned using the training set, and the performance of the model was evaluated using the validation set. This process was iterated 10 times, substituting the validation set each time. The mean of the 10 performances being assessed was deemed the final performance.^20,21 The overall architecture and workflow of the deep CNN system are presented in Figure 1.

Figure 1.

System architecture and pipeline: the system consists of three CNN-based modules - ROIs detection, tooth detection and tooth classification tasks. CNN, convolutional neural network; ROI, region of interest

The training was performed on a PC equipped with Intel^® Xeon^® 3.60 GHz CPU, 64 GB RAM and NVIDIA GeForce RTX 2080Ti GPU. The dimension of original data sets ranged from 1899 × 901 to 4561 × 2565. Before training, data augmentation was performed, including height and width shift (in the range of 0.2), rotation (in the range of 15 degrees), and scaling (in the range of 0.2).

Region of interest detection

Detection of the ROI was performed relying upon U-Net.¹⁶ U-Net is a type of CNN that performs semantic segmentation of images. U-Net converted an image to vectors used for classifying pixels and then converted those vectors back to an image for segmentation of the classified regions. U‐Net used a contraction path of convolutional blocks (encoders) followed by an expansive path of convolutional blocks (decoders). In the contraction path, the size of the image decreased step by step while the depth steadily increased. In the expansion path, the image size increased gradually while the depth diminishes continuously. The contextual features and their hierarchy were trained by the encoders, whereas the decoder performed the semantic segmentation. Dice coefficient was used at the metric, and binary cross-entropy was used as the loss function while training the CNN. We used an adaptive moment estimation (ADAM) optimiser compared to the original U‐Net, which used the classical stochastic gradient descent optimisation approach to update network weights.^22,23 We generated the bounding box containing the full set of teeth from the semantic segmentation and cropped the region inside the bounding box.

For each OPG, a binary mask image was programmatically generated. All the pixels inside bounding boxes were set to one, and outside pixels were set to zero to generate the binary mask. A total of 591 binary mask image was generated. OPG images and their binary mask images, were used to train and test the ROI detection module. The processing time for using U-Net in ROI detection was about 0.7 s per OPG.

OPG images were resized to 256 × 256 for training the U‐Net. For any test image, it was resized to 256 × 256 prior to sending it to U‐Net. The output of the U-Net was then resized to reflect the original input size. This downsizing helped to improve the computational cost; however, it did not affect the ROI detection since we only needed a rough segmentation of the tooth regions.

Tooth detection

The tooth detection was performed using the Faster R-CNN, an object detection architecture proposed by Ren et al.¹³ The Faster R-CNN is an improvement of the Fast R-CNN architecture, originally evolved from the R-CNN method. It was developed to enhance the performance of R-CNN through pipeline simplification and computational optimisation. The Faster R-CNN is ideal for object detection, but some teeth may be missed if it is used alone without U-Net-based ROIs detection.

The Faster R-CNN consisted of two components, namely Regional Proposal Network (RPN) and object detector.²⁴ By taking an image as input, the RPN output a set of rectangular object proposals. Each object proposal had an associated abjectness score which was then assessed by the object detector module for localisation and classification of objects. The RPN and object detector modules in the Faster R-CNN shared a common set of convolutional layers to reduce computational cost. A feature map, a compact representation of the source image, was produced by common convolutional layers. RPN generated the regional proposals by sliding the window over the feature map. For each window location, bounding boxes (anchors) were then generated. For each anchor, the RPN calculated the probability of whether it contained an object or not (i.e. background). The bounding boxes were tightened using a special regressor. The top N-ranked region proposals were then employed as an input for the object detector. The object detector improved the class score of a region assigned as a tooth or a background before creating the final bounding box coordinates.¹⁴

A total of 591 OPG radiographs with 17,135 teeth were used to train and validate the tooth detection module. Images were resized to 640 × 480 prior to training the Faster R-CNN. For training, cropped images representing teeth were produced based on the ground truth annotations of OPG images. During the training process, the parameters of the CNN were initialised using transfer learning. Particularly, pre-trained models that were already trained using ‘ImageNet’,²⁵ a huge image data set, were used to initialise the network parameters. These parameters were, in turn, fine-tuned using our OPG data set. We fine-tuned the top 70% layers to avoid overfitting the model. The initial learning rate was set up at 0.001 with exponential decay.

Tooth numbering or classification

The tooth numbering module primarily relies upon the VGG-16 convolutional architecture.¹⁹ Among different CNN model architectures, VGG-16 is one of the simplest yet most frequently used models.²⁶ In VGG-16, there are 16 layers with learnable weights, including 13 convolutional layers and 3 fully connected layers. A ‘softmax’ classifier classified the data based on the features extracted from the last fully connected layer. The classifier outputs a confidence score representing the probability of the teeth belonging to 32 defined categories. A finally, a teeth number is assigned based on the confidence score.

Cropped images representing teeth and their numbers generated based on the ground truth annotations by experts were used to train the CNN. Likewise, the teeth detection module, model weights pre-trained on the ImageNet data set were utilised to initialise the VGG-16. A batch size of 64 was used to train the CNN. SGD optimiser was used with a learning rate and momentum of respectively 0.0001 and 0.9. Categorical cross-entropy was used as the loss function. We fine-tuned the top 70% layers of the VGG-16. A total of 17,135 teeth were grouped into 32 categories as per experts labelling and were used for training and validating the tooth numbering module. Teeth images were resized to 224 × 224 for training the VGG-16.

Performance measurement

Concerning the ROI detection task, for each OPG under consideration, we computed the IoU score between the binary mask image and the U-Net output (binarised). The IoU score between binary mask I, U-Net output F is mathematically expressed as the following:

,$${IoU}_{I,\mathrm{}F}\mathrm{}=\mathrm{}\frac{\left|I\mathrm{}\bigcap B\left(F\right)\right|}{\left|I\mathrm{}\bigcup B\left(F\right)\right|}$$

where $B\left(F\right)$ is the binary image of $F$, generated using Ostu’s method.²⁷

The ground truth and CNN annotations were considered to agree if they intersected substantially with the tooth detection task. The unmatched boxes consisted of two error types; false-positives where redundant boxes were annotated, and false-negatives where existed carious locations were missed. The recall (i.e. sensitivity) and precision (i.e. positive predictive value) were utilised to evaluate the tooth detection task performance and expressed mathematically as the following.

,$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{TP}{TP+FN}$$

.$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{TP}{TP+FP}$$

Here, TP, FP, FN represent true-positive, false-positive, and false-negative, respectively.

With regard to the tooth numbering task, the ground truth and CNN annotations were considered to agree if the class labels (32 teeth numbers) generated by the system matched with the labels supplied by the dental experts. For each tooth number (t_N), true positives (TP), true negatives (TN), false positives (FP), false negatives (FN) of t_N, were respectively defined as all t_N instances that are classified as t_N, all non- t_N instances that are not classified as t_N, all non- t_N instances that are classified as t_N, all t_N instances that are not classified as t_N. Along with the recall, precision defined above, specificity, accuracy and F1 score expressed below were used to measure the performance of the tooth numbering task.

,$$\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}=\frac{TN}{TN+FP}$$

,$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{TP+TN}{TP+TN+FP+FN}$$

.$$\mathrm{F}1\mathrm{}\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{}\times \mathrm{}\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{}+\mathrm{}\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}$$

Results

ROI detection module

The ROI detection module achieved an average IoU of 0.71 in predicting the overlap between the ROI detection and the ground truth annotations. Unsurprisingly U-Net outputs were not precise enough to identify individual tooth regions (which was also not the aim here). However, they were detailed enough to determine the tooth regions as a whole. Figure 2 illustrates an example input image to the U-Net model, the output produced by U-Net, and cropped the ROI.

Figure 2.

Example demonstration of ROI cropping by U-Net. (a) Input image. (b) Output produce by U-Net. (c) Cropped ROI. ROI, region of interest

Tooth detection module

The tooth detection module detected teeth with a recall and a precision of 0.992 and 0.994, respectively. When Tuzoff’s method (without U-Net based cropping) was employed, the tooth detection method achieved a recall and precision of 0.9566 and 0.9418, respectively (Table 1). Figure 3 demonstrated a particular example when the Faster R-CNN detected a non-tooth region as a tooth region when applied directly on the OPG radiograph. This error was resolved when U-Net based ROI cropping was performed before applying the Faster R-CNN. Figure 4 showed a particular example of when the Faster R-CNN failed to detect two teeth when applied directly on the OPG. This error was resolved when U-Net based ROI cropping was performed before using the Faster R-CNN. Figure 5 demonstrated a representative example of false positives and false negatives after running CNNs.

Table 1.

Results of tooth detection task

Metrics	Without U-Net based cropping14	Proposed method
True positives	1,652	1,713
False negatives	75	14
False positives	102	11
Recall	0.9566	0.9919
Precision	0.9418	0.9936

Figure 3.

Example demonstration of the tooth detection using the Faster R-CNN. (a) Without applying U-Net. (b) with U-Net based ROIs cropping before using the Faster R-CNN. CNN, convolutional neural network; ROI, region of interest.

Figure 4.

Example demonstration of the tooth detection using the Faster R-CNN. (a) Without applying U-Net. (b) with U-Net-based ROI cropping before applying the Faster R-CNN. CNN, convolutional neural network; ROI, region of interest.

Figure 5.

Example demonstration of false positives and false negatives. (a) A tooth that was not originally numbered as 25 but was assigned tooth number 25 by the CNN, (b) A tooth that was originally numbered as 33 but was wrongly numbered as 32 by CNN. CNN, convolutional neural network.

Tooth numbering module

The resultant tooth numbering module achieved a recall of 0.9803 and specificity of 0.9993 compared to the ground truth data. Furthermore, the estimated F1 score was 0.9803, indicating a good balance between the recall and precision. Thus, the proposed deep learning performed well in permanent tooth detection and numbering on OPG images. A summary of the results, including 95% CIs is presented in Table 2.

Table 2.

Results of tooth numbering task

Metrics	Mean (95% CI)
Recall	0.9803 (0.9754–0.9852)
Specificity	0.9993 (0.9983–1.00)
Accuracy	0.9986 (0.9972–1.00)
Precision	0.9800 (0.9759–0.9841)
F1 score	0.9803 (0.9758–0.9848)

CI, confidence interval.

Discussion

The present study proposed a three-step procedure to train and validate a deep learning system to improve tooth detection and tooth numbering performance. Our findings demonstrated that using U-Net to detect ROIs before the Faster R-CNN-based tooth detection task showed high-performance for detecting and numbering permanent teeth equivalent to the experts’ level (i.e. the ground truth data).

Our findings were consistent with Tuzoff et al results for the tooth numbering task; however, our tooth detection task performance was slightly better.¹⁴ Similarly, Kılıc et al applied the Faster R-CNN for detecting and numbering the deciduous teeth in 421 OPGs, reporting very high precision (0.957) and recall (0.980) rates.¹⁵ The high performance of our method is attributed to the use of U-Net to detect the ROIs prior to applying the Faster R-CNN for the tooth detection task. The Faster R-CNN is suitable for detecting small objects; however, there is still a minimum size limit that it could efficiently handle. Like other CNN architectures, the Faster R-CNN also resizes the original image to a predefined size. Since the tooth regions are about one-third of the original image, the tooth regions can be cropped from the original image and then resized instead of resizing the original image. This approach would increase the ratio of the object’s size concerning the image size, thus ensuring better detection of objects, which was evident in our experiment.

Results of the present study demonstrated a higher performance compared to previous studies that used different deep CNN networks. For instance, Oktay used a CNN with a modified version of the Alex.net architecture to perform multi class classification; the accuracy was over 90%.²⁸ Silva et al applied Mask Recurrent-CNN for segmentation in a set of 1,500 OPGs,²⁹ yielding a precision and recall of 84 and 76%, respectively. Lee et al developed the Mask R-CNN model to automatically detect and number teeth with a precision of 0.86 and recall of 0.89 in a set of 30 OPGs.³⁰ Another study by Leite et al performed a deep learning method using Mask R-CNN for tooth detection and segmentation in a set of 153 OPGs. Their method achieved a recall and precision of 98.9 and 99.6%, respectively.³¹ It is worth mentioning that even though the results are indicative, a direct comparison with previous studies may be challenging due to variations in data sets used.

The U-Net-based ROI detection task took about 0.7 s per image. Worth mentioning that the code was not optimised to ensure a highly efficient computational time, which was also not the focus of the current paper. A direct comparison of the processing time with previous studies was not possible because simply their exact codes were not available. While the proposed approach added 0.7 s of extra time by adding one step, it reduced the computational time by eliminating the heuristic method following VGG-16-based tooth classification.

The present study had some limitations. A relatively small training data set was used in the present study. External data sets were not used in this study as publicly annotated data sets were not available. Most training data sets were obtained from young people, as the inclusion criteria included OPGs for individuals over 18 years. Despite the likelihood of selection bias, using OPGs from young individuals were essential to ensure that the proposed deep learning system was trained for all groups of teeth and a large number of teeth. However, our data sets included low and high-density radiographs with minimal positioning imperfections, artefacts or orthodontic appliances, and radiographs for older individuals with missing teeth. Further validation studies using larger data sets where images with crowns, bridges, or implants have been suggested as future work. Nevertheless, the current study aimed to establish a sound methodology and generate preliminary results that can feed into further large validation studies before its early translation into clinical practice.

The charting of tooth numbers is a routine dental practise, and this process is even more difficult on OPG images because of its inherent limitations. Deep learning aims to shift the current practice where review/charting, archiving, and retrieval of images can become automated and integrated. So that clinicians can read radiographs faster, saving time for more complex cases and focus more on treating patients. Our deep learning system achieved excellent performance for detecting and numbering permanent teeth on OPGs, similar to a recently developed deep CNNs. This is the first step to detect not only teeth and their constituent parts but to detect missing teeth, dental caries, and maxillofacial problems. Therefore, this first step should be as accurate as possible.

The current study focused on the automated detection and classification of permanent teeth; thus, further studies are required to detect and classify deciduous or mixed dentition on OPGs. In addition, there is a high potential that the proposed deep learning can be applied for other imaging modalities (e.g. cone beam CT) and other disciplines to generate medical reports automatically. Thus, there is a need to improve the deep learning system’s output, including applying more advanced augmentation methods, extending the training data set, and using more advanced CNNs.

Conclusions

The proposed deep learning-based detection system achieved high recall and precision in tooth detection and tooth numbering on OPGs. Such findings can be helpful to increase the acceptance and adoption of artificial intelligence among dental practitioners. The use of deep learning in daily practice would directly impact the practice workflow and the efficiency of clinical services by saving dentists' time and reducing manual charting. However, further research is required to evaluate the translation of the deep learning system in routine clinical practice.