Application of deep learning in teeth identification tasks on panoramic radiographs

Abstract

Objectives:

To investigate the current developments of Artificial Intelligence (AI) in teeth identification on Panoramic Radiographs (PR). Our aim was to evaluate and compare the performances of Deep Learning (DL) models that have been employed in the execution of this task.

Methods:

The systematic review was registered on PROSPERO. All recent studies that utilized DL models for identifying teeth on PRs were included in this review. An extensive search of the medical electronic databases including PubMed NLM, EBSCO Dentistry & Oral Sciences Source, and Wiley Cochrane Library was conducted. This was followed by a hand search of the IEEE Xplore database. The diagnostic performance of DL models in teeth identification tasks on PR was the primary outcome assessed in this review. The risk of bias assessment of the included studies was evaluated via the modified QUADAS-2 tool. Owing to the heterogeneity of the reported performance metrics, a meta-analysis was not possible..

Results:

The search yielded a total of 282 articles, out of which 13 relevant ones were included in this review. These studies utilized a diverse range of DL models for teeth identification tasks on PRs and reported their performances using a variety of metrics.

Conclusion:

The results of teeth identification tasks carried out by DL models are encouraging; however, there is a need for the shortcomings that have been identified in our preliminary review, to be addressed by future researchers.

Introduction

Dental panoramic radiographs are an essential adjunct for diagnosis, treatment planning and assessment in clinical dentistry. ¹ The diagnostic accuracy is dependent on the expertise of the dental healthcare providers who would benefit greatly by the automation of this process, effectuating a higher diagnostic yield. ²

Machine Learning (ML) and Deep Learning (DL) algorithms are computer-assisted frameworks that have been a focus of research for radiographic analysis and interpretation in medicine and dentistry. ³ Recently, however, interest in DL has overtaken ML due to the inherent nature of DL algorithms that make their results superior. ⁴ ML algorithms require a pre-processing step of ‘feature extraction’ whereby, a human annotator is required to manually label the key features in the input image for the algorithm to learn. ⁵ Contrarily, DL algorithms do not require constant manual identification of such features. Instead, they learn from the initial examples laid down preliminarily, and map the new input to output independently. ⁶ Since medical image pre-processing is a complex task requiring domain experts for continuous manual identification of important features; leveraging ML for the automation of radiograph interpretation is time consuming and expensive. Therefore, DL is currently dominating research in medical and dental image processing for the automation of this process. ⁶

In dentistry, PRs (such as Orthopantomograms) are a commonly used imaging modality for diagnosing oral and maxillofacial diseases. PRs are being used to train DL models and have been employed for various diagnostic tasks thus far. ⁷ These tasks include: tooth classification, diagnosis of dental caries, determination of working length, periapical lesions, vertical root fractures, maxillary sinus pathologies, ameloblastomas, odontogenic lesions and periodontal bone loss etc. ^5,8 The results from these studies hold great promise for the clinical implementation of their diagnostic abilities. The application of these results, however, necessitates the accurate association of pathology with the causative anatomic structure. ⁹ Since dental diseases are either directly associated with, or lie in close proximity to teeth; their identification is one of the most important pieces of the dental image automation puzzle. ML as well as DL models have been utilized for teeth identification in the past. ⁵ Due to the complex anatomy and overlap of anatomical boundaries on PRs, this still remains a particularly challenging task for an AI model. ²

The detection of teeth in PR will be referred to as ‘teeth identification’ in this review and the techniques are further classified into object detection, semantic segmentation and instance segmentation as shown in Figure 1. ^10,11 Our aim is to explore the current status of scientific evidence available for the identification of teeth on PR, to determine the techniques being employed (object detection, semantic and instance segmentation techniques) and the specific DL models utilized for teeth identification. Furthermore, this review also identifies gaps in research and discusses the future prospects of DL in teeth identification.

Figure 1.

Teeth detection tasks: (i) unannotated orthopantomogram. (ii) object detection of all teeth via bounding boxes. (iii) semantic segmentation of all teeth as ‘tooth’ via fluid outlines. (iv) instance segmentation of all teeth as each ‘individual tooth’ via fluid margins.

Methods and materials

This systematic review was conducted using a predetermined protocol in accordance to the Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The following PICO model was employed:

1. What are the developments of DL models in teeth identification tasks on PRs?

2. What are the types of DL models that have been employed for teeth identification tasks on PRs?

3. What are the diagnostic performance of the DL models in teeth identification tasks on PRs?

Population: PR radiographs (orthopantomograms)

Intervention: DL models employing teeth identification tasks

Comparison: Expert opinion, reference standard and/or ground truth

Outcome: Diagnostic performance of DL models

The study protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) and a registration number was obtained (Reg. No.: CRD42021249627) prior to commencing the review. This was done to ensure transparency in research and to avoid any unplanned duplication of the review questions. The authors designed focused and targeted review questions which are as follows:

1. What are the developments of DL models in teeth identification tasks on PRs?

2. What are the types of DL models that have been employed for teeth identification tasks on PRs?

3. What are the diagnostic performance of the DL models in teeth identification tasks on PRs?

Inclusion criteria

1. Studies that evaluated a DL model for teeth identification on PRs.

2. Studies that evaluated teeth detection/segmentation of all teeth on PRs.

3. Studies that reported the performance metrics of the DL model utilized.

4. Studies that explicitly mention the type of DL model used for teeth identification.

Exclusion criteria

1. Studies not in English language.

2. Studies only available as abstracts.

3. Studies registered as protocols only.

Literature search

A comprehensive literature search was conducted in May 2021 to identify relevant literature in the four major health sciences databases i.e., PubMed NLM, EBSCO Dentistry & Oral Sciences Source, and Wiley Cochrane Library. Furthermore, a manual search was also performed by the authors in Google Scholar and IEEE Xplore databases to identify relevant literature not present in the aforementioned databases.

Search strategy

Database-specific keywords and medical subject headings (MeSH) were predefined and a comprehensive search strategy was implemented. The authors used various combinations of these search terms and conducted a pilot literature search to formulate the final search strategy. Furthermore, hand search was carried out after reviewing the reference list of all included articles including those of the IEEE Xplore database. Article citations were exported to EndNote reference manager where duplicate references were removed. Three independent reviewers performed a detailed screening of the studies. Any disagreement between the reviewers was resolved through discussion and a consensus was reached where all authors agreed.

Search terms

‘Teeth segmentation’, ‘Image segmentation’, ‘Orthopantomography’, ‘Tooth Classification’, ‘Teeth detection and numbering’, ‘Panoramic radiography’, ‘Radiographic image interpretation’ ‘Dental X-ray image’, ‘Computer assisted diagnosis’, ‘Convolution neural network’, ‘Deep learning’, ‘U-net’, ‘CNN’, ‘Artificial neural network’, ‘R-CNN’.

Data extraction

A customized proforma was designed by the authors to extract the following data from included studies:

1. Study details (authors, journal, country and year of publication).

2. Type of DL model used (U-Net, R-CNN, ResNet, etc).

3. Diagnostic performance parameters (precision, recall, Intersection over Union (IoU), F1-Score, DICE index).

4. Study characteristics (sample size, annotators if specified), datasets studied, reference standard).

Risk of bias assessment

The risk of bias and methodological quality of the included studies was assessed using the modified Quality Assessment of Diagnostic Accuracy Studies QUADAS-2 tool. ^12,13 All authors (FUMR, SQHB, NHAD) of this review were involved in this process. The modified QUADAS-2 tool derived from a study was further modified to ensure better quality assessment of the included studies. ¹⁴ To the best of our knowledge, a specifically tailored tool for assessing the methodological quality in AI-related research in dentistry does not exist. ¹⁵ Due to this, questions related to performance metrics, use of validation and data augmentation techniques, information for future replicability and reference standard were further included to formulate a tool better suited to our review. ¹⁴ This tool was then piloted on three studies to ensure its applicability. The combination of all satisfied criteria was used to give a cumulative numerical score; a high quality (score 6–8), moderate quality (score 4–5) and low quality (score 0–3) depicting the methodological quality of each included studies. The outcome of risk of bias assessment are presented in Table 1.

Table 1.

Quality assessment of the selected studies using modified QUADAS tool. The ‘✓’ demonstrates a favourable response to the question and the ‘X’ demonstrates an unfavourable response to the question. The overall score reflects the quality and risk of bias for each study

Quality Assessment Questions	Nishitani et al 2021	Leite et al 2021	Lee et al 2020	Zhao et al 2020	Mahdi et al 2020	Mura-matsu et al 2020	Silva et al 2020	Muresan et al 2020	Chung et al 2020	Tuzoff et al 2019	Koch et al 2019	Wirtz et al 2018	Jader et al 2018
Does the sample size consist of >100 images?	✓	✓	×	✓	✓	×	✓	✓	✓	✓	✓	×	✓
Was there an explicit inclusion/exclusion criterion defined for dataset selection?	×	✓	✓	✓	×	×	✓	×	×	×	✓	✓	✓
Did the investigators mention sample size calculation?	×	×	×	×	×	×	×	×	×	×	×	×	×
Is the dataset multicentric?	×	×	×	×	×	×	×	✓	✓	×	×	×	×
Was there adequate information provided for replicating study?	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Did the investigators employed any validation techniques in the test data to reduce bias?	×	×	×	✓	×	✓	✓	×	×	×	×	×	×
Does the study mention type of specialist involve for annotation?	×	✓	✓	×	×	✓	×	×	×	×	×	×	×
Did the investigators report more than one performance metrics?	✓	✓	✓	✓	✓	×	✓	✓	✓	✓	×	×	✓
Did the investigators employ any data augmentation techniques?	×	✓	✓	×	×	✓	✓	✓	×	✓	✓	×	×
Overall score	3	6	5	5	3	4	6	5	4	4	4	2	4

High (6-8) score

Moderate (4-5) score

Low (0-3) score

Data synthesis

Since a high heterogeneity among outcome measures and DL models were observed amongst studies, a quantitative analysis of the results i.e., a meta-analysis was not possible. Hence, the findings of included studies are presented narratively.

Results

A total of 282 studies were identified following a detailed electronic and manual literature search. After removing duplicates and performing a detailed screening of the studies, 13 articles fulfilled our inclusion criteria. Hence, these 13 articles were subjected to a final analysis. The screening process is shown as PRISMA flowchart in Figure 2. A graphical summary of the methodology and results is illustrated in Figure 3.

Figure 2.

PRISMA flowchart

Figure 3.

Graphical summary of the review

Description of included studies

The main findings of the included studies are summarized in Table 2. All included studies were published between 2018 and 2021. The results are described based on the type of teeth identification methods i.e., object detection, semantic segmentation and instance segmentation techniques. This is followed by the identification of the type of DL model that was used as well as their diagnostic performances.

Table 2.

Main findings of the included studies

Author, Year, (Country)	AI Model used	Sample size	Annotator(s)/ Ground truth	Teeth Identification Task(s)	Performance metrics	Conclusions
Nishitani et al24 2021 (Japan)	U-Net	162 images Training set: 82 Validation set: 20 Test set: 60	Not mentioned	Semantic segmentation	Jaccard index (0.809) Dice index (0.894)	U-Net with the new loss function exhibited a higher segmentation accuracy of teeth in panoramic dental X-ray images than that obtained by U-Net with the conventional loss function
Leite et al1 2021 (Belgium)	ResNet 101	153 images Training set: 70 Validation set: 18 Test set:65	Oral radiologist with 20 years’ experience	Instance segmentation, Time Analysis	Sensitivity (0.98) Precision (0.99) IoU (0.95) Recall (0.98) F1 score (0.97) Hausdroff distances (7.9)	The AI tool yielded a highly accurate and fast performance for detecting and segmenting teeth, faster than the ground truth alone. Also, the time needed to perform fully manual segmentation of teeth on a panoramic radiograph may be reduced by 67% when using a segmentation method based on deep learning algorithms.
Lee et al2 2020 (Korea)	R-CNN	30 images 846 teeth	Oral radiologist with 5 years’ experience	Instance segmentation	Precision (0.85) Recall (0.89) IoU (0.87) F1 score (0.87) Visual analysis	The method achieved high performance for automation of tooth segmentation on dental panoramic images. The proposed method might be applied in the first step of diagnosis automation and in forensic identification, which involves similar segmentation tasks.
Zhao et al21 2020 (China)	TSASNet	1500 image Training set :1200 Validation set: 150 Test set:150	Not mentioned	Semantic segmentation	Accuracy (0.96) Recall (0.93) Specificity (0.97) Precision (0.94) Dice index (0.92)	Results showed that TSASNet can obtain superior segmentation performance on dental panoramic images over other state-of-the-art methods and has highly competitive performance compared with the current medical image segmentation methods.
Mahdi et al6 2020 (Japan)	Faster RCNN (ResNet100 +and ResNet 50)	1000 image Training set: 900 Test set :100	Not mentioned	Object detection	ResNet 50 Precision (0.97) ResNet 101 Precision (0.98) F1 score (0.98)	The proposed model can be used as a useful and reliable tool to assist dental care professionals in dentistry.
Muramatsu et al15 2020 (Germany)	ResNet 50	100 images	Dental radiologist	Object detection	Sensitivity (0.96)	High tooth detection sensitivity and classification accuracies were obtained using a limited dataset, suggesting the potential utility of the proposed method in the automatic filing of dental records.
Silva et al17 2020 (Bahia)	Mask R CNN PA Net HTC ResNet	543 images	Not mentioned	Semantic segmentatin, Instance segmentation, Teeth numbering	Mask R CNN Accuracy (0.96) F1 score (0.90) PA Net Accuracy (0.96) F1 score (0.91) HTC Accuracy (0.96) F1 score (0.89) ResNet Accuracy (0.96) F1 score (0.91)	Results showed that instance segmentation and numbering are feasible to be accomplished by an end-to-end deep network. In our experiments, PANet achieved the best results i.e.F1 score of 91.65% on semantic segmentation
Muresan et al25 2020 (Romania)	ERF Net	1000 image Training set: 700 Validation set: 100 Test set: 200	Not mentioned	Semantic segmentation	Accuracy (0.89) Precision (0.98) Recall (0.91) F1 score (0.93)	The proposed method solution is able segment accurately the teeth and identify the problems correctly as compared to other methods which are not able to identify all the problems correctly.

Four studies included in our review performed the teeth identification task through DL-based object detection. ^6,16–18 Others leveraged DL for either semantic segmentation or instance segmentation methods of teeth identification. Furthermore, two studies also performed a supplementary task of tooth numbering in addition to teeth identification. ^18,19

Datasets studied

We noticed a large variation in the datasets utilized by the included studies ranging from 14 radiographs to 1500 radiographs. ^20–23 Only four studies mentioned the details of annotators (oral radiologists/experts). ^1,2,17,18 The remaining studies did not mention any details of annotators and or the annotation methods involved in the training process. However, these studies were still included in our review in an attempt to assess as many DL models as possible.

DL models studied

ResNet is an advanced CNN with many deep layers incorporated in its framework with the ability to minimize the vanishing gradient problem via skip connections. It is a deeper neural network compared with its predecessors AlexNet and GoogleNet constituting 8 and 22 layers, respectively. Investigators have since then experimented with the number of deep layers in the ResNet architecture, such as Resnet 101 and ResNet50. ^1,17 Four studies employed the base algorithm ResNet, two of them utilized this algorithm for instance segmentation and the remaining two studies employed it for object detection tasks. ^1,16,17,19 For instance segmentation, two studies exhibited good overall performance of the ResNet architecture (0.97 and 0.91 F1-scores, respectively). ^1,19 For object detection, ResNet-based algorithms achieved high sensitivity (91%), precision (0.99) and recall scores (0.97). ^16,17

R-CNN was developed to increase the computational efficiency of computer vision tasks, it proposes 2000 regions of interest in a given image, known as ‘region proposal’, which are then fed through the CNN for processing. Fast and Faster R-CNN are a further development on the R-CNN model which facilitates real-time image processing. ^24,25

Three studies in our review leveraged R-CNN and its advanced version Faster R-CNN for teeth identification tasks. ^2,6,18 Only one of those utilized R-CNN specifically for instance segmentation task with moderate precision scores (0.85). ² The others used Faster R-CNN for object detection to identify teeth on PR. ^6,18 In contrast, Faster R-CNN achieved higher precision scores for object detection with a high precision score (0.99). ¹⁸ One study utilized both ResNet 50 and ResNet 101 as a back-bone for his Faster R-CNN also reported high precision scores (0.97 and 0.98, respectively). ⁶

Mask R-CNN is a further upgrade on the previous R-CNN and Faster R-CNN and is being employed particularly for instance segmentation tasks. ²⁰ Two studies utilized Mask R-CNN on PRs and reported acceptable performance (0.88 and 0.90 F1-scores, respectively). ^19,20

U-net is a relatively new DL algorithm composed of a fully convolutional network, i.e., it consists of convolutional layers with no hidden layers. The algorithm architecture consists of encoder–decoder layers which when visualized appear similar to the letter ‘U’. This technique is used to overcome the ‘bottleneck problem’ encountered in linear CNN models which results loss of important features in original images. Three studies utilized U-net-based algorithms and demonstrated satisfactory performance of the models with acceptable DICE index scores (0.7, 0.9 and 0.927, respectively). ^21,22,26

ERFNet was utilized in one study which is a linear encoder–decoder convolution network with a total of 23 layers. ²⁷ The first 16 layers are used for encoding the image and the remaining seven are utilized for decoding, allowing for segmentation of teeth with excellent results. (F1 score 0.93). ²⁷

An end-to-end model TSASNet was employed by one study for segmentation of PR, demonstrating high-grade performance (DICE index 0.92). ²³

Risk of bias assessment

A modified QUADAS −2 tool was utilized to assess the quality of studies. ¹⁴ By combining scores of the satisfied criteria, an overall score of each study was determined and is presented in Table 1. Two studies were able to achieve the highest score of 6. ^1,19 Whereas, one received the lowest score of 2. ²²

Discussion

To reliably utilize DL for dental image processing and PR automation, there is a need to substantiate the usefulness of the perpetually evolving DL algorithms. This can only be done by testing these algorithms under solid experimental conditions representing real-world scenarios.

As the focus of this review was mainly DL, all the included studies were published after 2018. The algorithms which have been utilized during this time represent the state-of-the-art Convolutional Neural Networks (CNNs) available to the research community. In this review, we noted that DL models have been appropriately harnessed to perform teeth identification tasks. Investigators have used multifarious performance metrics (sensitivity, precision, accuracy, F1-score, IoU, DICE index, etc) and have reported optimal performance. Moreover, two studies also performed teeth numbering in addition to teeth identification, with encouraging results. ^18,27

As we have already described, tooth identification tasks may be classified into three categories; object detection, semantic segmentation and instance segmentation. In object detection, teeth in the PRs were identified and localized using bounding boxes; and four of the included studies adopted this method with good results. ^6,16–18

Teeth segmentation is a further variation of teeth identification in which the teeth are detected and segregated from the background of the image. ²⁸ This segmentation task can be further classified as semantic and instance segmentation. ²⁹ In semantic segmentation, multiple items of the same class are identified as a single entity. ^28,29 Whereas, in instance segmentation multiple similar items are considered as being individual entity. All teeth are identified as ‘tooth’ and not as individual tooth class in semantic segmentation. Contrarily, in instance segmentation, all teeth can be individually identified according to their tooth class (for example: molar, premolar, canine, etc).

Overall, four studies performed object detection, ^6,16–18 with two studies performing teeth numbering task in addition to object detection. ^18,19 Four studies performed only semantic segmentation, ^21,23,26,27 and a further four employed purely instance segmentation for teeth identification. ^1,2,20,22 One study performed three tasks, namely, teeth numbering via object detection, semantic segmentation and instance segmentation on PRs. ¹⁹

For the automation of radiographic interpretation, the ideal dataset should be large enough to encompass a wide range of clinical scenarios. Additionally, the data should include variables such as different machines used for radiography and patients with diverse ethnicities/ geographical backgrounds, this will allow for mitigation of the phenomenon of ‘overfitting’ associated with DL algorithms. ³⁰ Moreover, annotation of ground truth needs to be done by multiple radiologists with varying level of experience in order to account for interexaminer variations. In our review it was observed that only four studies gave information regarding annotation of ground truth. ^1,2,17,18 Only two of these gave details regarding the experience of annotators involved. ^1,2 And only one study used more than one annotator for annotation of datasets. ¹⁸ Two studies employed multicentric data, all others utilized data from a single center; this may have made their results overfitted, resulting in limited generalizability on unseen data from another cohort. ^16,27

DL is data intensive and require large datasets to make accurate predictions. ³⁰ In our study, we noted heterogeneity in the datasets being employed; the largest dataset being that of 1500 PRs radiographs. Four out of thirteen studies utilized the same dataset for training DL algorithms. ^19–21,23 This dataset is popularly known as UFBA-UESC Dental Images Dataset. ^5,19 The PRs in this dataset are grouped into ten separate categories including teeth, missing teeth, dental appliances, restorations, implants, etc. This original dataset was annotated to be used for semantic segmentation task. Later, Jader et al upgraded the same dataset to include annotation to allow for instance segmentation. This dataset was named as UFBA-UESC Dental Images Dataset Deep. ^19–21

For the training and evaluation of DL model, datasets should be divided into three groups; training, validation and testing datasets. Whilst most studies used this template of evaluation of their models, four studies did not follow this protocol. ^2,17,19,22 Therefore, these studies have incorporated a high risk of bias in their results, which could have been easily avoided by stratifying their dataset into predetermined groups as stated above.

Like any other project our review also suffers from certain limitations, although we had a predefined PICO question, our review outcomes are rather broad. Some of the findings were included during the synthesis phase of the literature search since we decided to include conference proceedings as well. Owing to the fact that DL is an unexplored domain, we wanted to include the maximum information aiming to benefit the readers of this paper. The literature search was done by only one author with the help of a librarian; albeit the extraction was cross-checked by all three authors. The included studies were diverse with heterogeneous algorithms and performance metrics, hence no direct comparisons amongst studies and their performance was possible.

Our review also has one distinct advantage as in our search we also included the IEEE Xplore database to cover a large majority of engineering publications on the same topic which may otherwise have been missed. It can be argued that inclusion of these publications add gray literature to our review since these conference proceedings are not peer-reviewed. However, they were included in an attempt to incorporate all recent evidence available on teeth identification in our review.

Conclusions and future applications

For future research, there is a need to develop a standardized methodology and pre-determined performance metrics to increase the overall robustness and generalizability of DL algorithms for diagnostic purposes. ⁴ The EQUATOR network is available for such guidance in AI-based intervention studies. ³¹ The need for such a network specific for dentistry still prevails and is available in the form of a checklist at present. ³¹ However, the said checklist does not extensively cover all important aspects of reporting AI-based interventional studies and needs to be improved further for widespread applicability of DL models in dentistry. ³¹

Strategies should be devised to generate AI datasets which are properly collected, curated and made available to researchers globally, in a systematic, secure and ethical way. Data annotation process needs to be regulated with the development of a standardized software and tools so that dataset attributes are consistent and can be utilized by researchers all around the globe.

According to our review, teeth identification with the help of CNNs have shown promising results however the included studies suffered from certain limitations such as overfitting, high risk of bias and high heterogeneity. Because of these reasons, the utilization of DL models for performing teeth identification tasks in clinical practice is still questionable. However, the evidence for routine use of DL models as an adjunct diagnostic aid is sparse but developing steadily. ⁴

The throughput cost-effectiveness and efficiency of CNN-based models is still not very well understood and its effect on their overall acceptability in clinical practice needs further exploration. Nonetheless, as the preliminary results are encouraging it is hoped that the shortcomings of DL, which have been identified in our review will be addressed by future researchers.