Detection of unilateral and bilateral cleft alveolus on panoramic radiographs using a deep-learning system

Abstract

Objectives:

The purpose of this study was to evaluate the difference in performance of deep-learning (DL) models with respect to the image classes and amount of training data to create an effective DL model for detecting both unilateral cleft alveoli (UCAs) and bilateral cleft alveoli (BCAs) on panoramic radiographs.

Methods:

Model U was created using UCA and normal images, and Model B was created using BCA and normal images. Models C1 and C2 were created using the combined data of UCA, BCA, and normal images. The same number of CAs was used for training Models U, B, and C1, whereas Model C2 was created with a larger amount of data. The performance of all four models was evaluated with the same test data and compared with those of two human observers.

Results:

The recall values were 0.60, 0.73, 0.80, and 0.88 for Models A, B, C1, and C2, respectively. The results of Model C2 were highest in precision and F-measure (0.98 and 0.92) and almost the same as those of human observers. Significant differences were found in the ratios of detected to undetected CAs of Models U and C1 (p = 0.01), Models U and C2 (p < 0.001), and Models B and C2 (p = 0.036).

Conclusions:

The DL models trained using both UCA and BCA data (Models C1 and C2) achieved high detection performance. Moreover, the performance of a DL model may depend on the amount of training data.

Introduction

A deep-learning (DL) system is a machine learning technique based on a convolutional neural network, which mimics human brain neurons. When a substantial amount of data are inputted into the multilayer network, it can automatically abstract the characteristics of the data and create a trained model for various tasks on new input datasets. ¹ DL systems have often been used for the detection and classification of various oral and maxillofacial diseases and conditions on panoramic radiographs, such as radiolucent lesions in the mandible, ² root fractures, ³ maxillary sinus lesions, ⁴ impacted supernumerary teeth, ⁵ root morphology of the mandibular first molar, ⁶ apical lesions, ⁷ dental implants, ⁸ C-shaped canals in mandibular second molars, ⁹ deciduous teeth, ¹⁰ and sialoliths of the submandibular gland. ¹¹

Cleft lip and palate (CLP) is one of the most common congenital anomalies in the maxillofacial region and is caused by both environmental and genetic factors. ¹² The prevalence of CLP is 1 per 800 births in Europe and the United States, whereas it is higher in the Japanese population with a frequency of 1 in 500 births. ¹³ A cleft alveolus (CA) often occurs in CLP patients and is usually treated with bone graft techniques at the age of 8 to 10 years, when the maxilla has grown sufficiently for this surgery. ¹⁴ Therefore, patients are followed up regularly from birth through physical and imaging examinations. Panoramic radiography plays an essential role in evaluating the status of the CA because of its low level of radiation exposure to patients and low cost compared with computed tomography (CT) or cone-beam CT for dental use. ¹⁵

Although CA status can be easily recognized during a physical examination, oral and maxillofacial radiologists, who must routinely interpret many panoramic radiographs, cannot always perform such examinations and are forced to diagnose the presence of clefts using the panoramic appearance alone. In such cases, a computer-aided diagnosis/detection system created using a DL convolutional neural network would help radiologists, especially those who are inexperienced, avoid overlooking clefts. To date, only one study has reported the automatic detection of unilateral CA (UCA) on panoramic radiographs. ¹⁶ In that study, two models were created using 323 cases of UCA with and without normal data. Although the recalls were relatively high in both models (over 0.85), the model created without normal data falsely classified 12 out of 30 (40%) normal test images as CA images. By contrast, only one (3%) of the normal test images was misclassified as a CA image by the model created with normal data. Consequently, the DL model appears to perform well in detecting UCA on panoramic radiographs. However, its suitability for bilateral CA (BCA) detection has not been verified. If the panoramic appearances of UCA and BCA are identical when a cleft is examined, the DL model for detecting UCA, which was trained using only UCA data, should also be effective for detecting BCA. If this is not the case, another model should be trained on data that include BCA radiographs.

The purpose of this study was to create an effective DL model for detecting both UCA and BCA and to compare its performance with the performance of human diagnosis. For this purpose, we compared the detection performance of four models: a model created using UCA and normal data (Model U), one created using BCA and normal data (Model B), and two created using combined UCA, BCA, and normal data (Models C1 and C2).

Methods and materials

Informed consent was obtained from all patients who were included in the study. This study was approved by the ethics committee of our university (no. 496) and was performed in accordance with the Declaration of Helsinki.

Patients

Panoramic radiographs of 353 patients (153 female and 200 male) with UCA and 93 patients (39 female and 54 male) with BCA were retrospectively selected from our hospital image database, which contained images collected between August 2004 and July 2020. Among the patients with UCA and BCA, 194 (54.9%) and 75 (80.6%) patients, respectively, had a cleft palate. The mean age of CA patients was 8.8 years. All patients were examined repeatedly using panoramic radiography, and the radiographs taken immediately before bony transplant surgery were selected. All patients were verified as having UCA or BCA by medical records and CT examinations. The normal group without CA consisted of panoramic radiographs of 210 patients who matched the mean age and sex distribution of the CA patients. These patients had been examined for other purposes, such as pre-examination for orthodontic treatment. These radiographs were selected from the same database and collected over the same period.

The panoramic radiographs were taken using either a Veraviewepocs unit (J. Morita Corp., Kyoto, Japan), with a tube voltage of 75 kV, tube current of 8 mA, and exposure time of 16.2 s, or an AUTO IIINTR unit (Asahi Roentgen Industry, Kyoto, Japan), with a tube voltage of 75 kV, tube current of 12 mA, and exposure time of 12 s.

DL architecture

The DL process was performed on a computer running Ubuntu version 16.04.2 with an 11 GB graphics processor unit (NVIDIA GeForce GTX 1080 Ti; NVIDIA, Santa Clara, CA, USA), and Digits version 5.0 training system (NVIDIA). A customized DetectNet (https://developer.nvidia.com/blog/detectnet-deep-neural-network-object-detection-digits/) with object detection and classification functions was used. A DetectNet has five main parts: (1) data input and data augmentation; (2) a fully convolutional network, which performs feature extraction and prediction of object classes and bounding boxes per grid square; (3) loss function measurement; (4) bounding box clustering; and (5) mean average precision calculation. ² The adaptive moment estimation (Adam) solver was used with 0.0001 as the base learning rate.

Model training

For the training process, panoramic image data with annotated labels were needed for the training and validation data. The panoramic images were downloaded in JPEG format and cropped to 900 × 900 pixels. To create the labels, rectangular regions of interest (ROIs) were marked and the coordinates of the upper left (x1, y1) and lower right (x2, y2) corners were recorded using ImageJ software (National Institute of Health, Bethesda, MD, USA; Figure 1a). Thereafter, they were converted to text form (Figure 1b). In the normal group, labels without coordinates were created.

Figure 1.

(a) Region of interest (ROI) of the cleft alveolus (CA) area. The superior distal corner is defined as the most distal portion of the nasal cavity lateral wall, and the inferior medial corner is located at the alveolar ridge between the central incisors. The coordinates are then recorded. (b) Example of converting the coordinates of the ROIs in a patient with a classification of unilateral CA (UCA). For the bilateral CA (BCA) group, two coordinates were assigned.

The ROI of each CA area was determined using the following definitions. The superior distal corner was defined to be the most distal portion of the nasal cavity lateral wall, and the inferior medial corner was located at the alveolar ridge between the central incisors. Consequently, the BCA group had two ROIs.

We created four models (Models U, B, C1, and C2) in the present study. Model U was created using only UCA and normal images, Model B was created using BCA and normal images, and Models C1 and C2 were created using UCA, BCA, and normal images as training and validation data (Table 1). Model C1 was created using datasets that had the same number of CAs (184) as those used for Models U and B. To investigate the difference in performance caused by the number of learning data, we created Model C2 which had a larger amount of the learning data than Model C1. For creating Model C2, 231 and 32 images were added to the learning data of UCA and BCA group, respectively, with no change in the number of Normal group. Test data were randomly selected in equal numbers: 30 images (30 CAs) from the UCA group, 15 images (30 CAs) from the BCA group, and 30 images from the normal group. These test images were used for all models. The training and validation data were arbitrarily selected in a ratio of 80:20.

Table 1.

Summary of datasets (number of panormaic images)

		Model U	Model B	Model C1	Model C2
Learning process	UCA group	184 (184)	-	92 (92)	323 (323)
	BCA group	-	92 (184)	46 (92)	78 (156)
	Normal group	180 (0)	180 (0)	180 (0)	180 (0)
	Total number of datasets	364 (184)	272 (184)	318 (184)	581 (479)
Inference process	UCA group	30 (30)	30 (30)	30 (30)	30 (30)
	BCA group	15 (30)	15 (30)	15 (30)	15 (30)
	Normal group	30 (0)	30 (0)	30 (0)	30 (0)
	Total number of datasets	75 (60)	75 (60)	75 (60)	75 (60)

BCA, Bilateral cleft alveolus; UCA, Unilateral cleft alveolus.

Number in parenthesia denotes number of cleft alveolus.

Data augmentation was automatically applied in a Transformer layer in the DetectNet and performed various image processing operations, such as shift, flip, and desaturation.

To create each model, 1000 training epochs were performed. Applying the test data to the learning models, a rectangular red box was shown on the test images when the model detected a CA (Figure 2).

Figure 2.

Correctly detected unilateral cleft alveolus (UCA) in the right maxilla, indicated by a red rectangular box

The detection performance was evaluated using the values of recall, precision, and F-measure, which are calculated as follows: recall (sensitivity) = number of correctly detected CAs / number of all CAs. precision (positive predictive value)=number of correctly detected CAs / (number of correctly detected CAs + number of regions falsely detected as having CA). F-measure = 2 (recall×precision) / (recall+precision).

Comparison with the detection performance of human observers

To compare the DL performance with that of human observers, a radiologist and a dental resident evaluated the same test data that were used for the evaluation of the DL models. They evaluated both sides of the maxillary incisor regions and determined whether CAs were present or absent.

Statistical analysis

The differences in ratios of detected and undetected CAs in the evaluators were tested using the chi-square test. The threshold for significant difference was p < 0.05.

Results

In the results for the test images, at most two bounding boxes were observed. All bounding boxes estimated by the models were located in areas where CAs truly existed or would arise. These areas were similar to the areas annotated in the training and validation data. No false-positive boxes were found in areas other than these areas.

The test results and performance results of the models and observers are shown in Tables 2 and 3. The recall was 0.60 (36/60 CAs), 0.73 (44/60 CAs), 0.80 (48/60 CAs), and 0.88 (53/60 CAs) for Models U, B, C1, and C2, respectively. Significant differences were found in the ratios of detected to undetected CAs in the results of Models U (number of detected/undetected CAs: 36/24) and C1 (48/12)(p = 0.01), Models U (36/24) and C2 (53/7)(p < 0.001), and Models B (44/16) and C2 (53/7)(p = 0.036). No difference was found between the ratios of the results of Models B (44/16) and C1 (48/12).

Table 2.

Testing results by deep-learning models and observers (Number of clefts)

		UCA group	BCA group	Normal group	Total
Model U	Correctly detected cleft	28	8	-	36
	Undetected cleft	2	22	-	24
	Falsely detected area a	0	0	2	2
Model B	Correctly detected cleft	19	25	-	44
	Undetected cleft	11	5	-	16
	Falsely detected area a	8	0	0	8
Model C1	Correctly detected cleft	28	20	-	48
	Undetected cleft	2	10	-	12
	Falsely detected area a	0	0	1	1
Model C2	Correctly detected cleft	27	26	-	53
	Undetected cleft	3	4	-	7
	Falsely detected area a	0	0	1	1
Radiologist	Correctly detected cleft	30	26	-	56
	Undetected cleft	0	4	-	4
	Falsely detected area a	1	0	0	1
Dental resident	Correctly detected cleft	29	21	-	50
	Undetected cleft	1	9	-	10
	Falsely detected area a	1	0	0	1

BCA, Bilateral cleft alveolus; UCA, unilateral cleft alveolus.

^aFalsely-detected normal area as cleft alveolus.

Table 3.

Detection performance of deep-learning models and observers

	Recall	Precision	F-measure
Model U	0.60	0.94	0.73
Model B	0.73	0.84	0.77
Model C1	0.80	0.97	0.87
Model C2	0.88	0.98	0.92
Radiologist	0.93	0.98	0.95
Dental resident	0.83	0.98	0.89

Regarding the UCA group, the ratios of the results for Models U and C1 were both 28/2 CAs, whereas it was 19/11 CAs for Model B. Regarding the BCA group, the ratios of the results for Models B and C1 were 25/5 CAs and 20/10 CAs, respectively. However, the ratio of Model U was 8/22 CAs.

Models U, C1, and C2 had very few false-positive results, achieving precisions of over 0.9. These three models falsely detected a few normal areas in the normal group as CA areas (Figure 3). Model B falsely detected eight normal areas situated at the contralateral side-of the UCAs as CA areas, indicating that eight UCA patients were incorrectly assigned to the BCA group (Figure 4).

Figure 3.

Example of a normal case that Model A erroneously detected as having a cleft alveolus (CA) in the right maxillary area.

Figure 4.

Case with a unilateral cleft alveolus (UCA) on the left side-and cleft palate. Models A, C1, and C2 correctly detected the left cleft alveolus (a), whereas Model B falsely detected both the CA and contralateral normal sides (b).

For the human observers, the recalls were 0.93 (56/60 CAs) and 0.83 (50/60 CAs) for the radiologist and dental resident, respectively. Both observers erroneously detected a contralateral side-of a UCA as a CA area (Figure 5). This indicated that one UCA patient was misdiagnosed as a BCA patient.

Figure 5.

Case with a unilateral cleft alveolus (UCA) on the left side that the human observers falsely diagnosed as a bilateral cleft alveolus (BCA).

The performance of Model C2 was highest among the four models and was equivalent to the performance of the radiologist.

Discussion

Comparing the three models (Models U, B, and C1) created using an equal amount of data, the recall of Model C1 (0.80) was higher than those of Models U (0.60) and B (0.73). Regarding the UCA group, Models U and C1 performed well, with CA detection rates of 93%, whereas it was only 63% (19/30 CAs) for Model B. Regarding the BCA group, Models B and C1 detected 25/30 CAs (83%) and 20/30 CAs (67%), respectively. However, Model U detected only 8/30 CAs (27%). These results indicate that both UCA and BCA data are required to create an effective DL model for detecting CAs on panoramic images. The results of Model B included eight false-positive areas, all of which were located on the contralateral side of the UCA. This is probably caused by the lack of UCA data during the training of Model B. In contrast, Models U and C1 did not detect such areas. Comparing Models C1 and C2, which were developed using different amounts of combined UCA, BCA, and normal data, there were fewer undetected CAs in the results of Model C2, which had a recall of 0.88. This result confirms that the DL model performance was improved by increasing the number of training and validation data. Moreover, no significant differences were found in the ratio of detected to undetected CAs between the results of Models B and C1, whereas this ratio was significantly different in the results of Models B and C2. This result might be due to the increased detection rate of BCA in Model C2.

In the present study, we hypothesized that if the panoramic appearances of a UCA and BCA are identical when a cleft is examined, a DL model for detecting UCA built using only UCA data should also be effective for detecting BCA. The results of our study disproved our hypothesis because significant differences were found in ratios of detected to undetected CAs between Models U and C1 (p = 0.01) as well as between Models U and C2 (p < 0.001). This suggests that there might be a difference between the UCA and BCA findings on panoramic radiographs. The higher prevalence of an associated cleft palate in the BCA group might be a possible reason for the difference in the appearance of a cleft in the UCA and BCA groups. The CAs associated with a cleft palate may show more radiolucency than those without a cleft palate. Furthermore, the detection rate of CAs in the UCA group obtained by Model B (19/30 63%) was relatively high compared with that in BCA group obtained by Model U (8/30 27%). This suggests that the BCA might contain CAs with findings similar to those of the UCA.

In a previous study, ¹⁶ two DL models built for the automatic detection of UCA on panoramic radiographs with and without using normal data were compared. The results revealed that the model trained with normal data performed better and had fewer false-positive results. In the present study, therefore, panoramic radiographs without CAs were included as normal group data in the learning process. The low number of false-positive results in the present study could be attributed to this approach. Regardless, the UCA detection performances of Models C1 and C2 were equivalent to the performance of a model previously created ¹⁶ using both UCA and normal data, yielding a recall of 0.833 for CAs without a cleft palate. Moreover, Model C2 detected BCAs well.

The false-positive evaluations revealed differences among the models and human observers. Models U, C1, and C2 erroneously detected areas in the normal group, whereas the human observers falsely detected the contralateral side-of a UCA area (Figures 3 and 5). The models might learn the CA findings themselves, whereas the humans might take the right and left asymmetry into account when diagnosing CAs. Therefore, a UCA case with relatively symmetrical features might be misdiagnosed as a BCA by human observers. Model B, like the human observers, had no false-positives in the normal group, but in 26% of cases, a CA was falsely detected contralateral to the UCA side. This suggests that Model B might also take the right and left asymmetry into account when diagnosing CAs.

The present study had some limitations. First, the sizes of the datasets were relatively low, especially for the BCA group. With the current number of data, the potential of DL networks may not be fully exploited. Taking clinical use into account, future research should be planned to create a high-performance model using images obtained from multiple hospitals. Second, we did not analyze the differences in the panoramic appearance in great detail, even although there might be differences between the UCA and BCA findings on panoramic radiographs. In future research, these differences should be investigated. Third, in the present study, we did not take the presence or absence of a cleft palate into account. The presence of a cleft palate might also be a reason for differences in the panoramic appearance.

In conclusion, to create a sufficiently effective model for detecting both UCA and BCA, both UCA and BCA data are required. Furthermore, increasing the amount of data used for the learning process was found to improve model performance.