Performance of deep learning models constructed using panoramic radiographs from two hospitals to diagnose fractures of the mandibular condyle

Abstract

Objective:

The present study aimed to verify the classification performance of deep learning (DL) models for diagnosing fractures of the mandibular condyle on panoramic radiographs using data sets from two hospitals and to compare their internal and external validities.

Methods:

Panoramic radiographs of 100 condyles with and without fractures were collected from two hospitals and a fivefold cross-validation method was employed to construct and evaluate the DL models. The internal and external validities of classification performance were evaluated as accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC).

Results:

For internal validity, high classification performance was obtained, with AUC values of >0.85. Conversely, external validity for the data sets from the two hospitals exhibited low performance. Using combined data sets from both hospitals, the DL model exhibited high performance, which was slightly superior or equal to that of the internal validity but without a statistically significant difference.

Conclusion:

The constructed DL model can be clinically employed for diagnosing fractures of the mandibular condyle using panoramic radiographs. However, the domain shift phenomenon should be considered when generalizing DL systems.

Introduction

Panoramic radiographs are routinely used to screen various lesions and conditions in the maxillofacial region and to view the entire jaw bone. However, some regions are difficult to interpret because of the complicated relationship between anatomical structures and panoramic image layers.^1,2 The temporomandibular joint (TMJ) area is one such region.³ Condylar fractures are likely overlooked, particularly by inexperienced observers, for various reasons. In the superior portion, fracture lines may be superimposed with the articular eminence, and in the inferior portion, the boundary of the overlapped airway may be misdiagnosed as a fracture line. Moreover, the sagittal fracture line (perpendicular to the long axis of the condylar head) with a small bony gap is difficult to detect in panoramic radiography because the X-rays are projected parallel to the long axis of the condylar head.⁴

In recent years, deep learning (DL) artificial intelligence systems with convolutional neural networks (CNNs) have received considerable attention, and many investigators have applied DL with CNN to computer-aided detection/diagnosis (CAD) systems in the field of oral and maxillofacial radiology.^2,5–8 Additionally, DL with CNN has a potential use in supporting diagnoses of condylar fractures. Researchers have frequently used DL systems to classify and detect objects in panoramic radiographs to diagnose various diseases and conditions.^9–13

In general, the learning process requires a substantial number of data sets (i.e. hundreds of thousands or millions of data points) to construct versatile marketable models,¹⁴ particularly for diseases with significant heterogeneity.¹⁵ However, for a specific purpose or population, a relatively small amount of data (i.e. hundreds of data points) is sufficient as a data set.¹⁴ Almost all learning models constructed in the research setting are such models. Many previous studies using panoramic radiography have been performed with relatively small data sets within a single institution.^2,5–13 Although sufficiently high performance has been reported, only internal validity has been verified in such models, which indicates that model performance was tested using data sets obtained from the same institution and/or with the same imaging apparatus as those used for training the learning models. Sampling bias occurs in such cases,¹⁴ and the performance of the models might decrease when data sets from different institutions or data obtained with different apparatuses are applied. This phenomenon is attributed to the so-called “domain shift,” which indicates that the characteristics of data sets are different between those used for developing the learning models and those applied to models constructed as testing data sets.

The problem of generalizability when using learning models in clinics cannot be avoided. However, because no studies have verified the external validity for diagnosis using panoramic radiography, a solution to this problem may exist. Therefore, the aims of the present study were to verify the classification performance of DL models for diagnosing condylar fractures on panoramic radiographs using data sets obtained from two hospitals and to compare their internal and external validities.

Methods and materials

This study was approved by the Ethics Committees of Aichi Gakuin University (No. 586) and Ogaki Municipal Hospital (No. 20200423–13) and was performed in accordance with the principles of the Declaration of Helsinki.

Subjects

Patients aged ≥16 years who underwent panoramic and CT examinations at two hospitals (Hospitals A and B) were enrolled in the present study. Hospital A is a university dental hospital, and Hospital B is a general hospital. We selected 100 condyles on panoramic images retroactively since October 2019 from both hospitals. All cases were examined within 1 month after trauma and confirmed to have fractures of the mandibular condyle on CT images. We selected 100 condyles in 78 patients and 100 condyles in 77 patients from Hospitals A and B, respectively (Table 1).

Table 1.

Summary of subjects

Hospital	Subjects	Gender (Number of condyles)		Age (years)		Side (Number of condyles)
		Male	Female	Mean*	Range	Right	Left
Hospital A	with fracture	47	53	47.4	16–87	52	48
	without fracture	38	62	53.0	24–87	50	50
	Total	85	115	50.2	16–87	102	98
Hospital B	with fracture	61	39	61.0	16–88	50	50
	without fracture	42	58	56.4	16–88	50	50
	Total	103	97	58.7	16–88	100	100

*mean of age was calculated for condyles.

The distribution of fracture location and status of fracture fragment were evaluated according to the classifications modified from the reports of Lindahl¹⁶ and MacLennan.¹⁷ The fracture locations were divided into four parts: condylar head, upper neck, lower neck, and condylar base. The status of the fracture fragment was categorized into five types: no displacement, deviation, displacement, dislocation with deviation (deviation–dislocation), and dislocation with displacement (displacement–dislocation).

For the control group, 100 condyles without fractures confirmed by CT since October 2019 in 50 patients at both hospitals were abstracted retroactively using the following criteria: patients without symptoms at the TMJ area and patients without definitive deformity of the condyle. These patients were examined using CT mainly for extraction or pericoronitis of the mandibular third molars, benign tumors or cysts, or maxillary sinus diseases.

At Hospital A (193 condyles), panoramic radiographs were captured primarily using the Veraviewepocs X550 PCR (Morita Co. Ltd., Kyoto, Japan), with a tube voltage of 75 kV, tube current of 8 mA, and rotation time of 16.2 s. The remaining radiographs were captured using the AUTOIII-NTR (Asahi Roentgen Ind. Co. Ltd, Kyoto, Japan), with a tube voltage of 75 kV, tube current of 12 mA, and rotation time of 12 s. At Hospital B, radiographs were captured using the AUTOIII-NCM (Asahi Roentgen Ind. Co., Ltd, Kyoto, Japan), with a tube voltage of 75 kV, tube current of 12 mA, and rotation time of 12 s.

Image patch preparation

At Hospital A, panoramic radiographs were downloaded from the hospital database in JPEG format with a matrix size of 1186 × 1182 and standardized to 900 × 900 pixels. At Hospital B, radiographs were downloaded with a matrix size of 1722 × 1430 pixels and standardized to 900 × 900 pixels.

The rectangular image patches were cropped manually for the training and testing processes using Adobe Photoshop CS6 imaging software (Adobe Systems Co., Ltd., CA) according to the following definition (Figure 1). The upper end was set at the deepest part of the glenoid fossa. The lower end was set at the level of the occlusal plane or the midpoint of the mandibular ramus when the occlusal plane could not be determined, as was the case for patients with edentulism. The mesial end was a mandibular notch, and the distal end was the posterior end of the glenoid fossa. We created 200 (100 fracture, 100 control) image patches from the imaging data of each hospital.

Figure 1.

A rectangular ROI is cropped in a fracture’s left condylar area on a panoramic radiograph. ROI, region of interest.

DL architecture

We constructed the DL systems on Ubuntu OS v. 16.04.2 using a graphics processor unit (NVIDIA GeForce GTX 1080 Ti, NDIVIA, Santa Clara, CA) with 11 GB. The CNN used was AlexNet provided on the DIGITS Library v. 5.0 (NDIVIA, Santa Clara, CA; https://developer.ndivia.com/digits), which had five convolutional and three fully connected layers, and was used on the Caffe framework, also provided on the DIGITS library.

Construction and evaluation of the learning model

For each training process, 200 epochs were performed to construct a learning model. A fivefold cross-validation method was employed to evaluate the performance of the learning models (Figure 2). For data sets from Hospital A, 200 image patches were divided into five parts, each of which included 40 patches (20 fractures, 20 control patches), and were assigned in Aa, Ab, Ac, Ad, and Ae data sets. Next, in Fold 1, a learning model (Model A1) was constructed using four data sets (Ab, Ac, Ad, and Ae data sets), including 160 patches with training (130 patches) and validating (30 patches) data sets. In this process, data augmentation procedures were employed to increase the number of training data sets from 130 to 2340 patches using image processing software (Irfan View v. 4.44; http://www.Irfanview.com). The newly created Model A1 was tested using the Aa data set. For Fold 2, the data sets Aa, Ac, Ad, and Ae were used as training and validating data sets and Ab was used as the testing data set. The tests were assigned using the data sets obtained from the same hospital to those of the training and validating data sets and were used for the internal validity tests. These processes were repeated five times, with the training and testing data sets changed each time. To compare the learning model performance as the external validity test, the models were tested by applying the patches created from data obtained from the other hospital. For example, Model A1 created using data from Hospital A was tested with the Ba data set obtained using data from Hospital B. The learning models were constructed from the combined data sets from both hospitals, and the testing data sets were applied separately.

Figure 2.

Workflow of learning model construction and evaluation. The fivefold cross-validation method was modified and used. For each fold, a learning model was constructed based on four training data sets, including the validating data sets (black characters in white box), and the constructed model was evaluated by a testing data set from Hospital A. This process was repeated five times, with the training and testing data sets changed each time. Consequently, 200 testing image patches were evaluated, and these results were used for ROC analysis. Each model was also tested by a testing data set from Hospital B, and the results were used for ROC analysis. Similarly, using the data sets from Hospital B and combined data sets from Hospitals A and B, learning models were constructed and tested. ROC, receiver operating characteristic

The models provided the results of the testing processes as the probability (%) for the ground truth (presence or absence of fracture) for each image patch.⁵ The probability of the presence of a fracture corresponded to the true-positive fraction (sensitivity), and the probability of no fracture corresponded to the false-positive fraction (1 − specificity). Therefore, the receiver operating characteristic (ROC) curve could be described using the 200 results.

We calculated and compared the areas under the ROC curve (AUC). Using the ROC curves, we determined the cut-off values as the closest point on the ROC curve from the left upper corner of the graph for calculating sensitivity, specificity, and accuracy.

Statistical analysis

Differences in patient age between the two hospitals and between patients with and without fracture were tested using the t-test. Differences in AUC and distributions of gender, fracture location, and status of fracture fragment were evaluated using the χ² test. Statistically significant differences were considered for p-values of <0.05.

Results

Gender distribution was not different between Hospitals A and B for condyles both with and without fractures. A significant difference was found in the mean age of patients with fractured condyles between Hospitals A and B (t-test, p = 0.00004). The distribution of fracture location was different between patients from Hospitals A and B (χ², p = 0.026; Table 2). The fractured condyles were observed mostly in the upper and lower necks of patients from Hospitals A and B, respectively. The distribution of fractured fragment status was significantly different between Hospitals A and B (χ² test, p = 0.041; Table 3). Images from Hospital A showed many condyles without dislocation, whereas those from Hospital B had a high ratio of condyles with dislocation.

Table 2.

Location of fracture (number of condyles)

	Head	Upper neck	Lower neck	Base	Total
Hospital A	11	46	29	14	100
Hospital B	5	35	49	11	100

Distiribution of fracture location issignificantly different between the hosiptals with p = 0.026

Table 3.

Status of fracture fragment

	No displacement	Deviation	Displacement	Deviation-Dislocation	Displacement-Dislocation	Total
Hospital A	15	27	23	15	20	100
Hospital B	13	16	16	18	37	100

Distribution of fargment status issignificantly different between the hosiptals with p = 0.041

For internal validity, the models exhibited high performance, with AUC values of >0.85 when the testing data sets were applied to the learning models constructed using data sets from the same hospitals, as did the testing data sets (Figure 3, Table 4). Conversely, for external validity, the models exhibited low performance for data sets from different hospitals. When using the combined data sets from both hospitals, the models exhibited sufficiently high performance that was slightly superior or equal to the internal validity, without a statistically significant difference.

Figure 3.

ROC curves. The numbers 1 and 4 in the circle denote the results of internal validity tests. The number 2 and 3 show the results of external validity tests. The numbers 5 and 6 denote the results from the tests of the models constructed using the combined training data sets from Hospitals A and B. ROC, receiver operating characteristic.

Table 4.

peformance of learning modelsTable 4

Model (Origin of training and validating dataset)	Testing dataset	AUC	Cutoff value (%)	True positive(No. of condyles)	True negative(No. of condyles)	Accuracy (%)	Sensitivity (%)	Specificity (%)
Model A (Datatsets from hospital A)	Hospital A	0.85*	5.96	80/100	79/100	80.4	80.0	79.0
	Hospital B	0.58*	99.94	60/100	58/100	59.0	60.0	58.0
Model B (Datasets from hospital B)	Hospital A	0.58#	99.98	61/100	59/100	60.0	61.0	59.0
	Hospital B	0.86#	56.23	80/100	82/100	81.0	80.0	82.0
Model AB (Combined datasets from hospitals A and B)	Hospital A	0.89	7.00	83/100	80/100	81.5	83.0	80.0
	Hospital B	0.91	36.00	85/100	84/100	84.5	85.0	84.0

AUC, Area under the curve receiver operating characteristic curve; CNN, Convolution Neural Network.

*,#: Statistically significant difference with p value < 0.05.

Model A failed to classify 25 positive (i.e. with fracture) images from Hospital B; these were classified correctly by Models B and AB (Figure 4). Similarly, Model B could not correctly classify 24 fractured condyles from images from Hospital A; these were diagnosed accurately by Models A and AB (Figure 5).

Figure 4.

Image of a fracture of the left condyle from Hospital B. This image was correctly classified by Models B and AB but misclassified by Model A. The fracture fragment was dislocated anteriorly beyond the eminence. The black-lined box shows the cropped area for the testing process.

Figure 5.

Image of a fracture of the left condyle from Hospital A. This image was correctly diagnosed by Models A and AB but misdiagnosed as no fracture by Model B. The black-lined box shows the cropped area for the testing process.

Discussion

Internal validity was found to be sufficiently high, with AUC values of >0.85 for Models A and B when testing data sets from the same hospitals were applied. These AUCs and accuracies are equivalent to those reported by previous studies.^2,6–8 Model AB, which was trained using combined data sets from both hospitals, showed equal or slightly higher AUC and accuracy than those for internal validity. This result could be partially attributed to the inclusion of the training data sets from the same hospitals, as were the testing data sets, in addition to increasing the number of training data sets. Although the classification performance may be limited to the conditions employed in the present study, such as the number of training data sets (hundred order), pathology (condylar fracture), and diagnostic modality (panoramic radiography), it may not decline even when data sets from other hospitals are used in the training process. Additionally, these findings may indicate that an available DL system is improved by customization using a hospital’s own data sets.

To the best of our knowledge, no reports have verified external validity for panoramic radiographs. Although Lee et al¹⁸ collected approximately 7000 panoramic radiographs from three hospitals, the combined data and learning model that was constructed did not verify external validity. In the present study, when comparing external validity with internal validity, the performance appeared to be relatively low. This result might be related to the domain shift phenomenon, which should be considered when a model is constructed using data sets obtained from other institutions or a commercially available generalized model is actually used in the clinic.

Several researchers have investigated the external validity of CT,¹⁹ MRI,²⁰ and other roentgenographic imaging modalities^21,22 and have reported slightly low performances in comparison with internal validity. Kim et al²¹ used geographic external test sets (data obtained from a different institution) of Waters’ projection images for the diagnosis of maxillary sinusitis and reported an AUC of 0.88, which was lower than that of temporal external data sets (AUC = 0.93; data obtained from the same institution but in a different period). This relatively high performance in external validity might be a result of the large number (>80,000) of images used during the training process.

The present results show lower performances than those reported by previous studies. The small number of data sets in our study might be a reason. The domain shift phenomenon should also be considered. For example, some differences, such as patient age and apparatus used, were found between the data sets from the two hospitals. Among them, fracture characteristics were considered essential; therefore, we analyzed the distributions of fracture location and fragment status in the data and verified a significant difference. In data sets from Hospital A, fractures were frequently observed in the superior parts, whereas in data sets from Hospital B, they were observed predominantly in the inferior parts. Fractures without dislocated fragments were noted for Hospital A, whereas for Hospital B, fractures with dislocated fragments were noted in >50% of the fractured condyles. Therefore, Model A was likely to include misdiagnosis of dislocated fractured condyles, as shown in Figure 4. Conversely, for Model B, accurate classification of the fractured condyles without dislocation might be difficult, as shown in Figure 5. This difference was probably attributed to the difference in hospital characteristics, namely, Hospital A is an educational hospital and Hospital B is a general hospital with emergency outpatient services.

The present study has some limitations that can be addressed in future investigations. First, the data sets were too small, with data from only two hospitals used to generalize the results. A generalized versatile model requires hundreds of thousands or millions of data in the set. Although a solution is to use open-source data sets available on the Internet,¹⁴ no such data sets for panoramic radiographs are currently available. The collection of a substantial number of data sets from many hospitals would involve privacy and regulatory issues regarding personal information. A new approach using a federated learning system may compensate for this problem.¹⁴ The federated learning system, first introduced by Google in 2017,²³ enables the integration of multiple learning models without training data. These models are trained and created separately at several hospitals using their own patient data. Thereafter, only the models without personal data are collected and integrated at a central institution. For the clinical use of DL systems, their performance should be evaluated from the view of their generalization. In this regard, the present study was the first one to determine the external validity for diagnosis using panoramic radiography. In addition, only one CNN was used to construct the models. AlexNet, a basic and simplified CNN, was used because its performance was verified to be sufficiently high for panoramic diagnosis in our previous studies.⁵ However, future studies should evaluate other CNNs, especially pretrained or fine-tuned CNNs, which might improve performance. Finally, our results should be compared with those of human observers because the final responsibility for diagnosis should belong to humans, with the purpose of a CAD system being to assist them.

Conclusion

The DL system described in this study can be used to diagnose condylar fractures using panoramic radiographs as high classification performance was obtained, with AUC values of >0.85 when the data sets used for testing performance included internal data sets. However, we verified the domain shift phenomenon with lower external performance, and this issue should be considered for generalizing DL systems.