Osteoporosis screening support system from panoramic radiographs using deep learning by convolutional neural network

Abstract

Objectives:

This study was performed to develop computer-aided screening systems that could predict osteoporosis. The systems were constructed using panoramic radiographs of women aged ≥ 50 years through three types of deep convolutional neural networks (CNNs): Alexnet, VGG-16, and GoogLeNet; the performances of the constructed systems were evaluated.

Methods:

One oral radiologist classified 1500 panoramic radiographs into three types. In C1, the endosteal margin of the cortex was smooth and sharp, whereas porosities were observed in C2 and C3. The risks of osteoporosis were higher in C2 and C3 than in C1; C3 had the highest risk. This information was included with the images as training data; three CNNs were transfer trained. Using each trained CNN, the diagnostic accuracy was assessed using panoramic radiographs and bone mineral density inspection findings in the lumbar spine and femoral neck of 100 additional patients.

Results:

All CNNs exhibited relatively good agreement with the oral radiologist’s judgement (86.0%–90.7%). The predictive results of the three systems for osteoporosis of the lumbar spine showed sensitivities of 78.3%–82.6%, specificities of 71.4%–79.2%, and accuracies of 74.0%–79.0%. The predictive results for osteoporosis of the femoral neck showed sensitivities of 80.0%–86.7%, specificities of 67.1%–74.1%, and accuracies of 70.0%–75.0%.

Conclusions:

The constructed systems were generally more accurate than the previously developed conventional system. The new systems may facilitate osteoporosis prediction and prevent subsequent fractures by encouraging patients with suspected osteoporosis to undergo further inspections (e.g., dual-energy X-ray absorptiometry) and treatment.

Introduction

Osteoporosis is defined as a systemic skeletal disease that is characterized by low bone mass and microarchitectural deterioration of bone tissue, which leads to greater bone fragility and fracture susceptibility. ¹ Particularly in postmenopausal older females, the secretion of female sex hormones such as estrogen is markedly reduced; this results in a rapid systemic decrease in bone mineral density (BMD) and contributes to the onset of osteoporosis. ^2,3 Worldwide, approximately 200 million women have osteoporosis. ⁴ Currently, the risk of osteoporosis is greatest in North America and Europe; however, the risk is expected to increase in other countries along with population-level increases in lifespan. Notably, more than 12 million people in Japan have osteoporosis; this number is gradually increasing. ⁵ Klemetti et al and Taguchi et al reported that the porosity of the mandibular inferior cortical bone, immediately below the mental foramen on panoramic radiographs, was significantly negatively correlated with the lumbar BMD. ^6,7 Since the publication of these reports, morphological changes related to linear bone resorption of the inferior cortical mandibular bone in panoramic radiographs have been used for osteoporosis screening worldwide. ^8–11 Multiple studies have shown that these morphological changes on panoramic radiographs could be automatically examined through the use of computer image processing, enabling the identification of osteoporosis. ^12–16 From the results of previous studies, such image processing methods had a degree of diagnostic accuracy. ^12,15,16 However, images of the hyoid bone and ghost images of the contralateral mandibular ramus may appear near the lower mandibular edge on panoramic radiographs; the presence of such images can interfere with correct diagnosis. Furthermore, false-positives might occur because slight linear bone resorption is recorded as cortical bone porosity.

Osteoporosis screening techniques using panoramic radiography reveal porosity in the inferior margin of cortical bone; they also reveal the mandibular cortical bone width, ¹⁷ panoramic mandibular index (i.e., ratio of mandibular cortical bone thickness to the distance between the mental foramen and the inferior mandibular margin), ¹⁸ and mandibular ratio (i.e., total mandibular height divided by the height from the center of the mental foramen to the inferior mandibular border). ¹⁹ Aliaga et al proposed a method to automatically identify the necessary sites and connecting lines to determine the mandibular cortical bone width, panoramic mandibular index, and mandibular ratio; however, this method requires complex image processing methods. ²⁰

In recent years, deep-learning artificial intelligence (e.g., deep convolutional neural network [CNN] architecture) has been widely recognized as an effective method for classifying and learning the characteristics of features directly from medical images. ^21,22 In particular, Alexnet, ²³ Visual Geometry Group (VGG)−16, ²⁴ and GoogLeNet ²⁵ are widely used as pre-trained CNNs that can learn from large numbers of general images. These CNNs are presumably unable to correctly identify medical images without further training that focuses on the images to be classified. Correct identification can be achieved by performing CNN transfer learning that involves a large amount of data, in which the images to be classified are combined with interpretations by a diagnostic imaging specialist. In addition to medical images, dental panoramic radiographs have been subjected to image assessment by deep-learning artificial intelligence using CNNs; those CNNs have identified diseases of the maxillary sinus, ^26,27 cystic radiolucency, ²⁸ and the relationship between the third molar and the mandibular canal. ²⁹ The results suggest that diagnostic findings can be identified with a degree of accuracy. Therefore, the use of deep-learning artificial intelligence that involves CNN architecture for assessment of osteoporosis in panoramic radiographs may yield better diagnostic accuracy than the conventional image processing approach. Additionally, the conventional screening system based on mathematical morphological image processing ^12,15 can only detect linear bone resorption in cortical bone; it cannot classify resorption severity, according to reports by Klemetti et al and Taguchi et al. ^6,7 Such assessment of severity may be achieved by proper CNN transfer learning. Therefore, this study was performed to develop screening support systems that could predict osteoporosis based on panoramic radiograph findings using three types of CNNs: Alexnet, VGG-16, and GoogLeNet. The performances of the constructed systems were evaluated by comparing the mean BMD values of lumbar spine and femoral neck among three image types (described in the Methods); the diagnostic accuracies of the constructed systems were calculated on the basis of osteoporosis in the lumbar spine and femoral neck.

Methods and materials

Morphological classification of the inferior mandibular border on panoramic radiographs

Figure 1 shows the cortical bone morphology classification of the mandibular inferior margin into three types (C1, C2, and C3), based on the works by ⁶ and Taguchi et al. in 1996. ^6,7 Skeletal BMD is expected to be lower in C2 and C3 than in C1. C3 is considered more severe than C2. These three classifications were applied to existing CNN architectures.

Figure 1.

Classification of mandibular inferior cortical appearance. (C1) the endosteal margin of the cortex is even and sharp on both sides. Patients classified in this class are expected to have normal skeletal BMD. (C2) the endosteal margin shows semilunar defects (lacunar resoption) and/ or seems to form endosteal cortical residues on one or both sides. Patients classified in this class may have low skeletal BMD. (C3) the cortical layer forms heavy endosteal cortical residues and is clearly porous. Patients classified in this class are more likely than class two to have low skeleral BMD.

Participants and regions of interest in panoramic radiographs

Conducting this research using data obtained from the participants has been approved by our University Clinical Research Ethics Committee. In this study, panoramic radiographs from 1500 female patients aged ≥50 years (mean age±standard deviation: 65.5 ± 9.98 years) who visited our department between 2014 and 2021 were used for transfer learning in three CNNs (Alexnet, GoogLeNet, and VGG-16). These subjects were defined as Group A. These three CNNs were pre-trained using a large number of general images from the ImageNet database. ³⁰ Separately, panoramic radiographs from 100 female patients aged≥50 years (mean age±standard deviation: 64.5 ± 8.14 years) who visited our department between 2009 and 2012 were used to measure diagnostic accuracy after transfer learning. These subjects were defined as Group B; they were a subset of the participants in our previous study. ¹⁵ The patients in group B underwent bone density examination of the lumbar spine and femur via dual-energy X-ray absorptiometry (DXA) on the same day that panoramic radiographs were acquired. Lumbar lateral radiographs showed that the patients in group B had no compression fractures or other abnormalities in the lumbar spine. Group B images were used to evaluate CNN performance after transfer learning. All patients in groups A and B had no history of metabolic bone diseases and had not used medications that could affect bone metabolism. Table 1 shows the results of BMD measurements (via DXA) at the lumbar spine and femoral neck for patients in group B. Based on the World Health Organization classification, the presence of osteoporosis was defined as a BMD T-score of ≤ −2.5. Appropriate regions of interest (ROIs) including the inferior cortical bone immediately below the mental foramen were extracted from all images. In a previous study, the mean coordinates of the mandibular inferior cortical bone just below the mental foramen were calculated from 100 panoramic X-ray images, separate from the present subject. ¹⁵ The mean coordinates were used as reference coordinates. In the present study, a 256 × 256-pixel area was extracted from the center of this reference coordinate system and used as the ROI. Because the coordinates were on both left and right sides, the ROI on the right side-was extracted from the inverted image. In total, 3000 ROIs were extracted bilaterally from 1500 panoramic X-ray images. An oral radiologist with 21 years of clinical experience visually classified all images in these ROIs as types C1, C2, or C3, then labeled the results for all 3000 ROIs. After classification, the numbers of ROIs classified as types C1, C2, and C3 were 1771, 759, and 470, respectively. The results of this classification were assigned to each ROI image as training data. Panoramic X-ray images of patients in group A were acquired using AUGE (Asahi Roentgen, Kyoto, Japan); images of patients in group B were acquired using Cypher (Asahi Roentgen, Kyoto, Japan). Both X-ray machines were able to collect images with comparable quality. The resolution of all images was 2860 × 1536 pixels.

Table 1.

Result of bone density measurements of group B

	The second to fourth lumbar vertebrae bone density		Femoral neck
	BMD T-score≥−2.5SD	BMD T-score<−2.5	BMD T-score≥−2.5SD	BMD T-score<−2.5
Number of Participants	77	23	85	15
Average BMD (g/cm2)	1.06 ± 0.13	0.755 ± 0.07	0.743 ± 0.09	0.524 ± 0.09

BMD: Bone Mineral Density.

These subjects were a part of those used in our previous study. ¹⁹

Transfer learning in the CNN architectures

Ninety percent (2700) of all ROIs in training data from group A were used as training samples; the remaining 10% (300) of ROIs were used as validation samples. Training samples were used for transfer learning in the three CNN architectures. A schematic of each architecture is shown in Figure 2. The images of all ROIs were resized to ensure suitable input for each CNN (e.g., 224 × 224 pixels for Alexnet). In CNN transfer learning, sequential learning is repeated many times; this method involves training using a small subset of image data, followed by training using all images. The small subset is known as a mini-batch. The number of times that all training samples are iteratively trained is known as the epoch. In this study, 50 images were used as a mini-batch; one epoch was defined as 54 parameter updates, which indicated that all image data had been updated once. The training samples were repeatedly trained to obtain high accuracy; thus, the number of epochs was set multiple times. In this study, transfer learning was completed at 20 epochs. Using the training and validation samples, training accuracy and validation accuracy were measured to verify the accuracy of agreement with visual classification results (for 2700 training samples and 300 validation samples, respectively). As transfer learning progresses, both accuracies are expected to increase. The training accuracy is usually greater than the validation accuracy, but the difference is expected to gradually decrease as transfer learning progresses. All image processing and CNN training were conducted using MATLAB2021a, and Deep Learning Toolbox (MathWorks, Inc. Natick, MA, USA). An excessive number of epochs may cause overlearning; when this occurs, the validation sample cannot be classified correctly and the validation accuracy decreases. Because MATLAB can draw a learning curve, that shows plotting training accuracy and validation accuracy that change as learning by each mini-batch and epoch progresses, in real time as the epoch progresses. Thus, overlearning can be prevented by monitoring both accuracies.

Figure 2.

Schematic diagram of three CNN models

Evaluation of diagnostic performance of the new screening support systems and assessment of the relationship between classification results and BMD

Our newly developed screening support systems were used to diagnose the 100 patients in group B. The diagnostic accuracy was evaluated by comparing the results of the new systems with the results of osteoporosis diagnosis based on BMD measurements (via DXA) of the lumbar spine and femoral neck. In our previous studies, patients with eroded mandibular cortical bone (e.g., types C2 and C3) had low skeletal bone density and a high risk of osteoporosis. ³¹ Thus, the systems classified types C2 and C3 images as suspected osteoporosis and type C1 images as normal bone density. If the results of the new screening systems differed between the left and right sides, the result of the more severe side-was recorded. The mean BMD was calculated for each type of image (C1–C3) classified by the systems; differences among types of images were evaluated using the Mann–Whitney U test. All comparisons were two-sided, and <i>p-values < 0.05 were considered statistically significant. To evaluate diagnostic performance, the sensitivity, specificity, positive predictive value, negative predictive value, and accuracy were calculated using images of group B. Additionally, our conventional image processing system ¹⁵ was used to classify images of group B; the accuracy of this conventional classification was compared with the accuracy of our new CNN systems. SPSS v 11.0 statistical software (SPSS Inc., Chicago, IL, USA) was used for all statistical analyses.

Result

Overall, CNN architectures were developed that applied transfer learning to Alexnet, VGG-16, and GoogLeNet; this yielded new screening support systems. During transfer learning based on Alexnet, VGG-16, and GoogLeNet, both training and prediction accuracies improved with epoch progression. Figure 3 shows the learning curves for Alexnet, VGG-16, and GoogLeNet. The vertical axis shows the accuracy, while the horizontal axis shows the number of epochs and number of iterations. For each iteration, the raw training accuracy was plotted, the training accuracy smoothed using a moving average filter, and the validation accuracy. Transfer learning was completed for all CNNs without any loss of learning accuracy related to overlearning. After the completion of transfer learning, the validation accuracies were 86.0% for Alexnet, 89.7% for VGG-16, and 90.7% for GoogLeNet. The results of the classification of group B images by each system are shown in Table 2. For all systems, C1 was the most common classification, while C3 was the least common classification. Figure 4 shows the results of comparing the mean BMD of the lumbar spine and femoral neck among images classified as C1, C2, and C3. Significant differences were found between C1 and C2, and between C1 and C3, for all CNN-based systems. The results of diagnostic accuracy assessment are shown in Tables 3 and 4.

Figure 3.

The learning curves for Alexnet, VGG-16, and GoogLeNet

Table 2.

Classification of mandibular morphology in Group B images by 3 CNNs

	C1	C2	C3
Alexnet	59	35	6
VGG-16	66	26	8
GoogLeNet	64	25	11
Conventional	55	45a

^aConventional systems could only determine whether osteoporosis was suspected or not. Therefore, the total number of C2 and C3 was indicated.

Figure 4.

The results of comparing the average BMD of the lumbar spine and femoral neck between the C1 to C3 classified by the system

Table 3.

Diagnostic performances of the systems for lumbar osteoporosis

	Sensitivity % (95% CI)	Specificity % (95% CI)	PPV % (95% CI)	NPV % (95% CI)	Accuracy % (95% CI)
Alexnet	82.6 (75.2–90.1)	71.4 (62.5–80.3)	46.3 (36.5–56.3)	93.2 (88.3–98.1)	74.0 (65.4–82.3)
VGG-16	78.3 (70.2–86.4)	79.2 (71.2–87.2)	52.9 (43.1–62.7)	92.4 (87.2–97.6)	79.0 (71.0–87.0)
GoogLeNet	82.6 (75.2–90.1)	77.9 (69.8–86.0)	52.8 (43.0–62.6)	92.4 (87.2–97.6)	79.0 (71.0–87.0)
Conventional system	82.6 (75.2–90.1)	66.2 (56.9–75.5)	42.2 (32.5–51.9)	92.7 (87.6–97.8)	70.0 (61.0–79.0)

PPV: Positive predictive value; NPV: Negative predictive value; CI: Confidence interval

Table 4.

Diagnostic performances of the systems for femoral neck osteoporosis

	Sensitivity % (95% CI)	Specificity % (95% CI)	PPV % (95% CI)	NPV % (95% CI)	Accuracy % (95% CI)
Alexnet	86.7 (80.0–93.4)	67.1 (57.9–76.3)	31.7 (22.6–40.8)	96.6 (93.0–100.0)	70.0 (61.0–79.0)
VGG-16	80.0 (72.2–87.8)	74.1 (65.5–82.7)	35.3 (25.9–44.7)	95.5 (91.4–99.6)	75.0 (66.5–83.5)
GoogLeNet	86.7 (80.0–93.4)	72.9 (64.2–81.6)	36.1 (26.7–45.5)	96.9 (93.5–100.0)	75.0 (66.5–83.5)
Conventional system	80.0 (72.2–87.8)	61.2 (51.6–70.8)	26.7 (18.0–35.4)	94.5 (90.0–99.0)	64.0 (54.6–73.4)

PPV: Positive predictive value; NPV: Negative predictive value; CI: Confidence interval

Discussion

The systems developed in this study had better performance than the conventional system in terms of specificity, positive predictive value, and accuracy. Conventional systems tend to produce false positives in the presence of minimal linear bone resorption. Furthermore, conventional systems must avoid the hyoid bone if it is superimposed on (or close to) the mental foramen, which is the target ROI. The new systems were less affected by the hyoid bone; they were able to evaluate the area directly below the mental foramen. Figure 5 shows an image of a patient in group A where the classification agreed with the oral radiologist’s judgment, despite the superimposition of hyoid bone. In the conventional system, such images would have generated a diagnosis of suspected osteoporosis despite other characteristics consistent with a C1 classification, unless the overlapping hyoid bone had been avoided.

Figure 5.

An example of an image where the hyoid bone is superimposed on the ROI

Lee et al reported that VGG-16 can be used to predict osteoporosis status from panoramic X-ray images on the basis of transition studies and fine tuning. ³² However, the potential for classifying the severity of mandibular cortical bone erosion (e.g., C1 to C3) was unclear. This study showed that the severity of C1 to C3 could also be assessed with a certain degree of accuracy.

Furthermore, consistent with the findings of previous studies, ^7–11 patients whose images were classified as C2 and C3 had lower BMD in both the lumbar spine and femur, compared to patients whose images were classified as C1. Therefore, patients whose images are classified as C2 or C3 should be encouraged to undergo more comprehensive examinations (e.g., DXA) and, if necessary, begin drug treatment; this guidance may help to prevent osteoporosis and subsequent fractures. Notably, the VGG-16 and GoogLeNet-based systems showed no significant differences in BMD of the lumbar spine and femoral neck between patients according to image classification (i.e., C2 and C3); the differences were particularly small in the femur. The low number of patients (15.0%) with a BMD T-score of ≤−2.5 at the femoral neck may have contributed to this lack of significant differences; this low number of patients may also have contributed to the low positive predictive value of the femoral neck (31.7%–36.1%) as results of diagnosis by the new systems. The ROI of type C3 images used as training data from group A was smallest (15.7%) compared to the number of C1 and C2 images. Therefore, the systems may have been unable to learn sufficient features of C3 images, compared to C1 and C2 images. If the training data are unbalanced, classification into the less common class will presumably be incorrect. ³³ Performance might be improved by increasing the number of patients whose images were classified as C3, or by weighting and specifically training the C3 images. ³³ Additionally, there are large numbers of CNNs available in the public domain and the most suitable CNN for a particular study is unclear; presumably, the use of additional CNNs will enable the identification of more suitable networks in future studies. Singh et al reported that BMD could be predicted with a degree of accuracy by CNN transfer learning focused on the inferior mandibular margin morphology on panoramic X-ray images. ³⁴ In their work, the training data included both age and panoramic radiographs; they reported that the inclusion of age as training data enhanced accuracy. Therefore, use of the lower mandibular border classification and age as training data might further improve the accuracy of our systems. Sukegawa et al also constructed an ensemble model that adds patient clinical factors, such as age, body mass index, and sex, to the deep learning analysis of X-ray images. ³⁵ Such patient covariates may provide additional information regarding important osteoporosis classifications, and improve the accuracy.

Importantly, all training data in this study were based on the judgment of a single oral radiologist; however, the results of C1–C3 assessment may differ among skilled oral radiologists. Therefore, training data might be improved if the C1–C3 assessment is performed by several well-trained oral radiologists.

Conclusion

Osteoporosis screening support systems based on panoramic X-ray images were developed using CNN transfer learning; the performances of our constructed systems were superior to the performance of a system based on conventional image processing. Therefore, our new systems could help to reduce osteoporosis-related fractures by encouraging patients whose images are classified as C2 or C3 to undergo more comprehensive examinations (e.g., DXA) and, if necessary, begin drug treatment.

The convolution layer filters the original image and outputs a feature map. The normalization layer is a layer that returns the result of normalizing the input based on the input distribution. The pooling layer reduces the original image while retaining important information as features. In the fully connected layer, the image data from which the feature parts have been extracted through the convolution and pooling layers are combined into a single node, and the values transformed by the activation function are output. The softmax function is one of the activation functions and is used as an output layer for multiclass classification. The concatenation layer takes inputs and concatenates them along a specified dimension.