Convolutional neural network-based automatic cervical vertebral maturation classification method

Abstract

Objectives:

This study aimed to develop a fully automated artificial intelligence-aided cervical vertebral maturation (CVM) classification method based on convolutional neural networks (CNNs) to provide an auxiliary diagnosis for orthodontists.

Methods:

This study consisted of cephalometric images from patients aged between 5 and 18 years. After grouping them into six cervical stages (CSs) by orthodontists, a data set was constructed for analyzing CVM using CNNs. The data set was divided into training, validation, and test sets in the ratio of 70, 15, and 15%. Four CNN models namely, VGG16, GoogLeNet, DenseNet161, and ResNet152 were selected as the candidate models. After training and validation, the models were evaluated to determine which of them is most suitable for CVM analysis. Heat maps were analyzed for a deeper understanding of what the CNNs had learned.

Results:

The final classification accuracy ranking was ResNet152>DenseNet161>GoogLeNet>VGG16, as evaluated on the test set. ResNet152 proved to be the best model among the four models for CVM classification with a weighted κ of 0.826, an average AUC of 0.933 and total accuracy of 67.06%. The F1 score rank for each subgroup was: CS6>CS1>CS4>CS5>CS3>CS2. The area of the third (C3) and fourth (C4) cervical vertebrae were activated when CNNs were assessing the images.

Conclusion:

CNN models proved to be a convenient, fast and reliable method for CVM analysis. CNN models have the potential to provide automatic auxiliary diagnostic tools in the future.

Introduction

Bone age evaluation is very important to orthodontists in many aspects, such as the choice of treatment methods, ¹ the decision of treatment timing ² and the judgment of recurrence trend. ³ Handwrist radiographs were originally used for bone age assessment, and it is considered a reliable and reasonable method. ⁴ In orthodontics, Lamparski first proposed that bone age can also be evaluated by observing changes in the size and shape of the cervical vertebrae in 1975. ⁵ Since then, the results of the cervical vertebral maturation (CVM) method have been shown to be consistent with those of the handwrist method (HWM) in many studies, and the former may serve as an alternative to the latter. ^6,7 Moreover, HWM requires an additional radiograph in orthodontic practice, thus raising the patients’ undue concerns about the extra but quite negligible radiation doses, while the CVM method is performed on lateral cephalometric radiographs and is routinely available for the orthodontic examination which could eliminate the extra radiation exposure.

However, assessing the CVM is a complex and time-consuming process that can only be conducted by experienced specialists, even though there was an improved version of the CVM method proposed by Baccetti based on the second (C2), third (C3), and fourth (C4) cervical vertebrae in 2003. ⁸ The study by Flavio Uribe indicated that the accuracy of CVM classification is extremely sensitive to the clinical experience of the specialist, ⁹ so mastery of this method takes a prolonged time. Attempts should be made to develop auxiliary diagnosis methods to reduce the clinical burden.

Artificial intelligence (AI) is a subfield of computer science that aims to create systems that can perform cognitive tasks that are ordinarily performed by humans. ¹⁰ In recent years, computer vision technology based on convolutional neural networks (CNNs) has recently made remarkable achievements in the field of medical imaging. ¹¹ Tasks like image classification, object detection and semantic segmentation can be solved by this technology. In dentistry, CNNs are also widely used in many fields, ^12,13 such as cephalometric measurements, ^14,15 tooth detection, ¹⁶ temporomandibular disorder diagnosis ¹⁷ and caries detection ^18,19 CVM analysis is also an ideal target for CNN technology which takes pictures as input and outputs specific labels. Shin et al have proved that Tanner-Whitehouse 3 (TW3)-based fully automated bone age assessment system using CNN can be effectively utilized for HVM evaluation. ²⁰ Therefore, it can be assumed that the CNN models may be helpful in classifying CVM stages.

Thus, this study aimed to develop a fully automated AI-aided CVM classification method to provide an auxiliary diagnosis to aid orthodontists.

Methods and materials

The present research was a retrospective study. All data were processed to eliminate personally identifiable information in accordance with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The study was approved by the ethics committee of Peking University Hospital of Stomatology and the approval number is PKUSSIRB-202054025.

Data collection

A total of 6079 cephalometric images were collected from Peking University Hospital of Stomatology between 2017 and 2020. Only those samples that fulfilled the following criteria were included in the study: the patient medical records were complete, cephalometric images were qualified, and the patients were less than 18 years old. The exclusion criteria were as follows: the presence of congenital syndromes, metabolic diseases, diseases affecting growth and development, and history of special drugs. The patients were 2576 males and 3503 females, aged from 5 to 18 years. The age and sex distribution of the patients are shown in Figure 1.

Figure 1.

Age, sex, and cervical stages distribution of the enrolling patients.

The cephalometric images were routinely taken in three scanners of the same type, Veraviewepocs 2D (J Morita Corp, Kyoto, Japan), with the following parameters: scanning time 4.9 s, tube current 5–10 mA; tube voltage 90 kV. All the images were stored and read in JPG format.

Data processing

The rectangular region consisting of C2, C3, and C4 was manually cropped from the cephalograms to remove the influence of other characteristics. To facilitate the transmission of pictures to the CNN model, the pictures were resized to 224 × 224 pixels while keeping the picture scale unchanged.

The total data were divided into the training, validation, and test data sets, respectively in the ratio of 70, 15, and 15% (Table 1). The training data set was used to train the CNN models. The validation data set was used to tune the hyperparameters of CNNs. The test set was used to evaluate the performance of the model. The flowchart of the experimental process of this study is shown in Figure 2.

Table 1.

Numbers of patients assigned to training, validation, and test sets of the six CVM subgroups

	CS1	CS2	CS3	CS4	CS5	CS6	Total
Train set	715	712	701	703	709	713	4253
Validation set	153	153	150	151	152	153	912
Test set	154	153	151	151	152	153	914
Total	1022	1018	1002	1005	1013	1019	6079

CVM, cervical vertebral maturation.

Figure 2.

Flowchart of the experimental process. CVM, cervical vertebral maturation

Data labelling

To label the data, two experienced orthodontists examined the images manually and grouped them into six stages: cervical vertebral 1 (CS1), cervical vertebral2 (CS2), cervical vertebral 3 (CS3), cervical vertebral 4 (CS4), cervical vertebral 5 (CS5) and cervical vertebral 6 (CS6), with reference to the user guide delivered by McNamara. ²¹ For images whose classification the two orthodontists did not agree on, a third orthodontist with extensive experience was consulted to come to a final decision.

Convolutional neural network selection

Due to the lack of experience in applying CNN to CVM analysis, four classical CNNs were selected as candidates for our study:VGG16, ²² GoogLeNet, ²³ DenseNet161, ²⁴ and ResNet152. ²⁵ The high performance of these CNNs was validated in ImageNet Large Scale Visual Recognition Competition. ²⁶ The above four models had been trained in the present study and their performance on the test set was compared to identify the most suitable model for the CVM analysis.

Data augmentation

To prevent overfitting, the following data augmentation techniques were applied in this study: random translation within a quarter of the width of the image, random rotation in the range of 15° clockwise and 15° counter-clockwise, and adaptive histogram equalization. In each epoch of training, 50% of the training data set was randomly selected for data augmentation.

Strategy and hyperparameter set of training

The strategy of transfer learning and fine-tuning techniques were used in this study. The pre-trained models on the ImageNet data set of the four candidate CNNs were downloaded from Pytorch Zoo (https://pytorch.org/). After initializing, the networks with the pre-trained models, the output layer of the model was changed into six outputs so that it could correspond to the six stages of the CVM method. Because the characteristics of the medical image are quite different from those of the general database, the parameters of the whole CNN layers were fine-tuned. The networks were trained using a Stochastic Gradient Descent (SGD) optimizer for 100 epochs with a mini-batch size of 64. Hyperparameters of the CNNs were adjusted manually according to model performance on the validation set in order to maximize the classification capabilities of each model. After testing various times, the individualized hyperparameter combination was set as follows: initial learning rate = 0.001; momentum = 0.09; weight decay = 0.01. Our training was performed on the server of the computing platform of ******* university, with NVDIA Tesla P100 graphic processing unit.

Model testing and evaluation metrics

For each of the four CNNs, the model with the lowest loss on the validation set was selected to verify its final classification performance on the test set. To evaluate the classification performance of the CNN models, the following indicators were used to test the model performance: accuracy, precision, recall rates, F1 score and confusion matrix. The receiver operating characteristic (ROC) curves were also plotted and the area under the curve (AUC) was calculated.

For a deeper understanding of the learned model, a heat map was generated using class activation mapping (CAM). ²⁷ This map visually highlights the regions which are mostly informative in distinguishing the CVM classification.

Statistical analysis

The weighted κ statistic was used to verify the consistency of the model’s output with researcher annotation results. The classification results of the four CNNs were compared using McNemar’s χ² analysis. SPSS software (Windows v. 27.0; SPSS Inc. Chicago, IL) was used to perform data analysis.

Results

The model size and training results of the CNN models are shown in Table 2. The accuracy and loss values for each epoch during training are displayed in Figure 3. Although the VGG16 model had the most parameters, it had a worse training effect. Even though there were only 21.5 MB parameters in GoogLeNet, the training results were quite good. The training results of ResNet152 and DenseNet161 were very close to each other and better than those of VGG16 and GoogLeNet.

Table 2.

The size, max train accuracy and max validation accuracy of the four candidate models

	Model size (MB)	Max train accuracy	Max validation accuracy
VGG16	512	67.34%	63.04%
GoogLeNet	21.5	64.90%	63.37%
DenseNet161	102	68.04%	66.22%
ResNet152	222	68.51%	67.32%

CNN, convolutional neural network.

Model size: Total parameter size in the CNN model; Max train accuracy: The biggest accuracy on the train set among all the epochs. Max validation accuracy: The biggest accuracy on the validation set among all the epochs.

Figure 3.

Training results for the four candidate CNN models. (a) Training results of VGG16 (b) training results of GoogLeNet (c) training results of DesneNet161 (d) training results of ResNet152. The horizontal axis of the graph represents the training epochs. The left vertical axis of the graph represents the classification accuracy and the right vertical axis of the graph represents the loss value. CNN, convolutional neural network.

The results of weighted κ, average AUC and total accuracy which were used to compare the final classification performance of the four CNNs are given in Table 3. The weighted κ statistic showed that the classification results of the four CNNs were in good consistency with the researcher annotation results (all the weighted κ >0.7, p < 0.001). The ranking of classification accuracy, average AUC and weighted κ were as follows: ResNet152>DenseNet161>GoogLeNet>VGG16. ResNet152 proved to be the best among the four models (McNemar’s chi-square test: ResNet152 vs DenseNet161, p = 0.024; ResNet152 vs GoogLeNet, p = 0.026; ResNet152 vs VGG16, p < 0.001), with a weighted κ of 0.826, average AUC of 0.933, and total accuracy of 67.06%. The ranking of the inference speed for the four models was as follows: GoogleNet>VGG16>DenseNet161=ResNet152.

Table 3.

Classification performance of the four candidate CNNs on the test set

	Weighted κ	Average AUC	Total accuracy	Inference time
VGG16	0.796***	0.920	61.15%	0.26 s
GoogLeNet	0.811***	0.926	64.11%	0.083 s
DenseNet161	0.818***	0.928	64.22%	0.32 s
ResNet152	0.826***	0.933	67.06%	0.32 s

AUC, area under the curve; CNN, convolutional neural network.

Weighted κ: *p < 0.05, **p < 0.01, ***p < 0.001

The classification results of ResNet152 on the test set are shown as a confusion matrix in Figure 4a. The confusion matrix showed a good classification result. By observing the color and number distribution of each cell, it can be intuitively found that the data were mainly distributed on the diagonal cell or around the diagonal cell. The model always tends to achieve the correct classification; even when the final classification is incorrect, the model tends to a neighboring rather than distant category from the correct one. The precision, recall rates, and F1 score of each cervical stage were calculated according to the confusion matrix, and the results are shown in Table 4. CS2 and CS3 phases had the lowest precision, recall rates and F1 score, CS4 and CS5 phases were in the middle rank, while CS1 and CS6 phases had the highest precision, recall rates and F1 score. The ROC curves for each cervical stage are shown in Figure 4b and the AUC was also calculated.

Figure 4.

(a) Confusion matrix obtained using ResNet152 model on test set. (b) ROC curves of ResNet152 model for the cervical stages. ROC, receiver operating characteristic.

Table 4.

Precision, recall rates and F1 score of ResNet152 on test set for each CVM subgroup

	Precision	Recall	F1 score	Sample size of each subgroup
CS1	0.74	0.79	0.77	154
CS2	0.52	0.52	0.52	153
CS3	0.59	0.56	0.58	151
CS4	0.73	0.66	0.69	151
CS5	0.66	0.64	0.65	152
CS6	0.77	0.84	0.81	153

Precision = TP/TP +FP; Recall = TP/TP +FN; F1 score = 2*Precision * Recall/Precison + Recall. TP is true CVM, cervical vertebral maturation; Precison, Recall and F1 score are calculated as follows with confusion matrix in Figure 3.positive, FP is false positive, FN is false negative, and TN is true negative.

Figure 5 shows the heat maps created using the CAM. The method activated the significant areas which influenced the diagnostic results in the inference process of the CNN models. It shows the features that the trained model learned to perform the CVM analysis. In this study, the activation area of the neural network was on the areas of C3 and C4. The CAM was highly consistent with the regions on which orthodontists performed the CVM classification judgments.

Figure 5.

Representative class activation map of the correct classification in each cervical stage.

Figure 6 shows the process of inference using the ResNet152 model. The output of the neural network is the confidence that the image belongs to each category. The index corresponding to the highest confidence is the predicted label of the CNN model.

Figure 6.

Representative demo of inference process using the CNNs. The red text is the confidence of the ResNet152 model prediction. CNNs, convolutional neural networks

Discussion

In orthodontics, bone age is more frequently used than chronological age for growth prediction because the latter is easily influenced by gender, genetic characteristics, environmental factors, etc. This can also be proved in this research by the fact that the chronological age and bone age could overlap as is shown in Figure 1. HWM has been proved to be a mature method in determining bone age, but it is complained by the patients as exposed to extra radiation. CVM is also recommended in orthodontics with its relative facility, despite it is not as precise as HWM. The CVM method is not easy to master because it relies heavily on clinical experience. ²⁸ The present study tested an automatic classification method based on CNN to assist orthodontics in the decision-making process for CVM analysis. To find a suitable CNN model for CVM analysis, four CNN models were trained using the transfer learning method. Upon comparing the performance of the four models on the test set, ResNet152 proved to be the best model for our data set. The model exhibited a weighted κ of 0.826 and an average AUC of 0.933 compared to the researcher’s diagnosis. Besides, the time for ResNet152 model to analyze a single cephalogram is only about 0.32 s. CNN model proved to be an effective and fast method for CVM stage classification, demonstrating its potential as an AI-aided diagnostic tool.

For better application of CNNs algorithm in our research, four CNNs with different structures were compared for CVM classification. The ranking of the classification accuracy for CVM classification was as follows: ResNet152>DenseNet161>GoogLeNet>VGG16. By reviewing the literature, it could be seen that the structure of CNNs was updated and improved from VGG16 to ResNet152. ^22–25 Correspondingly, the classification accuracy of these CNNs on our data set is getting better and better. This reminds us that the progress of the computer algorithm can improve the accuracy of medical diagnosis.

As a representative algorithm of deep learning, the CNNs can effectively learn relevant features from a large number of images by convolution calculation and backpropagation algorithms, ²⁹ avoiding the process of manual feature extraction. It is more efficient and simpler compared to traditional machine learning algorithms in medical image analysis. ¹¹ Previous studies have used other machine learning algorithms for CVM analysis. For example, Pisa proposed a method for CVM assessment by Naïve Bayes algorithm with 188 lateral cephalograms as early as 2012 ³⁰ and Amasya developed an artificial neural network model for CVM analysis with 647 lateral cephalograms in 2020. ^31,32 However, due to the limitations of the selected algorithm, they only achieved semi-automated functions, meaning that all these methods required orthodontists to extract the features manually by depicting anatomical landmarks of the cervical vertebrae, whereas the CNN algorithm in the present study requires little involvement of orthodontists. CVM analysis based on CNNs is more intelligent and could reduce the workload for orthodontists.

The neural network is often described as a black box because it does not provide any feedback as to why and how it performs its predictions. CAM technology is one of the approaches for investigating the features neural networks learn during the training process. ^27,33 It highlights the discriminative regions used by CNN to identify the special categories. In the present study, the area of C3 and C4 was activated when CVM was assessing the images. CNN utilized features similar to those used by orthodontists, but some tiny features were ignored, such as the concavity in the inferior border of the second vertebra. ⁸

In this study, it is believed that there were two special points in applying CNN models to medical data. The first one is transfer learning, which is widely used in the medical imaging AI field. Transforming the experience learned from general pictures into medical imaging is necessary at present because it can help the CNN model to converge faster even if there is not enough data. ³⁴ It is very suitable for medical data which are usually relatively small and difficult to collect. The second point is data augmentation. As a common method to prevent overfitting in deep learning, data augmentation can help the CNN model extract more information from the training data set, thereby narrowing the gap between the training set and the validation set. ³⁵ Notably, selecting personalized data augmentation methods is important for medical data sets because general data augmentation methods may add unnecessary variance to highly structured medical data sets. ³⁶

The limitation of this study is that the trained model only achieved an accuracy of 67.06% on the test set, and the F1 scores of CS2 and CS3 were only about 0.5, which suggests that the algorithm needs further improvement. Three reasons are thought to be responsible for the limitation, and the first one is the quality and quantity of the dataset. Even though the sample size of this study was much larger than those of previous studies, it was still relatively small for deep learning. Besides, the quality of data labelling is severely affected by the subjective nature of CVM analysis. ²⁸ Establishing a standard database based on multicentres and more experts is a feasible solution to improve the model performance in the future. The second reason is that the CVM staging is sometimes imprecise due to the gradual changes in size and shape of the three vertebral bodies that occur over time, especially for the maturational stages that last a shorter length of time such as C2 and C3. The third reason lies in the CNN algorithm. The CAM technology shows that the CNN algorithm used in this study could not find some special features related to cervical stages, such as the concavity in the inferior border of the second vertebra. Creating a customized CNN model based on the existing network structure may be the solution to this problem. This needs future systematic work to develop a better CNN model for CVM analysis, which is beyond the scope of this work.

Conclusion

The present study demonstrated the effectiveness of applying CNN in CVM classification as an automatic AI-aided diagnostic tool that can assist orthodontists. CNN models could provide a convenient, fast and reliable CVM diagnosis in clinical practice. Future work should focus on creating a customized network structure and constructing a standard database that is still needed to enhance model performance.