Cervical vertebral maturation assessment on lateral cephalometric radiographs using artificial intelligence: comparison of machine learning classifier models

Abstract

Objectives:

This study aimed to develop five different supervised machine learning (ML) classifier models using artificial intelligence (AI) techniques and to compare their performance for cervical vertebral maturation (CVM) analysis. A clinical decision support system (CDSS) was developed for more objective results.

Methods:

A total of 647 digital lateral cephalometric radiographs with visible C2, C3, C4 and C5 vertebrae were chosen. Newly developed software was used for manually labelling the samples, with the integrated CDSS developed by evaluation of 100 radiographs. On each radiograph, 26 points were marked, and the CDSS generated a suggestion according to the points and CVM analysis performed by the human observer. For each sample, 54 features were saved in text format and classified using logistic regression (LR), support vector machine, random forest, artificial neural network (ANN) and decision tree (DT) models. The weighted κ coefficient was used to evaluate the concordance of classification and expert visual evaluation results.

Results:

Among the CVM stage classifier models, the best result was achieved using the ANN model (κ = 0.926). Among cervical vertebrae morphology classifier models, the best result was achieved using the LR model (κ = 0.968) for the presence of concavity, and the DT model (κ = 0.949) for vertebral body shapes.

Conclusions:

This study has proposed ML models for CVM assessment on lateral cephalometric radiographs, which can be used for the prediction of cervical vertebrae morphology. Further studies should be done especially of forensic applications of AI models through CVM evaluations.

Introduction

Every individual has a unique growth pattern, and understanding the growth process is important in medicine and dentistry.^1–3 Chronological age is not a reliable skeletal maturation indicator.^4–6 Bone age is more accurate than chronological age in determining an individual’s maturation. Skeletal maturation is important for the determination of optimal treatment timing in dentofacial orthopaedics and forensic sciences.^7,8 Skeletal age can be assessed by hand-wrist radiographs.^2,4 Alternatively, cervical vertebral maturation (CVM) evaluation can be performed on lateral cephalometric radiographs, which are taken routinely in the orthodontic diagnostic procedures. Thus, the radiation dose can be reduced by elimination of the need for a hand-wrist radiograph.^7,9 Some studies have suggested that CVM can be used instead of hand-wrist maturation,^9–11 while others have reported that this method is subjective and should be supported by other biological indicators.^12,13

Artificial intelligence (AI) is a field of science and engineering concerned with the computational understanding of intelligent behaviour and creating systems that can perform cognitive tasks. Machine learning (ML) is a subfield of AI in which, rather than explicit programming of instructions, the machine learns how to accomplish a task by mathematical analysis of datasets provided.^14,15 In ML, each element describing the sample is called a feature vector. The process of converting raw data into a dataset by extracting the features is called feature engineering.¹⁶

In current technology, four different approaches can be adopted to teach an algorithm. In supervised learning, the dataset is labelled by experts and used by an algorithm to build a model. The final model uses labelled features as input and gives a desired output. In unsupervised learning, the dataset is not labelled, and the algorithm creates a model that transforms features into another vector or value to solve the problem. Semi-supervised and reinforcement learning are other learning methods.^14,16

Various tasks can be accomplished with different ML techniques. Choosing the appropriate technique requires a detailed analysis of the problem. Classification algorithms learn from labelled examples and produce a model for categorization of the unlabelled examples. Classification and regression techniques are associated with supervised learning. Clustering is associated with unsupervised learning and unlabelled data.^16,17 Several classification techniques can be used in ML (e.g., artificial neural network (ANN), logistic regression (LR), decision tree (DT), random forest (RF) and support vector machine (SVM)).

In the human brain, neuron cells are connected with synapses. When a certain excitation threshold is reached, electrochemical signals are transferred through neurons.¹⁸ An ANN is a mathematical model that emulates the biological process in the human brain that transfers electrochemical signals through connected neuron cells.^14,19 A typical neural network architecture has three neuron layers. The input layer is responsible for receiving data. Each feature is gained by a separate neuron in this layer. Hidden layers are intermediate layers between the input and output layers.^19,20 Architectures with one intermediate layer are called shallow neural networks, while architectures with more than one hidden layer are called deep neural networks (deep learning).^14–16,18 The output layer can consist of one or more neurons and transforms the received signal to the desired output. Apart from which layer they belong to, each artificial neuron has its input and output values. Each neuron takes the signal from every neuron in the previous layer as input and applies a mathematical function, and the transformed signal is transmitted to the neurons in the next layer as output.^14,19,20 The neurons in the adjacent layers are connected through synapses, each with its own adjustable weight. Synapses with a positive weight stimulate or activate the next inactive node, while synapses with negative weight inactivate or inhibit the next node (if it is active). ANNs are trained to discriminate important patterns in input data, process and respond with an appropriate output.^19–21

The mathematical formulation for LR is similar to that of linear regression, but the function is not straightforward. Despite the name, it is not a regression but a classification algorithm that works by determining the similarity of the observed sample with the sample in the model. The DT algorithm is similar to a tree, with branching nodes and leaves. There is a spesific threshold for each node and consecutive branches are followed until a leaf node is reached. Once a leaf node is reached, the sample is classified. In the RF method, DTs with slight differences are combined to form a structure like a forest. By doing so, instead of a result of a single tree, the final decision is achieved by the voting of each tree. SVM algorithms set up an imaginary high-dimensional space and place the samples in the appropriate locations according to their features. Samples distributed in this imaginary space are then separated by a hyperplane, resulting in the classification of the data.^16,17

AI tools can be implemented into software to assist users in the decision-making process. Clinical decision support systems (CDSS) are computer-based tools to assist healthcare professionals in the process of clinical decision-making. Such systems can be developed with mathematical modelling tools and medical data processing techniques.^22,23 There are several studies in the literature about CVM analysis. Some of them are about the comparison of expert results statistically, while others include evaluation of data derived from radiographs and making predictions using AI methods.^4,5,20,24,25 The performance of different supervised ML classifiers has not been compared in previous studies.

This study aimed to compare the performance of five different classifier models developed using AI techniques (LR, DT, RF, ANN and SVM), which can be used for a potential bone age assessment software, for the purpose of achieving more standardized results in clinical applications.

Methods and materials

Ethical considerations

This retrospective study was granted ethical approval by the Ethical Committee of Suleyman Demirel University.

Patient selection

A total of 647 digital lateral cephalometric radiographs of individuals with chronological age between 10 and 30 years (mean age ±standard deviation: 15.36 ± 4.13) were selected from the archive of Suleyman Demirel University, Faculty of Dentistry, Department of Dentomaxillofacial Radiology. The age and gender distributions of the patients are shown in Table 1. Patients without congenital or acquired malformation of the cervical vertebrae, and radiographs with good visualization of the C2, C3, C4 and C5 vertebrae, were included. The gender and chronological age of each sample were saved. The chronological age was calculated by subtracting the birth date of individuals from the date of service. Selected radiographs were exported with Romexis (v.3.5, Planmeca, Helsinki, Finland) software in JPEG format. CVM evaluations were performed according to the method described by Baccetti et al.⁷ This study was conducted in seven steps, summarised in Table 2.

Table 1.

Age and gender distributions of the patients

		Frequency	Percent (%)
Age	10	52	8.0
	11	66	10.2
	12	81	12.5
	13	77	11.9
	14	53	8.2
	15	35	5.4
	16	54	8.3
	17	43	6.6
	18	42	6.5
	19	33	5.1
	20	34	5.3
	21–29	77	11.9
	Juvenile	461	71.3
	Adult	186	28.7
Gender	Male	304	47
	Female	343	53

Table 2.

Workflow of the study in seven steps. Steps 2, 3 and 4 were performed for the development of the CDSS. The dataset was labelled in step 5 and classified in step 6

Study design	Involved observers (N)	Labelling software	CDSS	Evaluated radiographs (N)	Task	Aim
Step 1	-	-	-	-	Development of the data labelling software	Image feature extraction and CVM analysis with CDSS
Step 2	2	-	-	100	Independent visual evaluation	Development of the CDSS
Step 3	2	Used	Off	100	Marking off the anatomical landmarks independently	Development of the CDSS
Step 4	-	-	-	100	Statistical analysis of the data obtained by Step two and Step 3	Development of the cervical vertebral morphology detection algorithm of the CDSS
Step 5	1	Used	On	647	Marking off the anatomical landmarks and CVM analysis with the CDSS	Labelling of the dataset in text format to be used as input for the models
Step 6	-	-	-	498/149 (Training /Testing)	Development of the ML models with five different techniques	Classification of the dataset according to the labels
Step 7	-	-	-	149	Analysis of the agreement between human and AI models with weighted κ	Comparison of the different models’ performances

CVM, Cervical Vertebral Maturation; CDSS, Clinical Decision Support System; κ, κ;

N, Number

Step 1—Development of the image labelling software

New software was developed for manual extraction of the image features of digital lateral cephalometric radiographs with marking of the anatomical landmarks. CVM evaluations for the dataset were done with the CDSS integrated into the software. The labelling software was developed using the C# programming language. The radiographs are shown with a 1:1 size ratio. Users had access to pan, negative and contrast and brightness adjustment tools. The distance between two calibration points could be changed. Keyboard shortcuts were created to make changing the marked points more practical. The system runs on the Windows (Microsoft, USA) platform. Image features were extracted in text format by the software.

All borders of the cervical vertebral body should be equal for the shape of a square.⁷ Due to the accuracy of measurements made with markings in the digital environment, this equality was difficult to achieve. Therefore, the subset of 100 randomly selected images (chronological age between 10 and 19, consisting of 10 radiographs for each age) was evaluated in steps 2 and 3, and the results of both steps were combined in step 4 to establish the new cervical vertebral morphology detection algorithm. The CDSS was activated only in step 5.

Step 2—Visual evaluation of the subsample (for CDSS)

The subsample of 100 radiographs was visually evaluated by two independent observers. The radiographs are shown with Romexis software with a 1:1 size ratio. Users had access to pan, negative and contrast and brightness adjustment tools. The presence of concavity under the C2, C3, C4 and C5 vertebrae was recorded using the following categories:

True: Visible concavity at the lower border of the vertebrae.

False: No visible concavity at the lower border of the vertebrae.

C3, C4 and C5 vertebrae body shapes were saved as the following categories:

Trapezoid: Posterior border longer than the anterior, and the superior border tapered.

Horizontal rectangular: Posterior and anterior borders equal, and horizontal borders longer than vertical.

Square: All vertebrae borders equal.

Vertical rectangular: Posterior and anterior borders equal, and horizontal borders shorter than vertical.

Step 3—Labelling of the subsample (for CDSS)

After visual evaluation of the subset of images, the same subset was imported to the labelling software. The goal of this step was to create an algorithm for the cervical vertebral body shape detection to be used in the CDSS. Therefore, the CDSS was not activated during this step. The anatomic landmarks (Figure 1, Table 3) were marked on 100 digital lateral cephalograms by the same observers, independently.⁷ Two more points were marked on the radiopaque ruler inside the nasal apparatus for calibration. Three ratios were calculated to determine vertebral body shapes with the CDSS.

Figure 1.

Anatomical landmarks marked on digital lateral cephalograms.

Table 3.

Anatomic landmarks used to determine cervical vertebrae morphology

Points	Definition*
C2up, C3up, C4up, C5up	The most posterior point on the upper border
C2um, C3um, C4um, C5um	The middle point on the upper border
C2ua, C3ua, C4ua, C5ua	The most anterior point on the upper border
C2lp, C3lp, C4lp, C5lp	The most posterior point on the lower border
C2lm, C3lm, C4lm, C5lm	The deepest point of the concavity at the lower border
C2la, C3la, C4la, C5la	The most anterior points on the lower border

^*:The definition is demonstrating the point on C2, C3, C4, and C5 vertebrae, respectively

• Base anterior ratio (BAR): The ratio between the inferior length and anterior height of the vertebral body. The length of the inferior border was measured linearly using marginal points at the inferior border.

• Posterior anterior ratio (PAR): The ratio between the posterior and anterior height of the vertebral body.

• Base posterior ratio (BPR): The ratio between the inferior length and posterior height of the vertebral body. The length of the inferior border was measured linearly using marginal points at the inferior border.

Step 4—Creation of the CDSS

The results of the last two steps were analysed statistically and a new approach was conducted for determination of the cervical vertebrae morphology. The flowchart of the algorithm used in the CDSS for cervical vertebrae morphology detection is shown in Figure 2. The threshold for the presence of concavity at the inferior border of the vertebrae (the distance of the deepest point of concavity to an imaginary line traced between most posterior and anterior points of the inferior border) was determined as a depth of 1 mm.

Figure 2.

Flowchart of the vertebrae body shape detection algorithm used in CDSS (N: No, Y:Yes)

CVM stage suggestions were made as described by Baccetti et al.^7,26 This is represented in Table 4, with conditions for each maturation stage shown in the columns. C5 morphology was not taken into consideration when identifying the CVM stage. Growth is a continuing process, and the maturation stage of an individual does not change overnight. It is expected and mentioned in the literature that cervical vertebrae do not always ‘follow the rules’ during the maturation process.⁴² Samples that did not fit any group were classed as ‘unidentified stage’. CVM stages of these examples were determined by an expert's evaluation of the overall radiograph. The CDSS was activated after this step.

Table 4.

Cervical vertebrae morphological features of each maturation stage

Feature	Vertebrae	CS 1	CS 2	CS 3	CS 4	CS 5	CS 6
Concavity	C2	-	+	+	+	+	+
	C3	-	-	+	+	+	+
	C4	-	-	-	+	+	+
Body Shape	C3	T	T	T* or RH	RH* or T	S* or RH	RV* or S
	C4	T	T	T* or RH	RH* or T	S* or RH	RV* or S

RH, Rectangular horizontal; RV, Rectangular vertical; S, Square; T, Trapezoid.

^-,Absence of concavity; +, Presence of concavity; CS, Cervical Stage

^*:Characteristic vertebral body shape of the stage, at least one of C3 or C4 body shape must be the characteristic

Step 5—Creating the dataset with the labelling software

The dataset was manually labelled in text format for supervised ML classifier models. A total of 647 digital images were imported to the developed software and the anatomical points marked on each sample. After all points had been marked, the integrated CDSS of the software made suggestions according to the morphology detection algorithm. Suggested CVM stage and morphological characteristics of the vertebrae, such as the presence or absence of concavity and vertebrae body shape, could be accessed via the user interface (Figure 3). In case of a conflict with the software, the examiner could change the proposed stage or morphology that was saved for the dataset. In total, 54 image features and results of CVM evaluation with CDSS were saved to be used with the AI models (Supplementary Material 1).

Figure 3.

Screenshot of the newly developed labelling software with integrated CDSS, demonstrating the marked points, CVM stage and morphology suggestions.

Step 6—Development of the ML models

The labelled dataset was separated into training and testing samples with a ratio of 498:149 (approximately 80%:20%). LR, SVM, RF, ANN and DT models were developed with the ‘Keras’, ‘scikit-learn’, ‘NumPy’ and ‘pandas’ libraries using the Python programming language (Python Software Foundation, https://www.python.org). Different models were designed for CVM stage and vertebrae morphology classification. RF models for CVM stage and vertebrae body shape classification consisted of 80 trees and 10 trees for the concavity model. SVM models were established with LinearSVC. The ANN models for CVM stage and vertebrae body shape classification were conducted for 120 epochs and both models consisted of one hidden layer using the Softmax activation function. The ANN model for concavity classification was run for 100 epochs and the Softplus activation function was used without hidden layers. The Adam optimization algorithm was used to optimize the learning rate in all ANN models.

Step 7—Statistical analysis

Spearman correlation coefficients were calculated to determine the relationship between chronological age and the CVM stage determined by the examiner. The results of five different supervised classifier models were compared with the result of the examiner with the CDSS. Weighted κ coefficients and percentage of the agreement were calculated using SPSS (IBM, Chicago, USA) and the ‘STATS_WEIGHTED_KAPPA’ add-on (https://github.com/IBMPredictiveAnalytics/STATS_WEIGHTED_KAPPA).

Results

The distribution of the CVM stages determined by the researcher’s visual analysis with the CDSS is shown in Table 5. The Spearman correlation coefficients between determined CVM stages and chronological age were found to be 0.607 for both genders: 0.734 for males and 0.395 for females, respectively (p < 0.01). Weighted κ coefficients for intraobserver agreement were found to be 0.686 for CVM staging, 0.72 for vertebrae body shapes and 0.734 for concavities.

Table 5.

Distribution of the CVM stages determined by visual analysis with CDSS

CVM Stage	Frequency	Percent (%)
CS1	109	16,8
CS2	62	9,6
CS3	77	11,9
CS4	148	22,9
CS5	137	21,2
CS6	114	17,6

CS, Cervical Stage; CVM, Cervical Vertebral Maturation.

For the CVM staging models, the highest agreement between the researcher’s visual analysis with the CDSS and classification results was found to be with the ANN model (κ = 0.926), followed by the DT model (κ = 0.921), while the lowest agreement was found to be with the LR model (κ = 0.866; Table 6 and 7).

Table 6.

The percentage of agreement between the results of visual analysis with CDSS and classification results of five different models for each CVM stage

%	CS1	CS2	CS3	CS4	CS5	CS6	Model’s average
LR	90.91	76.19	66.67	78.38	75	85	78.69
SVM	90.91	76.19	88.89	72.97	82.5	75	81.08
RF	95.45	95.24	66.67	91.89	80	65	82.38
ANN	90.91	90.48	88.89	83.78	87.5	80	86.93
DT	100	85.71	66.67	72.97	95	95	85.89
Stage’s Average	93.64	84.76	75.56	80.00	84.00	80.00	82.99

CS, Cervical Stage; DT, Decision tree; LR, Logistic regression; RF, Random forest, ANN, Artificial neural network; SVM, Support vector machine.

For the cervical vertebrae morphology models, the compatibility of the classification results with the researcher is shown in Table 7. For the classifications made with these models, the mean concordance was found to be 97.55% for the presence of concavity and 90.65% for vertebral body shape.

Table 7.

Weighted κ coefficients for the cervical vertebrae morphology models

Weighted κ coefficients	LR	SVM	RF	ANN	DT
CVM staging	0.866	0.874	0.908	0.926	0.921
Presence of concavity	0.968	0.964	0.895	0.952	0.933
Vertebrae body shape	0.847	0.858	0.937	0.912	0.949

CVM, Cervical Vertebral Maturation; DT, Decision tree;LR, Logistic regression; RF, Random forest, ANN, Artificial neural network; SVM, Support vector machine.

The accordance of each model in determining each different CVM stage is shown in Table 6. On average, the highest concordance was seen at the CS1 stage (93.64%).

Discussion

The growth process is different for each individual and can be affected by several different factors. Understanding growth events is important in diagnosis and treatment planning in craniofacial orthopaedics.^2,3,6 Evaluation of CVM on lateral cephalometric radiographs can reduce the radiation dose to the patient by eliminating the need for additional radiography.^7,9 Researchers have reported several methods to evaluate CVM.²⁷ The method described by Baccetti et al is common in the literature and its clinical application is simple.²⁸ CVM analysis is inherently subjective and can be influenced by the practitioner’s experience.¹² Studies have also reported low reproducibility of the method.²⁹ Tajmir et al reported that AI improves radiologists’ bone age assessment by increasing accuracy and decreasing variability. It was found that the use of AI by the radiologist is superior to either AI alone, a radiologist alone or a pooled cohort of experts.³⁰ We believe that AI applications will improve the standardization of diagnostic radiology in the future.

CVM assessment is reportedly possible with the use of a thyroid protective collar.²⁶ However, the radiopaque projection of the collar on C4 was observed in some of the samples. Since evaluation of the C4 vertebra is necessary for the method described by Baccetti et al, the use of thyroid protective collar was excluded from the present study. With the exclusion of the collar from the study, C5 became visible and was included in the study.

Patients’ gender was recorded; however, this information was not included in the inputs. The CVM method described by Baccetti et al is reported to be suitable for all genders.⁷ Gender was recorded according to the demographic profiles of the individuals but not used for the assessment of vertebral maturation.

In this study, different models were developed for CVM staging and cervical vertebral morphology classification tasks. The results of such classifications should be utilized separately. For CVM staging, the morphology of each vertebra was not determined and only the final stages were given by the AI models. For cervical vertebrae morphology determination, only the morphology of the cervical vertebrae was obtained by the AI models.

Several studies in the literature have performed CVM analysis using software. Baptista et al proposed a method for pattern classification to predict the CVM stage. The developed naïve Bayes classifier model achieved a weighted κ coefficient of 0.861, and 188 images were included in their study. If adjacent stages were taken to be acceptable, a new weighted κ coefficient was reported as 0.992.²⁴ Sokic et al used centroid-based clustering models to determine the CVM stage. K-means clustering and fuzzy C-means clustering techniques were used to classify 211 radiographs. The best results were achieved using the modified fuzzy C-means clustering model. Agreement with the researcher was around 70%, while it was over 99% if adjacent classes were accepted.⁴ Dzemidzic et al developed the Cephalometar HF V1 software and evaluated the consistency of results with the data obtained by expert analysis. In their study, the Cohen’s κ coefficient was found to be 0.985 between the researcher and the software.⁵ Padalino et al compared manual and computerized analyses of CVM, and concordance between these two methods was found to be 94% for the resident and 93% for the student.²⁰ Santiago et al developed software for quantitative analysis of CVM using a multinomial LR model. They used 236 lateral cephalometric radiographs of children, and the weighted κ coefficient was found to be 0.832 between the groups classified by model and hand-wrist maturation stages.²⁵

ML models adapt themselves to the dataset to perform the given task. The performance of the model depends on the nature of the task and the distribution of the samples in the dataset. LinearSVC is capable of performing multiclass classification on a dataset. New software with a CDSS for CVM analysis was developed for feature extraction from digital lateral cephalometric images. A semi-automatic method was adopted and proper CVM stage and vertebrae morphology were determined by the last decision of the user. The dataset was labelled in text format and classified using five different supervised ML models. The classification results were compared with the results of visual analysis with CDSS. The number of the evaluated radiographs was increased and more algorithm models developed compared with previous studies. The new approach was conducted for determination of cervical vertebrae morphology. The highest weighted κ coefficient between CVM stage models and labelled data was achieved using the ANN model (κ = 0.926, 86.93%).

The absence of hand-wrist radiographs can be described as a limitation of this study. However, our study focused solely on CVM analysis using lateral cephalometric radiographs and evaluation of the reliability of this method was not taken into consideration. Additionally, the dataset was prepared by the single observer’s evaluation. The CDSS was developed with the data from two different observers’ evaluation results to make the computer results more objective.

Conclusion

This study has proposed AI models for CVM assessment on lateral cephalometric radiographs, which can be used for the prediction of cervical vertebrae morphology. Each AI model is unique and proper hyperparameters should be optimized according to the dataset. The ANN model showed the best performance of the five different models developed. Further studies should be done, especially of forensic applications of AI models through CVM evaluations.