Neural networks for classification of cervical vertebrae maturation: a systematic review

Abstract

Objective

To assess the accuracy of identification and/or classification of the stage of cervical vertebrae maturity on lateral cephalograms by neural networks as compared with the ground truth determined by human observers.

Materials and Methods

Search results from four electronic databases (PubMed [MEDLINE], Embase, Scopus, and Web of Science) were screened by two independent reviewers, and potentially relevant articles were chosen for full-text evaluation. Articles that fulfilled the inclusion criteria were selected for data extraction and methodologic assessment by the QUADAS-2 tool.

Results

The search identified 425 articles across the databases, from which 8 were selected for inclusion. Most publications concerned the development of the models with different input features. Performance of the systems was evaluated against the classifications performed by human observers. The accuracy of the models on the test data ranged from 50% to more than 90%. There were concerns in all studies regarding the risk of bias in the index test and the reference standards. Studies that compared models with other algorithms in machine learning showed better results using neural networks.

Conclusions

Neural networks can detect and classify cervical vertebrae maturation stages on lateral cephalograms. However, further studies need to develop robust models using appropriate reference standards that can be generalized to external data.

INTRODUCTION

Artificial intelligence (AI) has powered voice-activated personal assistants and self-driving cars from the pages of science fiction to reality. A similar incursion into medicine has seen AI used to diagnose cancer, predict survival, simulate the spread of disease, visualize musculoskeletal tissue, and predict hospital attendance.1–5 Since one of its first uses in medicine as a rule-based expert system that advised antimicrobial therapy,6 medical applications of AI have grown tremendously. In the oral health sciences, reviews explain the use of AI in dentistry, orthodontics, dental imaging, and cleft lip and/or palate.7–10

A popular form of AI is machine learning, which provides computers knowledge through data and observations without explicit programming. Such systems improve with experience and more data. Extensive digital data help create machine-learning models with very high accuracy. A subset of machine learning, deep learning, has greater flexibility and can extract abstract features from raw data.11 It relies on multiple processing layers to detect features in a hierarchical structure.12 In medical imaging, machine-learning models can analyze chest radiographs to detect numerous lesions.13 They can make comprehensive radiologic examinations, sometimes outperforming resident doctors.14–16

Neural networks (NNs) are deep-learning algorithms that structurally emulate the neural connections in the human body.17 Each perceptron (the basic unit) receives one or more input from a previous layer. Each input value has a weight (or strength) assigned to it. The total input is a sum of all input values subject to the weights assigned to each. This is processed via a mathematical function to provide an output from the perceptron.18 This can be a final output value or an input value for the next layer of the NN.

An NN typically has an input layer, one or more hidden layers, and an output layer. The input layer has one or more perceptrons, and each receives one input. In subsequent layers, one perceptron may get multiple inputs. The input variables are scaled to a value ranging between the assigned all and none values (such as between 0 and 1) in the input layer. This is in contrast to a binary input of all or none (i.e., 0 or 1). These graded values are processed by the network to provide a graded output.18 Therefore, the output is ordinal data, making NNs a good choice for multicategory classification problems.

Lateral cephalograms can be used to determine the peak pubertal growth of the jaws depending on skeletal maturity.19 Individuals treated with functional orthopedic appliances before attaining skeletal maturity achieve optimal growth and develop a harmonious relationship between the jaws.20 These appliances are cost-effective, in contrast to surgical repositioning at later stages, which, additionally, has a high morbidity. Skeletal maturity is determined by a technique initially explained by Lamparski21 and later modified by Hassel and Farman.22 They used changes in the shape of the second to fifth cervical vertebrae for assessment. This method was modified further by Baccetti et al.23 to include the shape of the inferior border of individual vertebrae and limited to the fourth cervical vertebrae as they are seen with a protective radiation collar, and the stage can be defined based on a single radiograph.

A drawback of the (CVM) method is the need for specialized training that is not provided to general dentists, and orthodontists may also find it difficult to distinguish the shapes of vertebrae.24 Thus, patients who can benefit from treatment with myofunctional appliances may not get timely referral to a specialist and ultimately need surgery later. AI can detect features that may not be obvious to human observers. Automating or augmenting the process of detecting cervical vertebrae maturation stages with AI can help referral of suitable patients for orthopedic or myofunctional therapy and train or aid residents in diagnosis. The aim of this review was to assess the accuracy of identification and/or classification of the stage of cervical vertebrae maturity on lateral cephalograms by NNs compared with the ground truth determined by human observers.

MATERIALS AND METHODS

This systematic review was conducted according to the PRISMA guidelines. The question posed was, “What is the accuracy of neural networks in detecting cervical maturation stages on lateral cephalometric radiographs compared to human observers?”

Search Strategy and Study Selection

A search of PubMed (MEDLINE), Scopus, Embase, and Web of Science was conducted in November 2021. The queries used to search each database are listed in Table 1. The results obtained from each database were exported to EndNote online (Clarivate Analytics). After duplicates were removed, two reviewers (Dr Mathew and Dr Awadh) independently screened the remaining articles based on their titles and abstracts. Potentially relevant studies and those with insufficient information were selected for a full-text reading by two independent reviewers (Dr Alshehri and Dr Padala). Studies that satisfied the inclusion criteria were selected for data extraction.

Table 1.

Search Queries for Each Database

S.No	Database	Query	Results
1	Embase	(‘cervical vertebra' OR ‘bone age' OR ‘bone maturation') AND (‘artificial intelligence'/exp OR ‘artificial intelligence' OR ‘machine learning':jt OR ‘neural network':au)	96
2	Web Of Science	(artificial intelligence OR machine learning OR neural network) AND (cervical vertebrae OR cervical bone OR skeletal maturity)	109
3	Scopus	ALL ( artificial AND intelligence OR machine AND learning OR algorithm ) AND ( cervical AND vertebrae OR skeletal AND maturation OR maturation OR maturity )	154
4	PubMed	((((((artificial intelligence) OR (machine learning)) OR (neural network))) AND ((cervical vertebrae) OR (skeletal maturity))) AND (((((artificial intelligence) OR (machine learning)) OR (neural network))) AND ((cervical vertebrae) OR (skeletal maturity)))) AND (radiography OR cephalogram)	66

Inclusion Criteria

• Population: Lateral cephalograms with varying patient age that captured the cervical vertebrae

• Intervention: Using an NN to identify or classify the stage of skeletal maturity with inputs from a lateral cephalogram

• Comparator/reference: A human observer's identification and classification of the cervical vertebrae to assess skeletal maturity based on radiographic characteristics

• Outcome: Accuracy of the automated classification of cervical vertebral stages using an NN compared with a human observer

• Study type: Studies that compare the use of automated diagnostic systems with the predetermined human observer results were considered; experimental studies, randomized controlled trials, prospective and retrospective observational studies, case-control studies, and cohort studies were included

Exclusion Criteria

Conference papers, abstracts, opinion pieces, literature or systematic reviews, and studies that used any form of AI other than NNs were not included. Studies with full text unavailable in English were excluded.

Data Extraction

Two reviewers (Dr Alshehri, Dr Thomas) independently chose the studies eligible for data extraction according to the inclusion criteria, regardless of the qualitative assessment. Details from the studies were extracted and summarized in tabular form.

Methodologic Assessment

The review focused on the accuracy of diagnosing CVM stages on radiographs by NNs with the diagnosis by human observers as the reference. Therefore, the Cochrane tool for diagnostic test accuracy, the latest version of the Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2), was used for the methodologic assessment of studies.25 The assessment was carried out by two independent reviewers (Dr Palatinus, Dr Bhandi). The results were discussed among both reviewers, and a third reviewer was consulted for arbitration.

RESULTS

An initial search that was adapted according to each database provided 425 articles across all searched databases. The search queries and results are displayed in Table 1. The initial screening of the title and abstract by both reviewers was in almost perfect agreement (Cohen's Kappa = 0.93). Disagreements were arbitrated by a third reviewer (Dr Patil). After screening, 11 articles were considered for full-text evaluation; finally, eight articles were found eligible for inclusion in the review.26–33 The reviewers were in complete agreement. Three articles that were excluded either did not use NNs or did not report on accuracy.34–36 The PRISMA flow diagram is shown in Figure 1. The eligible studies were selected for data extraction and qualitative assessment. Most studies published the development of an NN model for classifying cephalograms according to the stages of CVM. One study compared the assessment of lateral cephalometric radiographs by a previously described model to four human observers. The data obtained from the studies are presented in Table 2.

Figure 1.

PRISMA flow chart of study selection.

Table 2.

Summary of Data Extracted From Selected Studies^a

S.No	Author, Year	Country	Age Range, y	Sample Size	CVM Method Used	Inputs	Reference Standards/Comparisons	Outcome
1	Kim et al., 202126	Korea	6–18	Training: 600 images Testing: 120 images	Baccetti et al.	Images of lateral cephalograms equally distributed across stages	Two specialists	The combination of the CNN with a region-of-interest detector and segmentor module was significantly more accurate (62.5%) than without them.
2	Seo et al., 202127	Korea	6–19	600 lateral cephalograms	Baccetti and Franchi	Cropped images of lateral cephalograms equally distributed across stages displaying the inferior border of C2 to C4	One radiologist	A pretrained network, Inception-ResNet-v2, had relatively high accuracy of 0.941 ± 0.018 when adapted. It also had the highest recall and precision scores among all pretrained models tested.
3	Amasya et al., 202028	Turkey	10–30	72 images	Baccetti and Franchi	Manually labeled image data set with 54 features and ratios with equal distribution across stages	Three dentomaxillofacial radiologists and an orthodontist	Interobserver agreement between researchers and the ANN model was substantial to almost perfect (wκ = 0.76–0.92). Percentage agreements between the ANN model and each researcher were 59.7%, 50%, 62.5%, and 61.1%.
4	Amasya et al., 202029	Turkey	10–30	647 images (498 for training and 149 for testing)	Baccetti and Franchi	Manually labeled image data set with 54 features and results of the evaluation by a clinical decision support system	Expert visual evaluation	Percentage agreement between the model and the visual analysis of the researcher was 86.93%, which was the highest among all models tested.
5	Kök et al., 202030	Turkey	8–17	419 individuals	Hassel and Farman	Measurements used in different combinations for seven neural networks	Human observer's classification	Highest classification accuracy was obtained from the model that used all 32 measurements and age as inputs. The overall accuracy was 94.2% for this model on the test data set.
6	Kök et al., 202031	Turkey	8–17	360 individuals	—	Measurements on the second, third, fourth, and fifth vertebrae used in different combinations as inputs for four models	Human observer's classification	Highest accuracy obtained with one of the neural networks was 0.95 when the training and test data were split into a ratio of 70%:30%.
7	Kök et al., 201932	Turkey	8–17	300 individuals	Hassel and Farman	Linear measurements performed on second, third, and fourth cervical vertebrae	Orthodontist	The neural network model had the second highest accuracy values for determining individual stages, except the fifth stage second-highest accuracy values (93%, 89.7%, 68.8%, 55.6%, 47.4%, and 78%) but was the most stable among all algorithms tested.
8	Makaremi et al., 201933	France	Not mentioned	1870 cephalograms	Baccetti and Franchi	Cropped images without filters and cropped images processed with mean, median, and entropy filter	Human observer	The pretrained models were not as effective as the neural network made by the researchers. The accuracy of the neural network did not exceed 90% on test images. The accuracy improved with more images and preprocessing with the entropy filter.

^aANN indicates artificial neural network; CNN, convolutional neural network.

Data Sets

The number of samples varied from 72 to 1870 among the studies. The studies used locally sourced data sets of different sizes when developing the models. The data were obtained from university settings; however, Makaremi et al.33 did not mention the source. The data sets were split into the training and testing data in five studies.26,28,30,31,33 The training data set was used to develop and validate the model. The age range of the data in all studies was adequate to include all of the CVM stages. Six studies used equal distribution of training data across stages.26,27,29–33

Reference Standards

The human observers who classified the lateral cephalograms used two methods: the Hassel and Farman method and the method modified by Baccetti et al.22,37 The experience of the human observers and their field of expertise varied. Interobserver agreement was calculated in two studies and was poor in one study.26,29 Five studies used a single expert's judgment as a reference standard.27,28,30–32

Input Features

Five studies used manually labeled data sets with measurements.28–32 Three used the image of the radiograph as input.26,27,33 Measurements used as inputs were linear measurements in the vertical and horizontal directions and ratios derived from them. One study that used photographs cropped the region of interest on the lateral cephalograms and used filters on the image.33 One study devised a region-of-interest detector and a segmenter module for the photographs to be used with the NN instead of manual measurements.26

Neural networks

Three studies used previously trained convolutional NNs (a type of NN) for identification of radiographs.26,27,33 Two modified them to suit the input provided.26,27 Six studies developed an NN specifically to classify radiographs.28–33 Makaremi et al.33 used both pretrained networks and developed a NN to classify radiographs. The number of hidden layers and perceptrons in the hidden layers of the networks varied. The pretrained networks adapted for this task had more than 50 layers depending on the network. The NNs devised by the researchers solely for the classification task had one hidden layer or simple six-layered architectures.

Performance of the NNs

The accuracy with which the systems could classify the radiographs differed. Accuracy was the agreement between the NN and the reference standard. The studies that developed new models showed an overall accuracy of greater than 90% for most models, but when a previously developed model was compared with human observers, the agreement ranged from 50% to 62%.

Studies that compared NNs to other forms of AI such as k-nearest neighbor, Naïve Bayes, support vector machine, logistic regression, random forest, and decision trees, used for the same task, found NNs were more accurate.28,32 The NN used by Kök et al.32 displayed the most stable results compared with other systems.

Qualitative Assessment

The studies were assessed according to the QUADAS-2 tool (Table 3). The risk of bias and the applicability of both tests and the patients were examined. The flow and timing of the tests was also examined. All studies presented low concerns regarding the applicability of the patient data sets and the tests.

Table 3.

Methodologic Assessment of Studies According to the QUADAS Tool^a

Author, Year	Risk of Bias				Applicability Concern
	Patient Selection	Index Test (Neural Network Classification)	Reference Test (Human Expert's Classification)	Flow and Timing	Patient Selection	Index Test (Neural Network Classification)	Reference Test (Human Expert's Classification)
Kim et al., 202126	L	L	?	L	L	L	L
Seo et al., 202127	L	?	?	L	L	L	L
Amasya et al., 202028	L	L	L	L	L	L	L
Amasya et al., 202029	L	L	?	L	L	L	L
Kök et al., 202030	L	L	?	L	L	L	L
Kök et al., 202031	L	L	?	L	L	L	L
Kök et al., 201932	L	?	?	L	L	L	L
Makaremi et al., 201933	L	L	?	L	L	L	L

^a? indicates unclear; L, low.

The risk of bias presented some concerns in all studies. Studies with no separate test data set to evaluate the accuracy of the NN classification were considered as having some concerns for bias in the index test. In case of the reference standards, studies that had poor interobserver or intraobserver agreement for the data or did not evaluate this parameter were considered at risk of bias in reference standards. The risk of bias according to each domain is presented in Figure 2.

Figure 2.

Risk of bias across different domains.

DISCUSSION

This review examined the accuracy of NNs in classifying lateral cephalograms according to the stages of cervical vertebrae maturation. In the eight included articles, NN models were either adapted from previously trained models or devised specifically for the task of classifying CVM. They showed high accuracy during development, being more than 90% accurate at classifying the stages. The agreement between individual human observers and the NN's classification ranged from 50% to 62% in one study.29

The data used for development of the models was obtained ethically. All studies used cross-sectional data to identify the CVM stage. It was classified into training and testing sets by some studies,26,28,30,31,33 whereas others studies27,32 used cross-validation to determine the system's performance. Validation helps prevent overfitting of the model to a data set and helps make a generalized model.38 In fivefold cross-validation, used by most included studies, the data are split into five sets. One is used as a validation test set, whereas the remaining four are used for training. The process is repeated by changing the training and validation sets during each iteration of data. It helps adjust the weights assigned to the inputs for the perceptrons to map an accurate output. Validation minimizes the errors in a system by repeatedly adjusting the weights for inputs until there is a minimum error in the output with no significant change in the weights.

The validation data set is seen by the NN before and is an unreliable assessment of the NN's generalization. Generalization of an NN is its ability to correctly provide an output for data that were not a part of its creation or training. Thus, unseen test data provide a better evaluation of its accuracy. It is believed that to provide generalizability, the number of training examples provided to the model should be more than the parameters.11 This is also influenced by the size and type of training data.39 The studies used test and training data from the same study center. A better test of the model's practical performance would be on data provided from a different setting. The models can be shared with researchers or data obtained from remote institutions to better establish their performance. It is also possible to compare the performance of different models on the same test data sets.

The inputs used by the studies were linear measurements, ratios, or images. The models developed with linear inputs and ratios performed better than those that used only photographic or linear inputs. This could be due to the increased noise in photographs, since training on preprocessed images helped improve the output. The inputs must accurately represent the environment in which the NN operates. Makaremi et al.33 showed that NNs performed better when the data were distributed evenly across all CVM stages. Most included studies specified having an equal or near equal distribution in the training data. Cervical vertebrae maturation stage was classified using the description by Hassel and Farman and Baccetti et al.22,23 Both methods were considered reliable in predicting the pubertal growth spurt.40 Systems developed using both methods had similar accuracy in classifying lateral cephalograms. The method of Baccetti et al.23 may be more applicable across different data sets, as it can be used regardless of a radiation collar.

The cervical vertebrae maturation methods have inherent drawbacks. Studies show that the method is reproducible, but there may be a poor correlation between the cervical stages and peak mandibular growth on longitudinal assessment.41,42 Previous studies also raised concerns about the reproducibility of CVM stages.43,44 Because the visual analysis of the researchers is considered as the ground truth for the model, it is imperative that this judgment is not biased by the analysis of a single expert. Using data with high inter- and intraobserver agreement could provide better training for NNs.

The ultimate aim of assessment of CVM stages is to determine skeletal maturity and multiple characteristics that can help identify it. Developing comprehensive models can help improve the estimation of the skeletal age. They can use additional inputs such as the chronological age, facial photographs, and secondary sexual characteristics, coupled with CVM indicators. Such comprehensive AI models may perform just as well or better than human practitioners.

CONCLUSIONS

• Neural networks can successfully classify the different stages of cervical vertebrae maturation from lateral cephalometric radiographs.

• The accuracy of the diagnosis probably varies due to different inputs used for developing the models and a lack of standardization of data with inter- and intraobserver agreement.

• Further studies can develop models considering these drawbacks.

• Generalization of the developed models can be tested using publicly available or cross-center data sets.