Diagnostic accuracy of artificial intelligence in determining extraction protocol in orthodontic patients: A systematic review

Abstract

OBJECTIVE:

To assess the diagnostic accuracy of artificial intelligence-based models in the determination of tooth extraction in orthodontic treatment planning.

MATERIALS AND METHODS:

A comprehensive literature search was conducted in multiple databases (PubMed, LILACS, Web of Science, Scopus, EBSCO, and Google Scholar) up to June, 2024. Studies that met the inclusion criteria based on the PIRD (Participants, Index test, Reference test, Diagnostic) framework were selected. The risk of bias of included studies was assessed using the QUADAS-2 tool, and their methodological quality was evaluated as well using a standardized checklist.

RESULTS:

Out of 361 retrieved records, eleven studies were included in this review. Nine of these studies achieved a score of over 50% on the AI quality checklist, indicating acceptable methodological quality. However, a comprehensive assessment using the QUADAS-2 tool revealed that all studies had some level of risk of bias, particularly in patient selection, the conduct of AI-based predictions, and the reference standard used.

CONCLUSION:

Neural networks and classifier models demonstrated the high level of accuracy ranging from 82% to 94% in determining the optimal tooth extraction protocol. However, to ensure reliable predictions, artificial intelligence-based models should be rigorously trained, incorporating a comprehensive range of factors.

Introduction

Artificial intelligence (AI) is already an established technology in the fields of business and science, and has now effectively made its way into the healthcare system. Today, a collection of algorithms is outperforming various tasks otherwise performed by healthcare professionals worldwide.[1] AI incorporates machine learning (ML) that essentially analyses data such as images to infer the probability of disease outcomes.[2] It creates algorithms that analyses structures and patterns in data, allowing it to predict results for unseen information. Neural networks (NN), an increasingly popular model of ML, composed of artificial neurons, are proving to be superior to conventional algorithms in the detection of complex imagery and data.[3] Furthermore, recent research has explored alternative methodologies, such as single classifiers, ensemble classifiers and automated machine learning (AutoML) systems.[4,5,6] The detection of caries, diagnosis and treatment planning of various pathologies and prediction of different treatment outcomes are all now facilitated by this technology.[7]

Routine orthodontic cases require an astute management strategy to achieve the desired outcome.[8] The implementation of AI in diagnostic imaging such as cephalometric landmark detection and analysis,[9] fully automated determination of the cervical vertebrae maturation stages,[10] evaluation of facial attractiveness,[11] classification of craniofacial skeletal patterns,[12,13] surgery/non-surgery decision-making in class III cases are now reflected in orthodontic-decision making process.[14]

Determination of an extraction protocol that is, whether to extract teeth or not, is often a dilemma especially in borderline orthodontic cases.[15] The decision primarily hinges on factors such as the degree of malocclusion, soft tissue profile, treatment paradigm, and the clinician’s orthodontic training.[15,16,17,18] It requires a multifactorial analysis which includes clinical, study models and cephalometric variables.[19] A single orthodontic case can be approached with multiple treatment plan options through subjective assessment.[20] In an attempt to offer an objective approach, prediction models utilizing multiple algorithms are being developed to determine the need for orthodontic extraction for effective clinical decision-making.[4] In recent years, numerous studies have been reported in the literature evaluating the diagnostic accuracy of these models. Furthermore, the identification of variables having the greatest impact on this decision can be deduced. Therefore, this review was undertaken to provide a synthesis of the available evidence in determining the accuracy of various AI-based prediction models in the formulation of an extraction protocol.

Materials and Methods

Protocol development

We followed preferred reporting items for systematic reviews and meta-analysis (PRISMA) guidelines (www.prisma-statement.org) and the Cochrane Handbook for Systematic Reviews of Interventions (www.cochrane-handbook.org) for the proper conduction and reporting of the analysis. The protocol was registered as CRD42024567811 in the PROSPERO database (http://www.crd.york.ac.uk/PROSPERO).

Information sources and search strategy

A comprehensive electronic database search was conducted relevant to the topic under evaluation with individual detail in PubMed/MEDLINE, LILACS, Web of Science, Scopus, and EBSCO. Gray literature searches through Google Scholar were undertaken as well that were limited to the top 100 most relevant articles published within the past decade as seen in Table 1.

Table 1.

List of search engines and their results

Search engine	Result
PubMed	81
LILACS	36
Web of Science	47
Scopus	75
EBSCO	22
Google Scholar	100

The search strategies used for all search engines are mentioned in Supplementary Table 1.

Supplementary Table 1.

Search strategies

PubMed

(((((((((((ai artificial intelligence [MeSH Terms]) OR (machine learning [MeSH Terms])) OR (deep learning [MeSH Terms])) OR (supervised machine learning [MeSH Terms])) OR (neural network[MeSH Terms])) OR (unsupervised machine learning [MeSH Terms])) AND (orthodontist [MeSH Terms])) OR (orthodontics [MeSH Terms])) AND (tooth extraction [MeSH Terms]))) AND (diagnosis [MeSH Terms] AND (prediction)))

LILACS

(“Artificial intelligence [title, abstract, subject]” OR “machine learning [title, abstract, subject]” OR “deep learning [title, abstract, subject]” “supervised machine learning [title, abstract, subject]” “unsupervised machine learning [title, abstract, subject]” OR “neural network [title, abstract, subject]” AND “orthodontics [title, abstract, subject]” OR “orthodontist [title, abstract, subject]” AND “extraction [title, abstract, subject]” OR “diagnosis [title, abstract, subject]” OR “prediction [title, abstract, subject]”)

Web of Science

(TS=(“Artificial Intelligence” OR “Machine Learning” OR “Deep Learning” OR “Supervised Machine Learning” OR “Unsupervised Machine Learning” OR “Neural Networks”)) AND TS=(“Orthodontics” OR “Orthodontist” OR “Tooth extraction” OR “Diagnosis” OR “Prediction”)

Scopus

TITLE-ABS-KEY (“Artificial Intelligence”) OR (TITLE-ABS-KEY (“Machine Learning”) OR TITLE-ABS-KEY (“Deep Learning”) OR TITLE-ABS-KEY (“Supervised Machine Learning”) OR TITLE-ABS-KEY (“Unsupervised Machine Learning”) OR TITLE-ABS-KEY (“Neural Network”)) AND (TITLE-ABS-KEY (“orthodontics”) OR TITLE-ABS-KEY (“orthodontist”) OR TITLE-ABS-KEY (“extraction”) OR TITLE-ABS-KEY (“diagnosis”) OR TITLE- ABS-KEY (“prediction”))

EBSCO

TI (“artificial intelligence”) OR TI (“machine learning”) OR TI (“deep learning”) OR TI (“supervised machine learning”) OR TI (“unsupervised machine learning”) OR TI (“neural network”) AND TI (“extraction”) AND AB (“orthodontics”) OR AB (“orthodontist”) OR (“diagnosis”) OR (“prediction”)

Google Scholar

With all the words “artificial intelligence,” “orthodontic,” “extraction;” with at least one of the words “machine learning,” “deep learning,” “supervised machine learning,” “unsupervised machine learning,” “neural network,” “orthodontist,” “diagnosis,” “prediction” anywhere in the article

Eligibility criteria

The systematic review included all observational studies measuring diagnostic accuracy of artificial intelligence models in determination of extraction protocol in orthodontic patients prospectively or retrospectively. The selection criteria were based on the following PIRD framework:

• Participants (P): general orthodontic population

• Index test (I): artificial intelligence-based models for determining tooth extraction protocol in orthodontic treatment

• Reference test (R): orthodontic pretreatment records (intra- and extraoral photographs, dental models, and lateral cephalogram) evaluated by orthodontic experts

• Diagnostic (D): accuracy in determining tooth extraction protocol

Exclusion of studies

The exclusion criteria disregarded case reports, reviews, single intervention trials without the control group, studies with artificial intelligence not investigating tooth extraction decision-making. Participants having incomplete set of records, patients with skeletal asymmetry and maxillofacial deformities requiring orthognathic intervention, congenital malformations, anomalies such as hypodontia, and systemic conditions were also excluded from the studies.

Study selection, data collection, and analyses

Two reviewers (SM and VS) performed data collection independently to record general information such as the author, year of publication, sample size, and study characteristics. They also recorded details about the intervention, comparison, study design, and the results of each individual study. Study selection procedure was comprised of title-reading, abstract-reading, followed by full-text-reading stages. After exclusion of studies that did not fit the criteria, the complete report of publications considered eligible for inclusion by either author was obtained and assessed independently. Disagreements were resolved by discussion and consultation between the two authors. A separate observer (JK) further analyzed the two datasets, and final data were prepared.

The primary objective of this review was to thoroughly analyze the existing literature and ascertain the accuracy or the success rate of artificial intelligence-based models in predicting tooth extraction decision in orthodontic population. The secondary objective of this systematic review was to determine the most reliable/predictive indicators amongst the multiple diagnostic parameters included in the studies.

Risk of bias and methodological quality assessment of selected studies

The risk of bias for the selected studies was evaluated using the QUADAS-2 tool[21] [Annexure I]. Three independent reviewers assessed patient selection, index test, and reference test domains. The flow and timing domain was deemed inapplicable for AI studies. Bias was categorized as low, high, or unclear. A low risk was assigned if all signaling questions were answered affirmatively within a domain. Conversely, a negative response to any signaling question indicated potential bias. Disagreements were resolved through consensus with a fourth reviewer.

A comprehensive assessment of the methodological quality of each included study was conducted using a specialized checklist for AI research.[22] This checklist evaluated key aspects, including data quality (sampling, processing, and protection), sample size adequacy, the presence of a reliable reference test, the quality of the test dataset (model and training), clustering, and the sufficiency of computational resources [Annexure II]. Each item was assigned a binary score (1 for presence, 0 for absence), and the total score was calculated to determine the overall methodological quality of each study.

Results

Study selection and criteria

A total of 361 records were found from five databases (PubMed, Scopus, LILACS, Web of Science and EBSCO) and Google Scholar. After removing the duplicates, 212 articles remained for screening based on title and abstract. Subsequently, 35 articles were shortlisted for full text reading. Eleven studies met the final inclusion criteria.[4,5,6,19,23,24,25,26,27,28,29] Steps taken during the reviewing process to identification and selection of studies are presented in Figure 1. The excluded studies are mentioned in Supplementary Table 2.

Figure 1.

PRISMA 2020 Flow Diagram. Source: Haddaway, N. R., Page, M. J., Pritchard, C. C., and McGuinness, L. A. (2022). PRISMA2020: An R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimized digital transparency and Open Synthesis Campbell Systematic Reviews, 18, e1230. https://doi.org/10.1002/cl2.1230

Supplementary Table 2.

List of excluded studies based on full text analysis with justification

	Author, (year)	Study title	Reason for exclusion
1.	Ahmed et al (2021)	Artificial Intelligence Techniques: Analysis, Application, and Outcome in Dentistry-A Systematic Review.	Review
2.	Bichu et al (2021)	Applications of artificial intelligence and machine learning in orthodontics: a scoping review	Review
3.	Turkestani et al (2021)	Clinical Decision Support Systems in Orthodontics: A Narrative Review of Data Science Approaches	Review
4.	Mohammad-Rahimi et al (2021)	Machine learning and orthodontics, current trends, and the future opportunities: A scoping review	Review
5.	Evangelista et al (2022)	Accuracy of artificial intelligence for tooth extraction decision-making in orthodontics: a systematic review and meta-analysis.	Review
6.	Cao, Cong (2022)	Application of artificial intelligence technology in orthodontics	Chinese language
7.	Thirumoorthy et al (2023)	Diagnostic accuracy of AI in orthodontic extraction decisions: “Are we ready to let Mr. Data run our Enterprise?” A commentary on a systematic review.	Review
8.	Fawaz et al (2023)	What is the current state of artificial intelligence applications in dentistry and orthodontics?	Review
9.	Leavitt et al (2023)	Can we predict orthodontic extraction patterns by using machine learning?	Incomplete factors assessment
10.	Kunz et al (2023)	Applications of Artificial Intelligence in Orthodontics—An Overview and Perspective Based on the Current State of the Art	Review
11.	Motamaen et al (2024)	Insights into Predicting Tooth Extraction from Panoramic Dental Images: Artificial Intelligence vs. Dentists.	Multiple factors assessment
12.	Kazimierczak et al (2024)	AI in Orthodontics: Revolutionizing Diagnostics and Treatment Planning—A Comprehensive Review	Review

Study characteristics and results of individual studies

Data extracted from eleven studies, published between January, 2010 and June, 2024, are presented in Tables 2 and 3. The artificial intelligence-based models were used splitting the data into training, validation, and test sets in two studies[28,29] and into training and tests sets in seven studies.[6,19,23,24,25,26,27] Two studies showed no information on the data sets.[4,5] Validation and training sets accounted for 60%–90% of the data, while testing sets comprised 10%–38.5%. To mitigate the impact of a separate validation set, six studies implemented cross-validation strategies, ensuring that all data points were utilized as test data to reduce the risk of overfitting.[4,5,24,25,26,27] The reference test of all studies consisted of a professional treatment planning by experienced orthodontists based on orthodontic diagnostic records. Three studies[19,23,27] showed no information regarding the number of orthodontists, whereas eight studies[4,5,6,24,25,26,28,29] specified a particular number ranging from 1 to 19 orthodontists. Out of the eight studies, six studies[4,5,6,25,28,29] have also mentioned their training experience which ranges from 9–18 years. Demographic, clinical, model, and cephalometric variables used to obtain the artificial intelligence models varied from 1 to 117 items in all studies. The type of artificial intelligence and algorithm in all the included studies fall under three broad categories: (1) Artificial Neural Networks (Multilayer perceptron/Convolutional Neural Network), (2) Single Classifiers (Logistic Regression/Support Vector Machine Decision/Tree Classifier/K-Nearest Neighbors/Naïve Bayes Classifier), (3) Ensemble Classifiers (Random Forest/Gradient Boosted Trees/Voting Classifier/Stacking Classifier).

Table 2.

Basic characteristics of selected studies

Author, year	Sample sets	Sample characteristics	Clinician’s experience	Diagnostic variables	AI algorithm	Conclusion
Xie et al.,[19] 2010	200 180: Training 20: Test	120: extraction 80: non-extraction	N/A	Clinical (2), Model (5), Cephalometric (18)	Artificial neural network (Back propagation)	The constructed artificial neural network in this study can correctly judge with 80% accuracy
Jung et al.,[28] 2016	156 64: Training 32: Validation 60: Test	94: extraction 62: non-extraction	One orthodontist >10 years of experience	Clinical (6) and cephalometric (12)	Machine learning (Back propagation)	The success rates of the classifiers were 93% for the diagnosis of extraction vs non-extraction
Li et al.,[29] 2019	302 182: Training 60: Validation 60: Test	222: extraction 80: non-extraction	Two orthodontists >12 years of experience	Clinical (10) and cephalometric (14)	Artificial neural network (Multilayer perceptron)	Neural networks show an accuracy of 94% and can provide good guidance for orthodontic treatment planning for less-experienced orthodontists.
Suhail et al.,[5] 2020	287 patients	N/A	Five orthodontists 9 years of experience	19 variables	Single classifier (Logistic Regression), Ensemble Classifier (Random Forest)	A random forest ensemble classifier was confirmed to show a high performance, within the range of the inter-expert agreement.
Etemad et al.,[24] 2021	838 754: Training 84: Test	208: extraction 630: non-extraction	Nineteen orthodontists	1st set: Clinical (15) and cephalometric (102) 2nd set: Clinical (9) and cephalometric (13)	Ensemble learning (Random Forest), Artificial neural network (multilayer perceptron)	Further improvement of the performance of machine learning models is needed for clinical use in the orthodontic field.
Real et al.,[4] 2022	214	N/A	Two orthodontists 18 years of experience	Clinical (1), Model variables (7), Cephalometric (33)	Auto-WEKA (multilayer perceptron)	Three different models achieved accuracies of up to 93.9% for predicting the need for tooth extractions. Prediction models for the need for dental extractions achieve their best performance when model and cephalometric data are combined.
Prasad et al.,[6] 2022	700 490: Training 210: Test	N/A	10–15 orthodontists 10–15 years of experience	33 variables	Random Forest Classifier, XGB Classifier, Logistic Regression, Decision Tree Classifier, K-Neighbors Classifier, Linear Support Vector Machine, Naïve Bayes Classifier	Overall, the ML-based AI model showed 84% accuracy in its treatment plan prediction compared with the treatment plan for the same cases decided by expert opinion of orthodontists.
Ryu et al.,[26] 2023	1636 (1500: maxillary 1636: mandibular) Training: 1300 maxillary and 1436 mandibular Test: 200 maxillary and 200 mandibular	Extraction 324: maxillary 329: mandibular Non-extraction 1216: maxillary 1307: mandibular	Two orthodontists	Photograph variables	Convolutional neural network (ResNet50, ResNet101, VGG16, and VGG19	The performance of VggNet was better than that of ResNet, and the results were more accurate in the maxilla than in the mandible in terms of consistency.
Mason et al.,[27] 2023	393 patients 275: Training 118: Test'	193: extraction 200: non-extraction	N/A	2 demographic, 4 clinical, 50 cephalometric variables	Logistic regression, Random Forest, support vector machine, Neural Network	The LR, SVM, and NN models were all found to be highly accurate when predicting the binary extraction decision.
Trehan et al.,[23] 2023	700 630: Training set 70: Test set	322: extraction 378: non-extraction	N/A	Model variable (1)	Convolutional neural network: ResNet-50	ANN model presented a sensitivity of 64.04% relative to the conventional method. The prediction accuracy was 65.12% for the extraction cases and 62.96% for the non-extraction cases relative to the conventional method.
Köktürk et al.,[25] 2024	1000 800: Training 200: Test	500: extraction 500: non-extraction	Four orthodontists 10 years of experience	Clinical (3), model (4), cephalometric (29)	Logistic Regression, Support Vector Machines, Random Forests, Gradient Boosted Trees, Multi-layer Perceptron (ANN), Voting Classifier, Stacking Classifier	The highest performing model was the Stacking Classifier with 84.1% accuracy and 0.912 AUC value.

Table 3.

Results of the diagnostic tests and most contributory features^a

Author, year	Diagnostic tests												Most contributory features
	ACC			SEN			SPE		PPV		NPV
Xie et al,[19] 2010	0.80			-			-		-		-		-anterior teeth uncovered by incompetent lips -IMPA -overjet
Jung et al,[28] 2016	0.93			-			-		-		-		-
Li et al,[29] 2019	0.94			0.94			0.93		-		-		-upper arch crowding -lower arch crowding U1-NAo
Suhail et al,[5] 2020	-			-			-		-		-		-crowding -tooth inclination
Etemad et al,[24] 2021													-Angle’s classification -gender -curve of Spee
Random Forest (22)	0.75			0.71			0.76		-		-
Multilayer perceptron (22)	0.79			0.72			0.81		-		-
Random Forest (117)	0.76			0.64			0.79		-		-
Multilayer perceptron (117)	0.75			0.68			0.77		-		-
Real et al,[4] 2022													-maxillary arch discrepancy -mandibular arch discrepancy -molar class-modified
S1 5 min Bagging	0.803			0.804			-		0.802		-
S1 15 Random committee	0.864			0.864			-		0.864		-
S1 30 Multilayer perceptron	0.803			0.804			-		0.802		-
S1 60 Multilayer perceptron	0.803			0.804			-		0.802		-
S1 overnight Multilayer perceptron	0.939			0.939			-		0.94		-
S2 5 min LMT	0.873			0.874			-		0.876		-
S2 15 min REP tree	0.817			0.818			-		0.816		-
S2 30 min REP tree	0.817			0.818			-		0.816		-
S2 60 min J48	0.799			0.799			-		0.798		-
S2 overnight Random Tree	0.841			0.841			-		0.845		-
S3 5 min SMO	0.719			0.72			-		0.715		-
S3 15 min Multilayer Perceptron	0.705			0.706			-		0.699		-
S3 30 min SMO	0.70			0.701			-		0.693		-
S3 60 min AdaBoost	0.705			0.706			-		0.699		-
S3 overnight Bagging	0.705			0.706			-		0.701		-
Ryu et al,[26] 2023	U		L	U		L	U	L	U	L	U	L	-
ResNet50	0.909		0.895	0.778		0.778	0.939	0.929	-	-	-	-
ResNet101	0.915		0.890	0.751		0.716	0.952	0.941	-	-	-	-
VGG16	0.910		0.898	0.811		0.796	0.933	0.928	-	-	-	-
VGG19	0.922		0.898	0.854		0.818	0.937	0.921	-	-	-	-
Mason et al,[27] 2023													-maxillary crowding/spacing -L1-NB (mm) -U1-NA (mm) -PFH: AFH -SN-MPo
Logistic Regression	0.82			-			-		-		-
Random Forest	0.76			-			-		-		-
Support Vector Machine	0.83			-			-		-		-
Neural Network	0.81			-			-		-		-
Prasad et al,[6] 2022													-Beta angle -ANB -Witts’s appraisal - Profile -overjet and overbite -maxillary and mandibular dimensions
XGB Classifier	0.914			-			-		0.91		-
Random Forest Classifier	0.92			-			-		0.914		-
Linear SVM	0.834			-			-		0.828		-
Decision Tree Classifier	0.88			-			-		0.875		-
Logistic Regression	0.856			-			-		0.854		-
K-Neighbor Classifier	0.739			-			-		0.738		-
Naive Bayes Classifier	0.719			-			-		0.705		-
Trehan et al,[23] 2023	0.65			0.64			-		-		-		-
Köktürk et al,[25] 2024													-maxillary and mandibular arch length discrepancy -Witts’s appraisal -ANS-Me
Multilayer perceptron	0.81			-			-		0.81		-
Random Forest	0.81			-			-		0.80		-
Logistic Regression	0.82			-			-		0.81		-
Support Vector Machine	0.82			-			-		0.81		-
Gradient boosted trees	0.83			-			-		0.83		-
Voting Classifier	0.83			-			-		0.82		-
Stacking Classifier	0.84			-			-		0.83		-

^aACC=Accuracy, SEN=Sensitivity, SPE=Specificity, PPV=Positive Predictive Value, NPV=Negative Predictive Value

Risk of bias or quality assessment of selected studies

All the studies were deemed methodologically acceptable according to the results of QUADAS-2 tool as shown in Figure 2. However, no study satisfied the all the risk of bias according to the ideal criteria. Most studies employed convenience sampling and exhibited unbalanced sample sizes between extraction and non-extraction groups. This methodological limitation was deemed unclear and questions the applicability of the findings to diverse patient populations. The substantial variability in the number of parameters across studies introduced bias, as AI models were not adequately trained on sufficient data to produce reliable outcomes. Five studies reported a bias with respect to the reference standard because of lack of information regarding the orthodontists involved.[19,23,24,26,27]

Figure 2.

(a) Risk-of bias graph. (b) Risk-of bias summary

Based on the methodological quality analysis presented in Table 4, nine studies[4,5,6,24,25,26,27,28,29] demonstrated a high level of quality, scoring over 50% of the total points. Only one study met all eight of the specified criteria.[25] Although five studies[4,6,24,25,26] considered data protection requirements, only two studies[23,25] selected an appropriate sample size as per the ideal criteria. Eight studies[4,5,24,25,26,27,28,29] performed adequate data splitting, and all studies effectively utilized computational tools for various purposes.

Table 4.

Methodological assessment of included studies

Study	Sampling	Processing	Protection	Sample size	Reference test	Clustering	Test database	Computational resource	Total
Xie et al.,[19] 2010	1	1	0	0	0	1	0	1	4
Jung et al.,[28] 2016	1	1	0	0	1	1	1	1	6
Li et al.,[29] 2019	1	1	0	0	1	1	1	1	6
Suhail et al.,[5] 2020	1	1	0	0	1	1	1	1	6
Etemad et al.,[24] 2021	1	1	1	0	0	1	1	1	6
Real et al.,[4] 2022	1	1	1	0	1	1	1	1	7
Prasad et al.,[6] 2022	1	1	1	0	1	1	0	1	6
Ryu et al.,[26] 2023	1	1	1	0	0	0	1	1	5
Mason et al.,[27] 2023	1	1	0	1	0	1	1	1	6
Trehan et al.,[23] 2023	1	1	0	1	0	0	0	1	4
Köktürk et al.,[25] 2024	1	1	1	1	1	1	1	1	8

Discussion

Orthodontic extractions represent a significant and irreversible decision that can profoundly impact treatment outcomes.[30] Variations in individual clinical experience may lead to diverse treatment approaches.[31,32] Artificial intelligence attempts to make the decision-making process of tooth extraction more streamlined and objective, eliminating subjective errors. The AI based models included in this review employed a diverse range of machine learning techniques, including artificial neural networks, convolutional neural networks, classification models, and regression algorithms. Notably, seven of the reviewed studies incorporated multiple algorithms and compared their performance in terms of predictive accuracy. Etemad et al.[24] evaluated the performance of Random Forest and multilayer perceptron models using two distinct variable sets, derived from prior studies. Although their study yielded lower accuracy compared to the work of Jung et al.[28] and Li et al.,[29] this discrepancy may be attributed to factors such as limited sample size, non-consecutive patient enrolment, restricted practitioner diversity, and a focus on an exclusively Asian population. A subsequent study utilized automated ML algorithm to generate three prediction settings testing the model and cephalometric variables individually as well as in a combined setting.[4] For each setting, five models were trained with time limits of 5, 15, 30, 60 min, and overnight. The overnight model using combined data achieved an accuracy of 93.9%. Notably, the model-based setting outperformed the cephalometric-based setting by 15.4%, suggesting the greater predictive power of model variables in extraction decision-making. This finding aligns with existing literature, which underscores the significance of maxillary and mandibular arch dimensions and discrepancies as key determinants in binary extraction decisions.[5,6,25,27,29]

To simplify the AI process and enhance user-friendliness for orthodontists, two studies exclusively utilized photographs for training their AI models. In 2023, Ryu et al.[26] proposed a method for extraction decision-making and crowding categorization based on intraoral occlusal photographs. Conversely, a similar study employed a single parameter i.e., extraoral right profile photographs of the patients to train a convolutional neural network model.[23] This model achieved lower accuracy rates of 65.12% for extraction cases and 62.96% for non-extraction cases, when compared to conventional methods. Although these methods offer accessibility, they rely solely on visual data and do not integrate additional model or cephalometric parameters, potentially limiting their predictive accuracy.

A multi-layered machine learning approach, incorporating seven algorithms across four model layers was employed by Prasad et al.[6] Following the diagnosis of jaw base classification, the second layer predicted the need for extraction or non-extraction of first and/or second bicuspids. The study demonstrated promising accuracy for the Random Forest, Decision Tree, and XGB Classifier models, corroborating the findings of Kok et al.,[33] who applied similar algorithms for determining Cervical Vertebrae Maturation Index (CVMI) stages based on cephalometric analysis.

Two studies aimed to identify the most effective machine learning method to support clinicians in orthodontic treatment planning.[25,27] Ensemble classifiers including Support Vector Machine consistently demonstrated strong performance, with Mason et al.[27] reporting an AUC of 92.5%. Among the ensemble classifiers, the Stacking Classifier model exhibited a high performance with an accuracy of 84.1 % and 91.2% AUC value.[25] While previous studies[19,28,29,34] have frequently employed neural networks for extraction decision-making, their black-box nature limits interpretability, making it difficult to determine that variables are most reliable. In contrast, the present analyses, utilizing multiple machine learning models, enable a deeper exploration of the factors that significantly contribute to the prediction of extraction. The methodological assessment performed in this review agrees with these findings considering all sampling, processing, and clustering techniques to minimize errors, and deemed only one study fit in the predefined criteria for quality.[25]

In essence, AI algorithms exhibit enhanced performance as the volume of training data expands.[35,36] Consequently, variations in sample size can influence the quality of model performance. Another challenge in machine learning is the problem of overfitting, where models become excessively complex and memorize training data, thereby compromising their ability to generalize to unseen data.[25] According to the present analyses, three studies reported a high accuracy levels of up to 94%.[4,28,29] However, the absence of cross-validation techniques in two of the above-reported studies raises concerns regarding the potential risk of overfitting.[28,29] Key parameters influencing orthodontic extraction decision are of prime importance in orthodontics. While crowding has traditionally been a primary factor, recent studies emphasize the growing importance of soft tissue considerations.[16,37,38,39,40] This is supported by two of the included studies, which identified incompetent lips and patient profile as significant variables.[6,19] Although many studies have cited arch length discrepancy as their prime variable, skeletal parameters (Beta angle, Witts appraisal, ANS-Me) and dental parameters (U1-NA, L1-NB, IMPA, molar relation, overjet) have also been reportedly significant.[4,5,6,25,27,29]

Limitations

This systematic review identified only two studies that fulfilled the ideal criteria for sample size and characteristics. Methodological differences, including variations in AI techniques, sample sizes, training and testing sets, lack of external validation and the clinical expertise of orthodontists involved in treatment planning, led to the range of diagnostic accuracy values. Due to insufficient data, a meta-analysis could not be conducted. Future research should prioritize high-quality studies that compare multiple algorithms to determine the most accurate tool for extraction decision-making. Furthermore, predicting specific extraction patterns could be a valuable additional outcome.

Conclusion

Neural Networks have demonstrated high accuracy of up to 94% in determination of tooth extraction protocol. However, their nature limits the interpretability of their decision-making processes.

Ensemble Classifiers, specifically Stacking Classifier, incorporating multiple machine learning models can significantly improve diagnostic accuracy as well as provide insight into the most contributory factors.

To ensure the effective application of artificial intelligence in orthodontics, future studies should consider factors such as balanced sample size, appropriate data splitting, rigorous cross-validation, and the inclusion of relevant clinical, model, and cephalometric variables.

Author’s contributions

S.M.: Investigation; Methodology, Writing - original draft

V.S.: Conceptualization, Writing - review and editing

J.K: Writing - original draft

M.K.: Data curation, Formal analysis, Validation

T.P.C.: Project administration, Supervision

A.J: Writing - review and editing, Resources

Conflicts of interest

There are no conflicts of interest.

Annexure I: QUADAS-2 checklist

QUADAS-2

Phase 1.

State the review question:

Patients (setting, intended use of index test, presentation, prior testing):

Index test(s):

Reference standard and target condition:

Phase 2: Draw a flow diagram for the primary study

Phase 3: Risk of bias and applicability judgments

QUADAS-2 is structured so that 4 key domains are each rated in terms of the risk of bias and the concern regarding applicability to the research question (as defined above). Each key domain has a set of signaling questions to help reach the judgments regarding bias and applicability.

DOMAIN 1.

PATIENT SELECTION

A. Risk of Bias
Describe methods of patient selection:
Was a consecutive or random sample of patients enrolled?	Yes/No/Unclear
Was a case-control design avoided?	Yes/No/Unclear
Did the study avoid inappropriate exclusions?	Yes/No/Unclear
Could the selection of patients have introduced bias?	RISK: LOW/HIGH/UNCLEAR
B. Concerns regarding applicability
Describe included patients (prior testing, presentation, intended use of index test and setting):
Is there concern that the included patients do not match the review question?	CONCERN: LOW/HIGH/UNCLEAR

DOMAIN 2.

INDEX TEST(S)

A. Risk of Bias
Describe the index test and how it was conducted and interpreted:
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes/No/Unclear
If a threshold was used, was it pre-specified?	Yes/No/Unclear
Could the conduct or interpretation of the index test have introduced bias?	RISK: LOW/HIGH/UNCLEAR
B. Concerns regarding applicability
Is there concern that the index test, its conduct, or interpretation differ from the review question?	CONCERN: LOW/HIGH/UNCLEAR

DOMAIN 3.

REFERENCE STANDARD

A. Risk of Bias
Describe the reference standard and how it was conducted and interpreted:
Is the reference standard likely to correctly classify the target condition?	Yes/No/Unclear
Were the reference standard results interpreted without knowledge of the results of the index test?	Yes/No/Unclear
Could the reference standard, its conduct, or its interpretation have introduced bias?	RISK: LOW/HIGH/UNCLEAR
B. Concerns regarding applicability
Is there concern that the target condition as defined by the reference standard does not match the review question?	CONCERN: LOW/HIGH/UNCLEAR

DOMAIN 4.

FLOW AND TIMING

A. Risk of Bias
Describe any patients who did not receive the index test (s) and/or reference standard or who were excluded from the 2×2 table (refer to flow diagram): Describe the time interval and any interventions between index test(s) and reference standard:
Was there an appropriate interval between index test(s) and reference standard?	Yes/No/Unclear
Did all patients receive a reference standard?	Yes/No/Unclear
Did patients receive the same reference standard?	Yes/No/Unclear
Were all patients included in the analysis?	Yes/No/Unclear
Could the patient flow have introduced bias?	RISK: LOW/HIGH/UNCLEAR

Annexure II: A modified AI research checklist, adapted from Schwendicke et al.[23], was used to evaluate the methodological quality of included studies

Item	Criteria
Data	Consecutive or random sampling, appropriate records, information protection
Sample Size	Over 400 patients, balanced extractions and non-extractions, regardless of sex, age, or malocclusion
Reference Test	Orthodontic treatment planning by experienced professional
Clustering	Clinical, model and cephalometric parameters of each patient
Test Dataset (data splitting)	Balanced distribution between validation, training, and testing sets or presence of cross validation
Computational Resources	High-quality computational image files without loss of resolution or quality

Funding Statement

Nil.