Evaluating the accuracy of sex estimation from human tooth volume: leveraging automated AI segmentation and comparative analysis of machine learning algorithms

Abstract

Background

Accurately estimating biological sex is a fundamental step in forensic odontology, anthropology and archaeology. Tooth volume has been recognized as a valuable feature for sex estimation. However, dental morphology exhibits population-specific variations, which remains underexplored in the northern Han Chinese population. Furthermore, manual segmentation of cone-beam computed tomography (CBCT) images is prone to subjectivity and error, and conventional modeling approaches may offer limited predictive accuracy. This study aimed to address these limitations by integrating automated artificial intelligence (AI) segmentation with multiple machine learning algorithms to enhance objectivity and predictive performance in sex prediction.

Methods

A cohort of 398 CBCT images was analyzed using a fully automated deep learning system for tooth segmentation, and a final cohort of 357 samples (185 females, 172 males) was retained. Volumetric data for teeth were automatically calculated, with missing values imputed via multiple imputation by chained equations (MICE). Four machine learning algorithms were applied to build binary classification models for sex estimation. Hyperparameter optimization was achieved through nested cross-validation, and model performance was evaluated on a test set using a comprehensive set of metrics.

Results

All 16 measured teeth showed significantly differences in volume between sexes, with males consistently showing larger mean volumes across all tooth types compared to females. Nested cross-validation showed that all models exhibited promising performance for sex estimation with mean ACC above 0.77 and mean AUC values at or above 0.85. On the test set, the models achieved accuracies ranging from 0.803 to 0.859 and AUCs from 0.893 to 0.903. SHAP interpretability analysis highlighted canines as having the greatest impact on predictions.

Conclusions

This study is the first to demonstrate the significant sexual dimorphism of tooth volumes in a north Chinese population. We utilized an AI-based automated segmentation accompanied with machine learning modeling pipeline to accurately estimate sex using tooth volume features, offering a valuable approach for forensic and anthropological applications.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13005-026-00633-x.

Introduction

Individual identification, including sex, age, ancestry and stature, plays an important role in forensic anthropology, forensic odontology and archaeology [1, 2]. Sex estimation represents a fundamental parameter of human remains or unknown individuals [3]. The accurate estimation of biological sex facilitates enhancing the efficiency and effectiveness of the identification process, and providing critical direction for subsequent analyses such as age or stature estimation, which are often sex-dependent [4–6]. There are various methods for sex estimation, such as DNA and skeletal morphology [7–10]. However, as a biochemical detection method with high accuracy, DNA test has disadvantages such as high cost, time-consuming process, equipment requirements and invasive procedure that involves the destruction of the sample [11]. Although morphological methods based on the length and angle of specific parts such as pelvic bones and head of the femur are non-invasive and comparatively easier. Morphology-based strategy is highly subjective and subjected to the skeletal integrity [12–14]. In real-world scenarios, human remains may often absent, fragmentary, or poorly preserved [15]. In such cases, teeth, as the hardest structure in the body, being highly resistant to extreme temperature, chemical and mechanical degradation during the postmortem phase, and protected by oralfacial structures, often serves as the most reliable or even only source of biological information [16, 17]. Enamel is one of the most highly mineralized tissues in the mammals. Amelogenin genes, localized on the sex chromosomes, are expressed as protein in the tooth organ and play a major role in the biosynthesis of enamel [18]. Specifically, the Y chromosome promotes tooth crown growth differently from the X chromosome, contributing to sexual dimorphism in tooth size [19]. Additionally, the development of teeth is influenced by sex hormones, indicating that they might be valuable for sex estimation [20].

Many studies focus on tooth size as an indicator for sex estimation. In these studies, researchers assess the teeth morphometric features with linear measurements such as mesiodistal [21], buccolingual width [22] and diagonal measurements of molars [23] and the distance between bilateral maxillary or mandibular canines [21]. However, they did not obtain consistent results with the accuracy from 56.5% to 85.8%, and it is time-consuming from impressions-making to measurement by vernier caliper. The intraoral scanner (IOS) emerging as a powerful tool for sex estimation in recent years [24]. A study integrated IOS and machine learning to estimate sex from measurements of the upper six anterior teeth and maxillary arch dimensions, showed that the Support Vector Classifier achieved the highest mean prediction accuracy of 94.5% [25]. With the development of Cone Beam Computer Tomography (CBCT) and digital dentistry, volumetric measurements in maxillofacial region structures such as maxillary sinus, mandible and teeth become a reality [26, 27]. It contains more information than linear measurement results. Wu et al. [27] reported a sex estimation formula by the volume of the maxillary sinus from CBCT images, and achieved accuracy values of 0.786 (left maxillary sinus) and 0.742 (right maxillary sinus), respectively. Bahan et al. [28] evaluated the accuracy of morphometric measurements of the mandible and reported that Gaussian Naive Bayes (GNB) model had the high testing accuracy (0.90). A CBCT image-based study on volumetric assessment of dental crowns by manual segmentation using 3D slicer software achieved an accuracy of 0.837 for sex estimation [29]. However, traditional 3D reconstruction approaches rely on the manual traced the boundary of specific structure, which are prone to inter- and intra-observer variability, and time-consuming.

Artificial intelligence (AI) has shown extraordinary image recognition performance in various medical settings such as pathological section, ultrasound and computed tomography [30]. In maxillofacial region, precisely identifying the complex anatomy such as inferior alveolar nerve and maxillary sinus provides a reference for surgery. Automated AI segmentation overcomes the limitations of traditional methods by providing an objective, reproducible, and highly efficient means of extracting 3D tooth geometry. ToothNet was the first network for tooth CBCT segmentation task proposed by Cui et al. [31]. Their method achieves high segmentation accuracy, however, it is difficult to apply simple image preprocessing methods to all images obtained from different CBCT devices, and they cannot perform all segmentation tasks completely automatically in an end-to-end manner. In 2024, Liu et al. [32] developed a deep learning-based fully automatic segmentation system which is the state-of-the-art, achieving an end-to-end segmentation and improving the accuracy effectively. The advantages of automatic AI segmentation are not only reducing the potential for human error but also saving time and resources, which is valuable in forensic casework or large-scale research studies.

Machine learning (ML) is a computational approach that learn patterns from data to support prediction and classification tasks. In machine learning field, one of the most frequently performed tasks is classification, such as male and female. The machine learning algorithm learns to predict based on specific characteristics learned from the data. These models often detect subtle sexual dimorphism features that univariate analyses or linear models may miss, improving predictive accuracy. Hence, this study aims to assess the effectiveness of tooth volume in sex estimation among the northern Han Chinese population. In this study, an automatic AI segmentation pipeline established by Liu et al. was utilized to achieve an end-to-end tooth segmentation of CBCT images. Based on the robust, objective, and efficient generation of 3D tooth volume data provided by AI-driven automation, multiple machine learning algorithms were applied and compared for sex estimation by yielding probabilistic biological sex inference. The null hypothesis for this study is that there is no significant difference in tooth volumes between sexes, and machine learning classifiers cannot estimate sex with high accuracy based on tooth volumes.

Materials and methods

Sample collection

This study retrospectively analyzed a total of 398 CBCT images obtained from the archives of Hospital of Stomatology Hebei Medical University, between 2023 and 2024. The sample comprised of 205 female and 193 male individuals of northern Han Chinese ancestry. To control for age-related changes in tooth volume, which can be influenced by factors such as dental development and attrition, the inclusion criteria were restricted to individuals aged between 18 and 30 years, and the exclusion criteria as follow: a history of surgery, severe facial trauma, facial malformations, syndromes, dental implants, bridges, crowns, and other tooth restorations that could alter tooth volume. In addition, images with poor quality, significant motion artifacts, and metallic or orthodontic artifacts were also excluded. This study was conducted in accordance with the principles established in the Declaration of Helsinki and received approval from the Ethics Committee of Hebei Medical University (No. 2025-088). All imaging data were used in accordance with prior written informed consent obtained at the time of clinical examination.

Data acquisition of CBCT scans

All CBCT images were acquired with a CBCT device (New Tom, VGi EVO, Italy) with the following parameters: 110 kVp, 5 mA, 16 × 16 cm field of view and 0.3 mm voxel size. Images were exported in DICOM format for subsequent measurements.

Automatic AI tooth segmentation

The tooth segmentation was performed using a fully automated deep learning system, which integrates adaptive CBCT image preprocessing and a hierarchical segmentation pipeline [32]. The segmentation pipeline consisted of three main stages, including image preprocessing, skeletal segmentation, and tooth instance segmentation, implemented on a PyTorch framework. For the specific task of individual tooth segmentation, the system employs the Swin-UNETR network architecture. The segmentation results obtained from this process were then used for measuring the volume of each identified tooth.

Tooth volume measurement and missing data imputation

Volumetric automatically measurements of individual teeth were computed using the same software platform. A set of 16 teeth was evaluated for each CBCT scan in this study, including the upper and lower left and right central incisor (FDI 11, 21, 31, 41), canines (FDI 13, 23, 33, 43), first premolars (FDI 14, 24, 34, 44), and first molars (FDI 16, 26, 36, 46). These 16 teeth were selected as representatives from four tooth types across both arches and sides, due to their relatively smaller morphological variability compared to lateral incisor, second premolars and third molars [33, 34]. An experienced dental radiologist visually inspected the 3D segmentation masks superimposed on the original CBCT slices in all three orthogonal planes. This review was strictly qualitative (pass/fail) to ensure that the AI correctly identified the anatomical boundaries and that no adjacent structures (e.g., alveolar bone) were erroneously included in the tooth volume. Any cases involving mislabeling or segmentation errors were excluded from further analysis, resulting in the removal of 41 scans from the initial dataset. Missing values in dental volume measurements were addressed using multiple imputation by chained equations (MICE) through the ‘mice’ package (version 4.4.2) in R software.

Statistical analysis

All statistical analyses were performed using R software (version 4.4.2). To compare difference between female and male samples, we implemented a comprehensive statistical analysis approach that automatically selects the appropriate test based on data characteristics (Figure S1). First, the normality of the data distribution of volume of 16 teeth in both groups was assessed using the Shapiro-Wilk test. For normally distributed data (Shapiro-Wilk p > 0.05), parametric testing was employed. Variance homogeneity was evaluated using Levene’s test. When variances were equal (p > 0.05), Student’s t-test was applied, otherwise, Welch’s t-test was used. For non-normally distributed data (Shapiro-Wilk p ≤ 0.05), the non-parametric Mann Whitney U test was employed. The effect size was calculated using Cohen’s d method for parametric comparisons, and Cliff’s delta method for non-parametric comparisons. P-values were adjusted for multiple comparisons using the Bonferroni method. To assess univariated discriminative performance for sex estimation, receiver operating characteristic (ROC) analysis was performed for each of the 16 selected teeth.

Machine learning analysis and model evaluation

To investigate the potential of sex estimation by tooth volume, a machine learning-based binary classification approach was implemented. The dataset was randomly split into training (80%, n = 286) and test (20%, n = 71) sets while maintain the similar sex distribution to prevent sampling bias.

Feature selection

A feature importance ranking was performed using the Random Forest (RF) algorithm on the training set. Feature importance was quantified based on the mean decrease in impurity. Teeth were ranked by their cumulative importance, and a subset was selected where the cumulative importance reached 80%.

Model development and hyperparameter tuning

Four machine learning algorithms, including Logistic Regression (LR), Random Forest (RF), XGBoost, and Support Vector Machine (SVM), were employed to develop classification models for sex estimation based on the full set of 16 features and the selected features. For hyperparameter tuning, nested 5-fold cross-validation (CV) was implemented within the training set. The outer loop divided the training set into 5 folds for model evaluation, while the inner loop used 5-fold CV for hyperparameter optimization. All machine learning models were developed in R software (version 4.4.2) using ‘glmnet’ for LR, ‘randomForest’ for RF, ‘xgboost’ for XGBoost, ‘e1071’ for SVM, and ‘caret’ for cross-validation.

Evaluation and comparison of models

The performance of the optimized models was evaluated on the test set. A range of metrics were used for the performance evaluation of binary classification, including accuracy (ACC), precision, sensitivity, specificity, F1-score and area under the receiver operating characteristic curve (AUC-ROC). R package ‘pROC’ was used for AUC calculation. Moreover, precision-recall curves (PR curves) and calibration analyses were also performed. PR analysis assessed model resilience to class imbalance by calculating average precision (AP).

SHAP analysis

Shapley Additive exPlanations (SHAP) analysis was performed to elucidate the decision-making process of machine learning models and identify the most discriminative dental features for sex estimation. SHAP analysis for model interpretability consists of feature importance and contribution analysis, non-linear relationships and feature interactions, and individual prediction analysis. R package ‘SHAPforxgboost’ was used for SHAP analysis.

Results

Automatic AI segmentation

The automatic AI segmentation algorithm described by Liu et al. were used to evaluate a cohort of 398 CBCT images of human teeth (Fig. 1). The AI segmentation achieved high accuracy for teeth, with an average Dice score of 96.5%, an average mean Intersection over Union (mIoU) of 88.4%, an average Hausdorff distance (HD) of 1.62 mm, and an average surface distance (ASD) error of 0.12 mm.

Fig. 1.

The measurement of tooth volumes utilized an AI-based automated segmentation system. An example of (A) tooth segmentation and (B) volume measurement. The volume of the selected tooth was calculated automatically

A subset of samples was excluded due to segmentation failures, primarily because of severe artifacts such as motion blur, metal streaking, or image noise. Out of the 398 samples, 41 were excluded, resulting in a segmentation success rate of 89.7%. The results demonstrate the robustness of the AI segmentation approach in handling a large dataset, with the retained 357 samples (185 females and 172 males) proceeding to subsequent analyses (Table 1). The dataset was then split into training (N = 286, 148 females and 138 males) and test (N = 71, 37 females and 34 males) sets, maintaining similar sex and age distributions (Figure S2).

Table 1.

Information of samples’ distribution

	Male	Female	Total
Age (years)	23.08 ± 4.2	23.08 ± 3.7	23.08 ± 4.0
Total (n)	172	185	357

Descriptive statistics

Descriptive statistics for the volumes of the 16 teeth are presented in Table 2. Among the 357 participants, the mean, standard deviation (SD), and median of volume for each tooth were calculated separately for females and males. The result demonstrates that molars exhibit larger SD than other types of teeth, and mandibular central incisors show the smallest SD among all four tooth types.

Table 2.

Descriptive statistics of sample dataset

Measured teeth	Sex	Mean (mm3)	SD	Median (mm3)
Right maxillary central incisor	female	456.38	60.14	454.27
	male	506.53	65.76	500.81
Right maxillary canine	female	499.20	72.62	492.24
	male	614.26	87.01	606.84
Right maxillary first premolar	female	469.55	64.16	465.52
	male	525.35	63.71	528.04
Right maxillary first molar	female	835.08	101.36	839.24
	male	950.90	108.55	935.9
Left maxillary central incisor	female	459.58	61.00	458.23
	male	509.40	65.18	497.95
Left maxillary canine	female	496.69	77.25	492.15
	male	611.71	88.28	590.74
Left maxillary first premolar	female	470.77	65.72	467.58
	male	524.12	64.97	527.60
Left maxillary first molar	female	842.93	104.79	840.04
	male	948.23	109.09	945.57
Right mandibular central incisor	female	186.49	26.75	183.29
	male	203.23	28.90	202.16
Right mandibular canine	female	416.56	60.26	412.92
	male	525.17	75.99	521.01
Right mandibular first premolar	female	383.74	52.90	381.49
	male	431.27	54.49	431.13
Right mandibular first molar	female	847.85	117.02	852.78
	male	952.89	117.40	947.37
Left mandibular central incisor	female	186.03	27.34	184.08
	male	203.08	27.57	200.74
Left mandibular canine	female	416.59	61.58	418.09
	male	525.52	82.96	526.58
Left mandibular first premolar	female	381.47	54.65	382.93
	male	429.09	53.60	426.52
Left mandibular first molar	female	836.06	128.02	860.23
	male	946.94	116.06	941.3

Sex differences in tooth volumes

Normality test results for all tooth volume are presented in Table S1. The statistical analysis results of sex differences in tooth volume, including adjusted p-values and effect sizes are presented in Table 3. All 16 measured teeth showed statistically significant differences in volume between sexes. The negative values of effect sizes denote the directionality of the volumetric differences between sexes. Overall, males exhibited larger mean volumes across all tooth types compared to females, with adjusted p-values less than 0.0001 and effect sizes less than 1. Specifically, 11 teeth demonstrated exceptionally large effect sizes (absolute Cohen’s d > 0.8 and Cliff’s delta > 0.47), including all canines, all first molars and three first premolars with absolute Cohen’s d values ranging from 0.87 to 1.59 and Cliff’s delta values from 0.49 to 0.72. Furthermore, several other teeth, including the left maxillary first premolar and multiple central incisors, exhibited moderate-to-large effect sizes (Cliff’s delta: 0.45; Cohen’s d: 0.60–0.79) (Table 3; Fig. 2). Notably, four canines (right and left of the maxillary and mandibular) showed the greatest difference between males and females, exhibiting the highest statistical significance among the evaluated features (with the smallest adjusted P-values).

Table 3.

Statistical analysis of sex differences in tooth volume

Measured teeth	Method	P-value	Effect size	Adjusted P-value
Right mandibular canine	Welch’s t-test	1.27E-38	-1.590573	2.04E-37
Right maxillary canine	Welch’s t-test	1.73E-33	-1.440491	2.78E-32
Left mandibular canine	Mann Whitney U test	2.87E-32	-0.72423	4.60E-31
Left maxillary canine	Mann Whitney U test	1.32E-29	-0.692018	2.11E-28
Right maxillary first molar	Student’s t-test	2.23E-22	-1.104251	3.57E-21
Left maxillary first molar	Student’s t-test	1.46E-18	-0.985198	2.33E-17
Left mandibular first molar	Mann Whitney U test	1.91E-16	-0.503865	3.06E-15
Right mandibular first molar	Mann Whitney U test	9.93E-16	-0.49164	1.59E-14
Right mandibular first premolar	Student’s t-test	1.43E-15	-0.885616	2.29E-14
Left mandibular first premolar	Student’s t-test	2.13E-15	-0.879644	3.41E-14
Right maxillary first premolar	Student’s t-test	3.41E-15	-0.872579	5.45E-14
Left maxillary first premolar	Mann Whitney U test	2.02E-13	-0.449969	3.24E-12
Right maxillary central incisor	Student’s t-test	4.32E-13	-0.79727	6.91E-12
Left maxillary central incisor	Student’s t-test	6.70E-13	-0.790201	1.07E-11
Left mandibular central incisor	Student’s t-test	1.03E-08	-0.621251	1.64E-07
Right mandibular central incisor	Student’s t-test	2.73E-08	-0.602203	4.36E-07

Fig. 2.

Differences analysis in tooth volumes between females and males. The red boxes represent females, and the green ones represent males. The columns from left to right represent four tooth types: central incisors, canines, first premolars, and first molars. The top two rows indicate maxillary teeth and the bottom two rows indicate mandibular teeth

ROC analyses based on individual tooth

ROC analysis of individual tooth volumes revealed moderate to good discriminative ability for sex estimation, with AUC values ranging from 0.67 to 0.87 across the 16 teeth (Figure S3). Among all individual teeth, the four canines exhibited the highest AUCs: right mandibular canine (0.871), left mandibular canine (0.862), right maxillary canine (0.854), and left maxillary canine (0.846), respectively.

Model construction using multiple machine learning algorithms

Feature selection and comparison of performance metrics across algorithms

Feature selection using RF importance identified 9 teeth that cumulatively accounted for 80% of the total importance, with the mandibular and maxillary canines ranking at the top (Fig. 3A). Figure 3B displays a positive correlation between RF feature importance and the univariate AUC, demonstrating that teeth with higher feature importance tended to have stronger discriminatory ability. The ROC performance of models based on all 16 features and the selected 9 features on the independent test set was compared. For models using all features, the AUCs were 0.893 for LR, 0.903 for RF, 0.899 for SVM, and 0.896 for XGBoost (Fig. 3C). For models using the selected features, the AUCs were 0.886 for LR, 0.879 for RF, 0.886 for SVM, and 0.894 for XGBoost (Fig. 3D, Figure S4). Overall, the ROC curves indicate that models based on all 16 features generally outperformed those based on the selected 9 features.

Fig. 3.

Feature importance analysis and its impact on model performance. A Bar chart illustrating the 9 selected features based on a cumulative RF feature importance reaching 0.8. B Scatter plot demonstrating a strong positive correlation between the RF mean decrease gini (feature importance) and the univariate AUC of individual teeth. C ROC curves for the four machine learning models evaluated on the test set using all 16 features. D ROC curves for the corresponding models evaluated on the test set using the 9 selected features

The performance of four machine learning algorithms evaluated using nested cross-validation in the training set based on 16 features are presented in Table 4. RF exhibited the highest accuracy among models with ACC values of 0.818 and 0.859 on the training set and the test set, respectively. The mean nested cross-validation AUCs were 0.858 for LR, 0.892 for RF, 0.852 for SVM, and 0.872 for XGBoost, respectively. All four models demonstrated good discriminatory performance, with test-set AUCs close to or above 0.89. Among these models, RF achieved the highest test-set AUC (0.903), followed closely by SVM (0.899) and XGBoost (0.896), whereas LR showed the lowest but still competitive test-set AUC (0.893) (Fig. 4). On the test set, models trained on these 9 features yielded ACC values of 0.831 (LR), 0.803 (RF), 0.789 (SVM), and 0.817 (XGBoost). In comparison, models using all 16 features achieved slightly higher ACC values: 0.831 (LR), 0.859 (RF), 0.817 (SVM), and 0.803 (XGBoost). Figure 5A displays the comparative performance of the four models on the independent test set across Accuracy, Precision, Sensitivity, Specificity, and F1-score. Results showed that RF outperformed the other algorithms across most performance metrics.

Table 4.

Results based on all features using nested cross-validation on the training set

Models	Accuracy	Precision	Sensitivity	Specificity	F1-score	AUC
Logistic Regression	0.78 ± 0.02	0.78 ± 0.06	0.74 ± 0.06	0.82 ± 0.05	0.77 ± 0.02	0.86 ± 0.04
Random Forest	0.82 ± 0.03	0.81 ± 0.02	0.81 ± 0.08	0.82 ± 0.04	0.81 ± 0.04	0.89 ± 0.02
Support Vector Machine	0.77 ± 0.02	0.78 ± 0.04	0.72 ± 0.05	0.80 ± 0.06	0.75 ± 0.02	0.85 ± 0.02
XGBoost	0.81 ± 0.05	0.81 ± 0.05	0.80 ± 0.12	0.82 ± 0.06	0.80 ± 0.07	0.87 ± 0.05

Fig. 4.

Comparison of ROC curves derived from nested cross-validation and the independent test set for machine learning models based on all 16 teeth. The light blue curves represent ROC curved from the outer folds of nested cross-validation within the training set (with the mean AUC provided), and the red line represents the final evaluation results obtained on the independent test set

Fig. 5.

Comprehensive performance evaluation of the machine learning models on the independent test set (based on all features). A comparison of model performance metrics. B Confusion matrices for four models on the test set. C Precision-recall curves (PRC) for the four models on the test set. D Calibration curves for the four models on the test set

Confusion matrix, PR curves and model calibration curves analyses

Confusion matrices (Fig. 5B, Figure S5) illustrate the classification characteristics of each algorithm. Comparing these four models, RF achieved the highest number of True Positives (N = 112 on the nested CV, N = 28 on the test set) and the lowest number of False Negatives (N = 26 on the nested CV, N = 6 on the test set), which is consistent with its high overall accuracy. In the test dataset, the PR curves analyses show that SVM achieved the best precision-recall performance, followed closely by LR and RF (Fig. 5C). The calibration curves indicate that all models displayed acceptable calibration, with predicted probabilities following the ideal diagonal trend (Fig. 5D).

SHAP analysis for model interpretability

SHAP analysis was conducted for XGBoost model, which was performed best among four models. The beeswarm plot (Fig. 6A) reveals a clear hierarchy of dental volumetric features contributing to sex classification. Mandibular canines (both left and right) demonstrated the highest impact on model predictions, followed by the right maxillary canine. The dependence plots (Fig. 6B) for the top 20% influential features (left mandibular canine, right mandibular canine, and right maxillary canine) demonstrate a non-linear relationship between tooth volumes and their impact on sex classification. At low volumes, the SHAP values are strongly negative, indicating a significant push towards female prediction. As the volume increases, the SHAP values rapidly transitions through zero and becomes increasingly positive, indicating a stronger influence towards male prediction. The individual SHAP explanations (Fig. 6C) shows example 1 and 3 predicted as ‘male’ due to significant positive contributions from tooth volumes, while example 2 predicted as ‘female’ due to negative contributions.

Fig. 6.

SHAP analysis for XGBoost model: (A) The beewarm plot of the importance of dental volumetric features contributing to sex classification; (B) The dependence plots of the top 20% influential features; (C) The individual SHAP explanations of two examples

Discussion

Sex identification represents a fundamental parameter in the biological profiling of human remains, possessing significant forensic anthropological, forensic odontological, and archaeological value [2]. Compared to conventional sex estimation approaches, teeth offer significant advantages [35]. Teeth have been reported to have high sexual dimorphism in studies based on linear dental measurements, such as mesio-distal, buccal-lingual, diagonal distances of crown, or length of root [20–23]. However, different reference points on the tooth surface can lead to errors or deviations in the measurement of linear indicators. Generally speaking, volumetric evaluation is less susceptible to error because it takes into account the entire three-dimensional structure of teeth, while the reference points for linear measurements might be variable among different examiners. In this study, we firstly revealed that tooth volume has significant sexual dimorphism among the population of north China. Similar to a study based on a Brazilian population, they found that male had larger dental crown volume than female [29]. They explained the difference of dental volume between males and females might be due to the fact that Y chromosome promotes both amelogenesis and dentinogenesis, whereas the X chromosome only affects amelogenesis [36–38].

The descriptive statistics of the dataset revealed that the first molars exhibited the largest standard deviations in volume. This greater variability in tooth volume may potentially be attributed to larger inherent structural or developmental variations characteristic of molars. In our study, canines play the most important role in sex prediction when compared with other dental types such as incisors, premolars and molars, which is in agreement with previous research. Ceren et al. [26] evaluated volume of the maxillary canine using CBCT images and showed that the tooth volume of males is 660 ± 116 mm³ and female is 504 ± 78mm³. The volume tooth of female in Ceren’s study is similar to our study (right: 499.20 ± 72.62 mm³; left: 496.69 ± 77.25 mm³), and the volume tooth of male is slightly larger than the values of our study (right: 614.26 ± 87.01 mm³; left: 611.71 ± 88.28 mm³), which might due to population differences. Moreover, the standard deviation of canine volume in Ceren’s study is greater than our study, which might be explained that the inter-sample consistency of AI automatic segmentation approach is superior than manual segmentation. In this study, tooth segmentation was performed using the fully automatic AI segmentation system developed by Liu et al. [32]. The shift from manual or semi-automatic methods to a fully automated workflow drastically enhances efficiency, allowing for the rapid processing of large volumes of CBCT data. Crucially, automation minimizes human intervention, thereby reducing the inherent subjectivity and potential for inter- and intra-observer variability associated with traditional measurement techniques like caliper-based measurements on casts or even manual landmarking on digital models.

Thus, the important contribution of this study is the use of AI-based automatic segmentation of tooth volume combined with multiple machine learning algorithms, this framework facilitates the development of robust binary classification models for sex prediction. Unlike simpler statistical models which might struggle to capture complex patterns, machine learning algorithms are well-equipped to identify and utilize intricate, potentially non-linear relationships between the volumes of teeth and sex, and have the potential to achieve higher accuracy and robustness in sex estimation. Here, the null hypothesis for this study posited that there is no significant difference in tooth volumes between sexes, or equivalently, that machine learning models based on these volumes would not achieve predictive accuracy significantly better than random chance (AUC = 0.5). Our results reject this null hypothesis, all models evaluated using nested cross-validation on the training set obtained mean AUC values at or above 0.89, herein, RF model exhibited the highest ACC value of 82% and AUC value of 0.89 ± 0.02. RF model also demonstrated high discriminative ability with the highest ACC value of 85.9% and AUC value of 0.903 on the independent test set, indicating strong discriminative power attributable to sexual dimorphism in tooth volumes. This supports the alternative hypothesis that volumetric features enable accurate sex estimation.

Compared with previous studies using classical methods, such as linear discriminant analysis (LDA), our study shows better performance in sex estimation. While LDA is simpler, more flexible, and proposed as a useful tool for skeletal sex estimation [39], it performs modestly in craniomaxillofacial studies, with accuracies of 65% using craniometric parameters [40] and 63.24% using left maxillary sinus volume [41]. Morphometric assessment methods are laborious and depends on precise anatomical landmarks determination [42]. In recent years, advances in three-dimensional image analysis have made these assessments less subjective and more efficient through computer-aided techniques. CBCT imaging provides a platform for linear and/or volumetric measurements of dentition [43]. Daniele et al. [29] reported an average accuracy of 83.7% using combined volumetric analysis of the upper and lower canines. Another study examining 610 CBCT scans demonstrated that total tooth length had the highest predictive power, with AUC values of 0.77 and 0.75 using LightGBM and LR algorithms, respectively [44]. However, these methods typically rely on manual landmark identification, which is prone to subjective bias and inter-observer variability. Our results showed comparable or improved performance relative to these previous CBCT-based findings and suggest that automated AI-driven tooth segmentation combined with machine learning may provide a promising framework for sex estimation.

We also compared the performance of models using all variables with models based on a reduced 9-feature subset selected according to feature importance. The reduced-feature models preserved most of the predictive signal, but they did not consistently outperform the full-feature models on the independent test set. In general, models using all 16 features achieved slightly better ACC and AUC values. A possible explanation is that although feature selection improves parsimony and reduces dimensionality, the 16 variables do not constitute high-dimensional data for the dataset in the current study, thus lower-ranked variables may still contribute complementary information when combined with other teeth. Precision-recall and calibration analyses (PRC-AUC values of 0.830–0.873) confirm robust handling of class imbalance and probability estimates, collectively demonstrating that AI-driven volumetric analysis improves sex estimation accuracy. To elucidate the decision-making process of the XGBoost model and identify the most discriminative dental features for sex estimation, SHAP analysis was performed. This analysis provides insights into both global feature importance and feature-specific contributions to individual predictions [45]. The substantial positive SHAP values showing rightward distribution for most tooth features when their volumes are high (yellow points in Fig. 6A) indicate that larger teeth volumes strongly predict male classification. Notably, left mandibular canine, right mandibular canine, and right maxillary canine ranked the top 3 important features in estimating sex. While the left maxillary canine exhibited a relatively high importance ranking among features, it was observed to be slightly lower compared to the other three canines. Several factors might contribute. One potential factor is the larger SD observed for the left maxillary canine volume than that of the others in our dataset, suggesting greater variability. Additionally, the limited sample size in this study may introduce certain degree of noise or instability into the feature importance calculations, thereby SHAP might assign lower importance due to noise. The SHAP analysis provides not only improved model transparency but also valuable insights into the complex interplay of tooth volume features in sex prediction.

Additionally, multiple evaluation metrics including accuracy, sensitivity, specificity, precision, F1, and AUC were used in the present study, which has more detailed model evaluation than cut-off value. Some previous studies using teeth or other craniomaxillofacial morphology for sex estimation had satisfactory results, for instance, with accuracy value of 94.48% for frontal sinuses [46], and 86% for the anterolateral angle of the maxillary sinus [47]. However, these studies assessed models’ performance using accuracy as the primary, and often sole, performance indicator. One of the limitations of relying solely on accuracy for evaluating classification models is that accuracy is susceptible to skewed class distributions such as imbalanced datasets. Recognizing this limitation, our study used a comprehensive evaluation strategy, incorporating metrics abovementioned, for assessing models’ performance in sex estimation. Among these evaluation metrics, the AUC values provide a comprehensive metric for assessing model performance across all possible classification thresholds.

While this study demonstrates the potential of automatically segmented tooth volumes for forensic sex estimation using machine learning methods, it is essential to acknowledge that the study exists several limitations. Primarily, the sample size of 357 individuals, though sufficient for a pilot exploration, remains relatively modest for training highly complex machine learning models intended for broader applications. Furthermore, the study populations were drawn retrospectively between 18 and 30 from a single clinical center in Hebei province, China, which may limit the representativeness of the sample in terms of age distribution, and ethnic diversity. Therefore, further study will focus on larger, independent datasets encompassing diverse populations. In this study, we divided the entire dataset at the outset into an 80% training set and a 20% held-out test set, with all cross-validation, hyperparameter tuning, and model selection performed exclusively within the training set. The held-out test set serves as an independent internal validation by assessing performance on data not used in training. While this approach provides a robust estimate of model performance within our cohort and helps mitigate overfitting, future studies should incorporate external validation from different institutions or populations to confirm generalizability of the findings. As a retrospective study, lacking clinical examinations to assess dental erosion or alterations and potential sources of bias related to image quality, such as variations in CBCT resolution or artifacts, could affect tooth volume measurements and introduce bias. Future prospective studies should incorporate such assessments.

Conclusion

Our study firstly demonstrates that teeth volumes show significant difference between the sexes in population of north China. We utilized an AI-based automated segmentation accompanied with machine learning modelling pipeline to accurately estimate sex using tooth volume features from CBCT images within the studied northern Han Chinese population. Based on volumes of 16 teeth, four machine learning algorithms using nested cross-validation obtained accuracies of 0.766–0.818 and AUCs of 0.852–0.892 on the training set. On the independent test set, all machine learning models exhibited promising performance, with accuracies of 0.803–0.859 and AUCs of 0.893–0.903. Our study shows the effectiveness of tooth volume in sex estimation among northern Han Chinese populations. Future multicenter studies are required to validate its generalizability across diverse demographics and varying imaging protocols.

Abbreviations

AI

Artificial Intelligence

AUC

Area Under the ROC Curve

CBCT

Cone Beam Computed Tomography

FDI

Fédération Dentaire Internationale

LR

Logistic Regression

MICE

Multiple Imputation by Chained Equations

RF

Random Forest

ROC

Receiver Operating Characteristic

SD

Standard Deviation

SHAP

Shapley Additive exPlanations

SVM

Support Vector Machine

Authors’ contributions

J. W.: Conceptualization, Methodology, Validation, Formal analysis, Investigation, Data curation, Writing – original draft. X. L.: Conceptualization, Methodology, Validation, Resources, Funding acquisition, Data curation, Writing – review and editing. J. Z.: Methodology, Formal analysis, Data curation. J. W.: Methodology, Formal analysis, Data curation. J. L.: Methodology, Formal analysis, Validation, Data curation. C. W.: Methodology, Formal analysis, Validation, Data curation. E. Z.: Investigation, Methodology, Formal analysis, Data curation. T. S.: Visualization, Writing – review and editing, Data curation. L. Y.: Visualization, Writing – review and editing, Data curation. S. L.: Methodology, Data curation, Writing – review and editing, S. L.: Conceptualization, Writing – review and editing, Supervision. B. C.: Conceptualization, Funding acquisition, Writing – review and editing, Supervision, Project administration.

Funding

This research was financially supported by Postdoctoral Fund of Hebei Medical University (No. 30705010059), the Major Program of National Natural Science Foundation of China (No. 82293650/82293651), Science Research Project of Hebei Education Department (QN2025342), the Natural Science Foundation of Hebei Province (H2024206170), the Hebei Provincial Backbone Talent Program For Postdoctoral Platform (B2025003013), Medical Science Research Project of Hebei (No.20240885/20250776), University Students Innovative Program (USIP2025152).

Data availability

All data generated or analysed during this study are included in this published article and its supplementary information files. The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate

This study was conducted retrospectively and the ethical standards of the institutional and/or national research committee and the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards were followed in the whole process. Informed consent was approved by the Ethics Committee of Hebei Medical University (No. 2025-088).

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.