Den-SOFA: Dental Student Outcome Forecasting Assistant using explainable machine learning models

Abstract

Background

Artificial intelligence (AI) and machine learning (ML) are increasingly being explored in dental education to support academic assessment and identify students at risk of poor performance. However, predictive modeling in this setting remains challenging because complex, nonlinear relationships among academic and demographic variables influence student achievement. Moreover, the value of such models lies not only in prediction but also in interpretability. This study evaluated Den-SOFA, an explainable ML framework, as an exploratory proof-of-concept for predicting outcomes on restorative dentistry exit examinations.

Methods

This descriptive-analytical educational data-mining study analyzed data from 96 dental students. Twenty-six academic and demographic variables were included, such as cumulative GPA, theoretical and practical course grades, academic progression indicators, and demographic characteristics. The predictive performance of logistic regression, random forest, XGBoost, CatBoost, and artificial neural networks (ANN) was evaluated for binary (pass/fail) and multiclass (grade A–F) outcomes. Performance was assessed using accuracy, F1-score, AUC-ROC, MCC, sensitivity, and specificity. Model interpretability was examined using SHAP analysis.

Results

For pass/fail prediction, the ANN model achieved the highest discrimination (AUC-ROC = 0.906; accuracy = 0.86; MCC = 0.71), while the random forest showed comparable performance (AUC-ROC = 0.864; accuracy = 0.86) with greater interpretability. Multiclass grade prediction showed only moderate performance across models, with the best AUC-ROC of 0.775. SHAP analysis identified academic term, cumulative GPA, phantom laboratory performance, and theoretical course grades as the most influential predictors, whereas demographic variables contributed minimally to model performance.

Conclusion

Den-SOFA provides an exploratory and hypothesis-generating framework for examining how explainable ML may be applied to dental education data. In this single-institution dataset, the models demonstrated moderate-to-good internal predictive performance, particularly for binary outcome prediction. External validation in larger and more diverse cohorts is required before any operational use in student support or assessment planning can be considered.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12909-026-09949-3.

Introduction

High-stakes examinations are central to health-professions education, serving as gatekeeping mechanisms for academic progression, clinical readiness, and licensure eligibility [1]. These assessments are intended to verify that learners meet minimum standards across knowledge, procedural competence, and professional judgment [2]. Despite their widespread use, such examinations carry well-documented limitations, including psychological stress, limited predictive validity for real-world clinical performance, examiner subjectivity, and susceptibility to measurement bias [3–5]. These shortcomings are particularly concerning within competency-based education frameworks, where assessment should reflect the full range of professional capability rather than performance on a single occasion [6]. In dental education, these challenges are further amplified. Student competence depends on the integration of theoretical knowledge, psychomotor skill development, hand–eye coordination, and clinical decision-making, dimensions that written or single high-stakes examinations consistently fail to capture in full [7, 8]. The result is a persistent gap between examination outcomes and actual clinical readiness, with limited capacity for educators to identify at-risk students early and provide targeted support before failure occurs.

Artificial intelligence (AI) and machine learning (ML) have emerged as promising tools for addressing these gaps [7, 8]. Prior studies have applied logistic regression, random forests, gradient boosting, support vector machines, and artificial neural networks to educational datasets, demonstrating that ML can detect nonlinear learning patterns, handle large and complex datasets, and generate predictions from combinations of academic and demographic variables [8–12]. Ensemble methods have generally outperformed classical models in robustness and accuracy, while neural networks have shown particular strength in capturing complex predictor interactions [13]. However, most existing models function as black boxes: they produce predictions without offering educators transparent, interpretable explanations of the factors driving those predictions. This limits their practical utility in real educational settings, where instructors must understand why a student is at risk to intervene meaningfully [7, 8, 14]. A further gap exists in the specific context of dental education. While ML-based academic prediction has been applied across several health-profession domains, studies that combine predictive modeling with explainable AI (XAI) specifically for dental students’ exit-examination outcomes remain scarce. The need for transparent and educator-friendly prediction frameworks in this domain is unmet.

The present study addresses this gap by proposing Den-SOFA (Dental Student Outcome Forecasting Assistant), an ML-based framework integrating XAI methods to predict exit-examination outcomes among undergraduate dental students. Den-SOFA targets both binary (Pass/Fail) and multiclass (A–F) outcome classification and employs SHAP (SHapley Additive exPlanations) values to generate global and local explanations of each model’s predictions. It is important to note that while Den-SOFA predicts examination outcomes based on historical academic data, its value for improving student performance ultimately depends on how educators translate these predictions into concrete support actions. The study is guided by four research questions:

• RQ1: Which ML algorithms achieve the highest predictive accuracy for dental students’ final academic outcomes?

• RQ2: Which academic and demographic variables demonstrate the strongest predictive association with exit-examination performance?

• RQ3: To what extent can XAI methods quantify the contribution and relative importance of individual predictors?

• RQ4: How can educators leverage Den-SOFA outputs for early identification of at-risk students and the design of targeted interventions?

Materials and methods

Study design and data source

This descriptive-analytical study employed an educational data mining approach to develop and evaluate the Dental Student Outcome Forecasting Assistant (Den-SOFA) system. Data were retrospectively collected from the academic archives of the Faculty of Dentistry, AJA University of Medical Sciences, Tehran, Iran. The dataset comprised 96 undergraduate dental students who completed the restorative dentistry exit examination between 2019 and 2022. All academic records were extracted from the university’s educational management system. Student-identifiable information was rigorously anonymized and excluded before analysis, ensuring data privacy and ethical compliance. The source code and implementation files are publicly available at: https://github.com/smartdxgen/Den-SOFA.

Ethics approval

The study protocol was reviewed and approved by the Research Ethics Committee of AJA University of Medical Sciences, Tehran, Iran (Ethics Code: IR.AJAUMS.REC.1403.230). Given the retrospective nature of the data collection and the anonymization of all records, the ethics committee waived the requirement for informed consent from individual students.

Methodological framework

The objective was to predict student performance in binary (Pass/Fail) and multiclass (grades A–F) exit-examination categories. The methodological pipeline encompassed five key phases: (1) Data Collection (2), Data Preprocessing (3), Model Development (4), Model Evaluation, and (5) Explainable AI (XAI) Analysis. The XAI Framework of the Den-SOFA is illustrated in Fig. 1.

Fig. 1.

Den-SOFA Predictive Framework: Process Workflow

Three hypotheses guide the study: (1) Ensemble ML models (e.g., Random Forest, Gradient Boosting) will achieve significantly higher predictive accuracy for exit-examination outcomes than classical logistic regression (2). Curriculum-embedded academic indicators will demonstrate stronger predictive associations with exit-examination performance than demographic variables (3). SHAP-based explanations will identify a consistent set of high-importance predictors across both binary (Pass/Fail) and multiclass (A–F) outcome models.

Data collection

This study employed a descriptive–analytical design within an educational data mining and machine learning framework to predict dental students’ performance in the Restorative Dentistry exit examination. Before analysis, the dataset underwent quality assessment and preprocessing. The initial dataset consisted of 102 undergraduate dental student records. After data screening, six records were excluded due to incomplete or missing essential academic information. Consequently, the final analytic dataset included 96 students, and these records were used in the subsequent modeling and analysis.

Input variables

The dataset comprised demographic, academic, and performance-related variables extracted from the university’s educational management system.

• Demographic variables included age, gender, birth order, region quota category, scholarship status, and residence in Tehran.

• The region quota system represents an educational policy designed to balance university admission opportunities across geographic regions and socioeconomic conditions. For analytical purposes, quota categories were grouped into high-competition (Region 1), moderate-competition (Region 2), and low-competition (Region 3) categories.

• Academic variables included entrance examination score, grade point average (GPA), number of passed and failed units, academic term, and the Basic Sciences Comprehensive Examination score. This examination is a national standardized assessment administered at the end of the fourth semester of dental education and designed to evaluate students’ foundational knowledge in the biomedical and preclinical sciences. Course-specific academic performance variables included Restorative Dentistry Theory I, Restorative Dentistry Theory II, Dental Anatomy, and Phantom laboratory training, with all grades recorded on a 0–20 scale, consistent with the Iranian medical education grading system.

• To improve predictive performance, several derived variables were constructed through variable engineering. These included the average theoretical score (combining Theory I and Theory II), the average practical score (combining Phantom and Dental Anatomy), the number of failed courses, the average grade of failed courses, and the academic success rate calculated based on passed units.

Outcome variables

The final restorative dentistry exit examination score served as the primary continuous outcome variable. For classification modeling, this score was subsequently transformed into categorical grade levels (A–F) and a binary pass/fail outcome. A schematic representation of the extracted variables and engineered variables is presented in Fig. 2.

Fig. 2.

Schematic representation of data variables and derived variables used in the Den‑SOFA study

Preprocessing of data

Descriptive statistical analyses were conducted to summarize the demographic and academic characteristics of the study population. Continuous variables were reported as mean ± standard deviation (SD), while categorical variables were presented as frequencies and percentages. Missing values in non-critical variables were handled using median imputation for continuous variables and mode imputation for categorical variables. To minimize the risk of data leakage, all preprocessing procedures used for predictive modeling were performed after data splitting and were fitted only on the training data. The learned preprocessing parameters were then applied to the validation or test data as appropriate. Importantly, imputation parameters were estimated exclusively from the training data within each cross-validation fold and were not calculated using the full dataset.

Categorical variables were transformed into machine-readable formats using one-hot or ordinal encoding, depending on the nature of the variable. Encoding procedures were also fitted within the training folds only and then applied to the corresponding validation or test data. For models sensitive to feature scaling, continuous variables were standardized using parameters estimated from the training data only.

Several derived academic variables were generated from existing educational variables to improve the representational capacity of the models. These included the average theoretical score, calculated from Restorative Dentistry Theory I and II; the average practical score, calculated from Phantom Laboratory and Dental Anatomy courses; and the academic success rate, defined as the ratio of passed units to the total number of attempted units. To prevent target leakage, all predictor variables (including engineered features) were derived exclusively from academic records available before administration of the final restorative dentistry exit examination. No variables were computed using information from the exit examination score itself, nor were any post-outcome academic indicators included.

The final restorative dentistry exit examination score was initially recorded as a continuous variable. For classification modeling, this outcome variable was transformed into categorical grade levels (A–F) and a binary pass/fail outcome, enabling both multiclass and binary prediction tasks.

Inferential statistical analyses were conducted to examine relationships between educational variables and students’ final examination scores. Independent-samples t-tests were used to compare mean values of continuous variables between outcome groups. Pearson correlation analysis was used to evaluate linear relationships among continuous variables such as GPA, theoretical course averages, and final examination scores. For variables that did not satisfy normality assumptions or exhibited ordinal characteristics, Spearman’s rank correlation coefficient was used to assess monotonic associations.

To address class imbalance, the Synthetic Minority Oversampling Technique (SMOTE) was applied only to the training data during model development. SMOTE was not applied before train-test splitting and was not applied to the validation or test sets. During cross-validation, SMOTE was performed separately within each training fold after preprocessing and before model fitting, while the corresponding validation fold remained untouched. This approach was used to reduce the risk of information leakage and overly optimistic performance estimates.

Predictive modeling framework

The predictive modeling framework was designed to evaluate the effectiveness of multiple supervised machine learning algorithms for forecasting dental students’ academic outcomes on the final restorative dentistry examination. To compare classical statistical approaches with advanced machine learning techniques, five predictive models were developed and evaluated: Logistic Regression, Random Forest, XGBoost, CatBoost, and Artificial Neural Network. Details of the model architectures and hyperparameter settings are provided in Supplementary Tables S1 and S2.

• Logistic Regression (LR): Logistic Regression was implemented as a baseline linear classifier, providing interpretable coefficients that describe the relationship between predictor variables and the probability of academic success or failure.

• Random Forest (RF): Random Forest, an ensemble tree-based learning algorithm, was employed to capture nonlinear relationships and complex interactions among variables. The method constructs multiple decision trees using bootstrap sampling and aggregates their predictions to improve model stability and reduce variance.

• Extreme Gradient Boosting (XGBoost): XGBoost, a gradient boosting framework, was utilized due to its strong performance in structured data prediction tasks. The algorithm sequentially builds decision trees that correct the errors of previous trees while incorporating regularization mechanisms to prevent overfitting.

• Categorical Boosting (CatBoost): CatBoost was selected because of its ability to effectively manage heterogeneous datasets containing categorical variables and its use of ordered boosting to reduce prediction bias during training.

• Artificial Neural Network (ANN): A multilayer feed-forward artificial neural network was developed to model complex nonlinear interactions among educational variables. The network consisted of multiple hidden layers with nonlinear activation functions, enabling the capture of higher-order relationships between predictors and academic outcomes.

Evaluating ML models’ performance

To ensure a fair and reproducible comparison across algorithms, all ML models were evaluated using identical predictor variables and the same data-splitting strategy. The dataset was divided into training and testing subsets using stratified sampling to preserve the proportional distribution of outcome classes. A 70%–30% split was implemented, with the training set used for model development and hyperparameter tuning, and the held-out test set reserved exclusively for final evaluation on previously unseen data. Hyperparameter optimization was performed using grid search with five-fold cross-validation within the training set, enabling systematic exploration of parameter combinations while reducing the risk of overfitting. The best-performing parameter combination was selected based on cross-validated performance, and the final model was then evaluated once on the held-out test set. Model performance was assessed using multiple complementary evaluation metrics to provide a comprehensive comparison of predictive capabilities. For the binary classification task (pass/fail prediction), the following metrics were calculated (the evaluation metrics were defined in Table 1):

Table 1.

Evaluation metrics definition

Metrics’ Formula
Accuracy = (TP + TN) / (TP + TN + FP + FN) (1)
Precision = TP / (TP + FP) (2)
Recall (Sensitivity) = TP / (TP + FN) (3)
Specificity = TN / (TN + FP) (4)
F1-Score = 2 × (Precision × Recall) / (Precision + Recall) (5)
MCC = (TP × TN − FP × FN) / √((TP + FP)(TP + FN)(TN + FP)(TN + FN)) (6)

TP (true positives) represents correctly predicted positive cases, FP (false positives) represents incorrectly predicted positive cases, TN (true negatives) represents correctly predicted negative cases, and FN (false negatives) represents incorrectly predicted negative cases

• Discriminative Performance: The ability of the models to distinguish between outcome classes was evaluated using the Area Under the Receiver Operating Characteristic Curve (AUC) and overall accuracy.

• Class-Specific Performance: To assess prediction performance at the class level, precision (positive predictive value), recall (sensitivity), and the F1-score were calculated. The F1-score is the harmonic mean of precision and recall and provides a balanced measure when class distributions are unequal.

• Reliability and Calibration: Model calibration was assessed using calibration curves, which evaluate the agreement between predicted probabilities and observed event frequencies. Because class imbalance can affect predicted probabilities and calibration, these results were interpreted cautiously, especially given the limited sample size.

Explainable AI (XAI) for predicting students’ performance

To improve model interpretability, XAI techniques were applied to the trained predictive models. Specifically, Shapley Additive exPlanations (SHAP) values were computed for the best-performing models to quantify the contribution of each input variable to the model output at the global and local levels. At the global level, SHAP summary plots and variable-importance rankings were generated to visualize the overall influence of predictors across the dataset. At the local level, SHAP waterfall plots were used to illustrate how individual variables contributed to a single student’s prediction. These explanations were used to provide post-hoc insight into model behavior and to identify variables that were most influential in the predictions. However, the SHAP analysis in this study should be interpreted as descriptive and hypothesis-generating rather than as evidence of validated educational utility.

Results

Descriptive characteristics of the study population

Following the exclusion of outliers and incomplete records (n = 6), data from 96 undergraduate dental students were included in the final analysis. The distribution of demographic and academic characteristics is summarized in Table 2 and illustrated in Figure S1 in Supplementary A.

Table 2.

Association between predictor variables and outcome measures

Characteristic	Overall (N = 96)	Fail (n = 34)	Pass (n = 62)	P-Value
Gender				0.287
Male	75 (78.1%)	31 (32.3%)	44 (45.8%)
Female	21 (21.9%)	3 (3.1%)	18 (18.8%)
Age Category (age_cat)				0.066
≤ 23	29 (30.2%)	15 (15.6%)	14 (14.6%)
24	25 (26.0%)	10 (10.4%)	15 (15.6%)
25	19 (19.8%)	4 (4.2%)	15 (15.6%)
≥ 26	23 (24.0%)	5 (5.2%)	18 (18.8%)
Region-quota				0.296
1	32 (33.0%)	4 (4.0%)	28 (29.0%)
2	46 (48.0%)	7 (7.0%)	39 (40.0%)
3	17 (18.0%)	5 (5.0%)	12 (13.0%)
Missing	1 (1.0%)	0 (0.0%)	1 (1.0%)
Entrance Score (entrance_cat)				0.486
< 9600	69 (71.9%)	22 (22.9%)	47 (49.0%)
> 9600	21 (21.9%)	9 (9.4%)	12 (12.5%)
Missing	6 (6.2%)	3 (3.1%)	3 (3.1%)
GPA Category (gpa_cat)				0.002
< 16	23 (24.0%)	16 (16.7%)	7 (7.3%)
16–16.5	22 (22.9%)	5 (5.2%)	17 (17.7%)
16.5–17	29 (30.2%)	9 (9.4%)	20 (20.8%)
≥ 17	21 (21.9%)	4 (4.2%)	17 (17.7%)
Theory1 Category (theory1_cat)				0.007
< 14	32 (33.3%)	15 (15.6%)	17 (17.7%)
14–16	42 (43.8%)	18 (18.8%)	24 (25.0%)
≥ 16	19 (19.8%)	1 (1.0%)	18 (18.8%)
Missing	3 (3.1%)	0 (0.0%)	3 (3.1%)
Theory2 Category (theory2_cat)				0.495
< 14	3 (3.1%)	2 (2.1%)	1 (1.0%)
14–16	33 (34.4%)	12 (12.5%)	21 (21.9%)
≥ 16	60 (62.5%)	20 (20.8%)	40 (41.7%)
Anatomy Category (anatomy_cat)				0.476
< 16	26 (27.1%)	7 (7.3%)	19 (19.8%)
16–18	47 (49.0%)	17 (17.7%)	30 (31.3%)
≥ 18	23 (23.9%)	10 (10.4%)	13 (13.5%)
Phantom Category (phantom_cat)				0.010
< 14	30 (31.3%)	16 (16.7%)	14 (14.6%)
14–16	39 (40.6%)	14 (14.6%)	25 (26.0%)
≥ 16	27 (28.1%)	4 (4.2%)	23 (24.0%)
Basic Sciences Category				0.096
< 74	3 (3.1%)	2 (2.1%)	1 (1.0%)
74–76	1 (1.0%)	1 (1.0%)	0 (0.0%)
≥ 77	88 (91.7%)	28 (29.2%)	60 (62.5%)
Missing	4 (4.2%)	3 (3.1%)	1 (1.0%)
Grade Category (grade_category)				< 0.001
A (17–20)	16 (16.7%)	0 (0.0%)	16 (16.7%)
B (15–17)	27 (28.1%)	0 (0.0%)	27 (28.1%)
C (12–15)	19 (19.8%)	0 (0.0%)	19 (19.8%)
D (10–12)	20 (20.8%)	20 (20.8%)	0 (0.0%)
F (< 10)	14 (14.6%)	14 (14.6%)	0 (0.0%)

The mean age of participants was 24.5 ± 1.44 years, with an age range of 22 to 29 years. The highest concentration of students was observed in the 23–25-year age group, indicating a relatively homogeneous age distribution. Regarding gender, 84.4% of students were male, and 15.6% were female.

The mean cumulative grade point average (GPA) was 16.48 ± 0.62, ranging from 14.70 to 17.94. The GPA distribution was approximately normal and symmetric, with minimal skewness (Skewness = − 0.27), and most values clustered between 16 and 17, reflecting strong overall academic performance. The mean entrance examination score was 9572.83 ± 151.70, with a median of 9550 and a range of 9200 to 10,000. The distribution demonstrated positive skewness (Skewness = 0.96) and moderate kurtosis (Kurtosis = 1.15), indicating that while most students clustered near the mean, a subset achieved exceptionally high entrance scores.

Analysis of regional quota distribution showed that the largest proportion of students belonged to Region 2 (moderate competition; 44.4%), followed by Region 1 (high competition; 32.2%) and Region 3 (lower competition; 17.8%). A small proportion (5.6%) had missing or unspecified quota information (e.g., guest students). This distribution reflects the heterogeneous admission pathways within the cohort and potential differences in educational background.

In terms of coursework performance, the mean theoretical score (average of Restorative Dentistry Theory I and II) was 15.44 ± 1.26, with a range of 12.25 to 18.00. Theoretical scores followed a near-normal distribution (Skewness = 0.03, Kurtosis = − 0.55), indicating relatively uniform theoretical knowledge acquisition across students. The mean practical score (average of Phantom Laboratory and Dental Anatomy courses) was 15.75 ± 1.25, with a median of 15.69 and a range of 12.50 to 18.63. Practical scores exhibited a highly symmetric distribution (Skewness = − 0.004, Kurtosis = − 0.13) and were slightly higher than theoretical scores, suggesting stronger performance in skill-based and preclinical activities.

Associations between student characteristics and the final restorative dentistry examination outcome (Pass vs. Fail) were evaluated using independent t-tests and categorical comparisons. Among demographic variables, gender was not significantly associated with the outcome (p = 0.287), and age category also did not reach statistical significance (p = 0.066), suggesting a limited influence of these demographic factors on examination performance. In contrast, prior academic performance was significantly related to the outcome. GPA category was significantly associated with pass/fail status (p = 0.002). More than 80% of students with a GPA > 17 passed the examination, whereas over half of the students with a GPA < 16 were classified as failing. Regarding theoretical coursework, Restorative Dentistry Theory I showed a significant association with final examination status (p = 0.007), while Theory II did not demonstrate a statistically significant relationship (p = 0.495). Among practical courses, the Phantom laboratory course was the only practical variable significantly associated with examination outcome (p = 0.010), highlighting the importance of preclinical technical skills in predicting final clinical performance. Notably, grade category exhibited the strongest discriminatory power between the Pass and Fail groups (p < 0.001). All students with Grades A–C achieved a Pass outcome. In contrast, students with Grades D or F were exclusively classified as Fail, demonstrating a clear threshold effect across academic performance categories.

The final restorative dentistry exit examination score had a mean of 13.45 ± 3.81, with scores ranging from 2.00 to 18.40 and a median value of 14.00. The distribution showed mild negative skewness, indicating a tendency toward higher scores among participants. Based on predefined grading categories, the highest frequency was observed in Grade B (15–17), with 25 students (28.4%). This was followed by Grade D [10–12] in 21.6% of students and Grade C [12–15] in 18.2%. Additionally, 16 students (18.2%) achieved Grade A (17–20), while 12 students (13.6%) fell into Grade F (< 10). Overall, more than two-thirds of students demonstrated moderate to good academic performance. Regarding the binary outcome, 61 students (67.8%) passed the exit examination, whereas 29 (32.2%) failed. Although the majority achieved passing outcomes, the relatively high failure rate highlights persistent educational challenges in specific courses or academic stages.

Correlations between academic variables

To examine relationships among categorized academic variables, Pearson correlation analysis was conducted, and the results were visualized using a correlation heatmap (Fig. 3). In the heatmap, red shades indicate positive correlations, meaning higher values in one variable are associated with higher values in another. In contrast, blue shades indicate negative correlations, reflecting inverse relationships between variables. The average practical score (Practical_Avg) demonstrated a strong association with the Phantom laboratory course score (r = 0.80), indicating that performance in this core preclinical course substantially contributes to overall practical competency. Within theoretical coursework, the average theoretical score (Theory_Avg) showed strong correlations with both Restorative Dentistry Theory I (r = 0.77) and Theory II (r = 0.73), confirming the internal consistency of the theoretical performance metrics. In addition, the Dental Anatomy score exhibited a relatively strong correlation with the average practical score (r = 0.76), suggesting a meaningful relationship between anatomical knowledge and practical skill acquisition. Importantly, no strong negative correlations were detected among the evaluated variables, indicating a coherent and mutually reinforcing structure among theoretical knowledge, practical performance, and academic progression indicators within the dataset.

Fig. 3.

Pearson correlation heatmap illustrating relationships among variables

Evaluation of binary-class prediction of final academic status categories

The predictive performance of the developed machine learning models was assessed for binary classification of students’ academic outcomes (Pass vs. Fail) using multiple evaluation metrics, including accuracy, precision, recall (sensitivity), specificity, F1‑score, area under the AUC‑ROC curve, and MCC. These complementary metrics were selected to provide a comprehensive evaluation of model discrimination ability and classification balance. The comparative performance of the models is summarized in Fig. 4; Table 3.

Fig. 4.

AUC-ROC and confusion matrix of best ML Model (ANN) for Prediction of Academic Status of Dentistry Students

Table 3.

Comparison of ML models’ performance in the prediction of the academic status of dentistry students

ML Model	Accuracy	F1-score	precision	Recall	Specificity	Sensitivity	AUC-ROC	MCC
logistic regression	0/821	0/848	0/933	0/778	0/9	0/778	0/839	0/651
Random Forest	0/857	0/882	0/938	0/833	0/9	0/833	0/864	0/710
XGBoost	0/821	0/857	0/882	0/833	0/9	0/833	0/839	0/621
CatBoost	0/821	0/848	0/933	0/778	0/9	0/778	0/883	0/625
ANN	0/857	0/882	0/938	0/833	0/9	0/833	0/906	0/710

Overall, all models demonstrated good predictive performance, with accuracies ranging from 0.821 to 0.857 and AUC‑ROC values between 0.839 and 0.906. Among the evaluated algorithms, the ANN and Random Forest models achieved the highest overall performance, each with an accuracy of 0.857, a precision of 0.938, a recall of 0.833, and an F1‑score of 0.882. Notably, the ANN model produced the highest discriminative ability, with an AUC‑ROC of 0.906 and an MCC of 0.710, indicating strong agreement between predicted and actual outcomes. The Logistic Regression model demonstrated robust baseline performance, achieving an accuracy of 0.821, an AUC‑ROC of 0.839, and an MCC of 0.651, confirming that even linear models could capture meaningful predictive patterns in the dataset. Similarly, XGBoost and CatBoost models showed comparable performance, with accuracies of 0.821 and 0.883, AUC‑ROC values of 0.839 and 0.883, and MCC values of 0.621 and 0.625, respectively. Across all models, specificity remained consistently high (0.90), indicating a strong ability to correctly identify students who successfully passed the course, while recall values of 0.778–0.833 demonstrate a reliable capacity to detect students at risk of failing. Collectively, these findings indicate that the implemented ML models (particularly ANN and Random Forest) provide accurate and balanced predictions of dental students’ academic outcomes, supporting the feasibility of the Den‑SOFA predictive framework for early identification of students requiring academic support.

Evaluation of multi-class prediction of final grade categories

The performance of the machine learning models was further evaluated for the multi‑class prediction of students’ final grade categories (A–F). Model performance was assessed using accuracy, precision, recall, F1‑score, MCC, and AUC‑ROC to provide a comprehensive evaluation in a multi‑class classification setting. The best model results are summarized in Fig. 5; Table 4. Overall, the models demonstrated moderate predictive performance in distinguishing among the five grade categories, reflecting the increased complexity of multi‑class prediction compared with binary outcome classification. Among the evaluated algorithms, the ANN achieved the best overall performance, with an accuracy of 0.464, precision of 0.458, recall of 0.464, F1‑score of 0.458, and the highest MCC value (0.330). The ANN also achieved an AUC‑ROC of 0.758, indicating a moderate level of discrimination among grade categories. The Logistic Regression model demonstrated comparable performance (accuracy = 0.464; F1‑score = 0.452; MCC = 0.328; AUC‑ROC = 0.697). In contrast, Random Forest and XGBoost showed lower predictive performance, with accuracies of 0.393 and 0.357, respectively. Their MCC values (0.242 and 0.197) indicate weaker agreement between predicted and observed grade categories. Across all models, specificity remained relatively high (0.839–0.865), whereas recall values ranged from 0.357 to 0.464, reflecting the inherent difficulty of accurately differentiating among multiple grade categories in a limited dataset. These findings indicate that multiclass prediction of detailed grade categories is substantially more challenging than binary pass/fail prediction. Therefore, the multiclass results should be interpreted as exploratory, and further methodological refinement and validation on larger datasets will be necessary to determine whether reliable prediction of detailed academic grade categories is achievable.

Fig. 5.

AUC-ROC and confusion matrix of best ML Model (ANN) for prediction of academic grades of dentistry students

Table 4.

Comparison of ML models’ performance in the prediction of final grades of dentistry students

ML model	Accuracy	F1-score	precision	Recall	Specificity	Sensitivity	AUC-ROC	MCC
Logistic Regression	0/464	0/452	0/445	0/464	0/865	0/4624	0/697	0/328
Random Forest	0/393	0/394	0/400	0/393	0/848	0/4005	0/775	0/242
XGBoost	0/357	0/359	0/364	0/357	0/839	0/3605	0/702	0/197
ANN	0/464	0/458	0/458	0/464	0/865	0/4790	0/758	0/330

Explainable artificial intelligence analysis

Explainable artificial intelligence analysis was conducted using SHAP to quantify how each predictor variable contributed to the model’s predictions at both the global (population) and local (individual) levels. The results were visualized using a SHAP summary plot and individual waterfall plots. These analyses were performed to enhance post-hoc interpretability of the predictive model rather than to demonstrate validated educational actionability.

Global explanation of population prediction

In the summary plot, each dot represents a student, and its horizontal position indicates the magnitude and direction of the variable’s influence on the prediction. Because the outcome was defined as Fail = 1 and Pass = 0, points on the right side push the prediction toward failure, whereas those on the left push it toward passing. Dot color represents the variable value (red = high, blue = low, purple = intermediate), enabling simultaneous interpretation of feature value and effect on the prediction. Among all predictors, academic term showed the widest spread of SHAP values, indicating the strongest influence on the model’s predictions:

• Lower academic terms were generally associated with a higher predicted risk of failure, whereas higher terms contributed more to passing predictions.

• GPA category and entrance score category demonstrated clear and intuitive patterns, with higher values tending to shift predictions toward passing and lower values toward failure.

• Age and birth order showed mixed distributions of SHAP values across both sides of the axis. This pattern suggests that although younger students and those with lower birth order may show a slightly higher proportion of failure, these variables alone have limited predictive influence once academic factors are considered.

• For Phantom laboratory performance and Theory I, the SHAP summary plot displayed both low-value (blue) and high-value (red) points across positive and negative SHAP ranges, reflecting interactions with other model features. However, a general trend was observed in which higher scores (≥ 16) were more frequently associated with passing predictions, whereas lower scores (< 14) were more often associated with failure predictions.

Overall, the SHAP analysis indicates that the model primarily relies on academic progression indicators and prior academic performance (academic term, passed units, GPA, entrance score, and course-specific grades) to predict student outcomes within the Den-SOFA system. These findings may help identify candidate variables for further investigation, but they do not establish that SHAP-guided feedback improves educational outcomes or intervention effectiveness.

Local explanation of an individual prediction

To demonstrate how the model explains predictions for an individual student, a local SHAP waterfall plot was generated for a representative student who failed (Fig. 6B). This example illustrates how the model combines multiple student characteristics to explain an individual prediction in a transparent and clinically understandable manner. In this example, the model predicted f(x) = − 0.95, compared with the model’s average expected output, E[f(X)] = 0.814, indicating that the combination of this student’s characteristics strongly shifted the prediction toward failure. The factors that contributed most to this prediction were mainly academic variables. The strongest negative influences were academic term (− 1.04) and passed units category (− 0.74), followed by GPA category (− 0.18), Phantom category (− 0.17), practical average category (− 0.07), and Theory II category (− 0.06), all of which pushed the prediction toward failure. In contrast, some factors, such as region quota (+ 0.19), entrance score category (+ 0.15), birth order category (+ 0.06), scholarship status (+ 0.04), and birth order (+ 0.03), slightly pushed the prediction toward passing. However, their effects were relatively small and insufficient to offset the stronger academic risk factors. This local explanation improves transparency by showing which features most influenced the prediction for a specific student. Nevertheless, it should not be interpreted as evidence that changing these variables would alter the outcome, nor as proof that SHAP-based explanations improve educational decision-making, remediation planning, or student support effectiveness. The present analysis demonstrates model interpretability rather than validated educational utility.

Fig. 6.

Global and local explainable AI Plots

Discussion

This study suggests that explainable machine learning approaches may have potential for predicting dental students’ academic performance in restorative dentistry within the studied cohort. The Den-SOFA framework was developed as an interpretable predictive modeling approach intended to support educational analytics and hypothesis generation rather than immediate operational deployment. Given the relatively small sample size, single-institution design, absence of external validation, and use of SMOTE-based resampling, the findings should be interpreted cautiously. Among the evaluated algorithms, the ANN demonstrated the highest internal discriminative performance for pass/fail prediction, followed by the Random Forest model within the applied validation framework. However, these results should not be interpreted as definitive evidence of the superiority of nonlinear models over linear approaches. Several performance metrics were relatively close across models, particularly for multiclass grade-category prediction. Therefore, the observed differences should be regarded as dataset-specific internal performance patterns rather than robust evidence of algorithmic superiority.

Previous studies have explored ML approaches in health professions education. However, direct performance comparisons with the present study should be made cautiously because of major differences in dataset size, predictor variables, outcome definitions, institutional context, and validation strategies. Therefore, these studies are discussed as contextual background rather than direct benchmarks for evaluating the performance or generalizability of the Den-SOFA framework. For example, Lestari et al. predicted preclinical dental students’ grades using pre-admission characteristics such as age and pre-university CGPA and reported moderate predictive performance (accuracy 57%) [15]. Their findings suggest that static entry characteristics alone may have limited ability to capture the complexity of academic progression. In contrast, the present framework incorporates dynamic academic indicators obtained during the educational process, which may provide additional information for modeling educational outcomes. In a large multi-institutional study (n ≈ 900), Almalki and colleagues examined the predictive validity of pre-admission criteria using regression models and found that high school grades and aptitude scores were associated with later academic GPA [16]. While such findings are informative for admission policies, they are less suited for ongoing academic monitoring during training. Beyond dentistry, similar trends have been reported in medical education research. Mastour et al. developed an explainable machine learning framework to predict performance on medical licensing examinations, reporting high discriminative performance in their dataset [13]. Their SHAP-based analysis identified prior academic achievement as the most influential predictor, which is consistent with the present study, where academic progression indicators such as GPA were among the most influential features.

Predictive performance of ML models

In this study, several nonlinear algorithms demonstrated higher internal predictive performance than the linear model within the applied validation framework for both binary and multiclass prediction tasks. For pass/fail prediction, the ANN achieved the highest internal performance among the evaluated models (AUC = 0.906 ± 0.021; accuracy = 0.864 ± 0.028; MCC = 0.71), while Random Forest also showed relatively stable results. Logistic Regression and XGBoost yielded lower performance in this dataset. However, these findings should be interpreted cautiously. The relatively small sample size (n = 96), single-institution design, and absence of external validation limit the ability to draw firm conclusions regarding the comparative superiority or generalizability of nonlinear approaches. The observed performance differences reflect internal validation results and may be sensitive to sampling variability. Therefore, the present findings should be regarded as exploratory and hypothesis-generating rather than confirmatory. The Den-SOFA framework represents a preliminary proof of concept for structured academic risk modeling. It requires further evaluation in larger, independent, and preferably multi-institutional cohorts before its robustness, transportability, calibration stability, and practical utility can be established.

Determinants of academic success

Academic indicators (not demographics) were the primary predictors of examination results. No significant differences were found for sex or age, indicating limited influence once academic data are considered. In contrast, cumulative GPA was the most powerful determinant: students with a GPA ≥ 17 had an 82.4% probability of passing, whereas a GPA < 16 corresponded to a probability of failure > 50% (p < 0.001). Phantom laboratory performance (p = 0.010) and Restorative Dentistry Theory I scores (p = 0.007) were also important differentiators. These findings align with extensive evidence that continuous academic performance and mastery of practical skills outweigh sociodemographic traits in predicting educational success.

Interpretability through explainable AI

A distinctive feature of Den‑SOFA is the integration of explainable AI (SHAP) for both global and case‑specific interpretation. SHAP analysis highlighted influential features such as GPA, academic term progression, total passed units, and entrance scores, whose values were associated with model predictions. Students predicted to fail often exhibited lower GPA values and slower academic progression. Visualizing these patterns can help clarify how the model generates predictions and may support exploratory understanding of potential academic risk patterns. However, SHAP values should be interpreted as model-based explanations rather than causal effects. They describe the contribution of each feature to a model prediction under the structure of the fitted model and the observed data distribution. Therefore, SHAP findings should not be interpreted as evidence that a given predictor directly causes academic success or failure.

Limitations

The primary limitations of this study are the relatively small sample size and the single-institution design. The predictive pipeline was developed and internally validated using data from only 96 students from one institution. Given the use of several flexible, high-capacity ML algorithms, including ANN, Random Forest, XGBoost, and CatBoost, this sample size limits the robustness and generalizability of the models. Although cross-validation was applied, internal validation alone cannot exclude overfitting or optimistic performance estimation. Another important limitation is the absence of external validation and independent testing. All models were trained, optimized, and evaluated using the same institutional dataset. Consequently, the reported performance metrics may partly reflect institution-specific factors such as grading policies, curriculum structure, and assessment formats rather than broadly generalizable predictive patterns. In addition, only structured academic variables were included in the models. Potentially relevant behavioral, psychological, socioeconomic, or learning-process factors were not available and may provide additional explanatory value in future studies. Finally, although SHAP analysis improved model interpretability, the study did not evaluate whether SHAP-informed feedback improves remediation planning, academic counseling, or student outcomes. Therefore, the results should be interpreted cautiously and considered preliminary and hypothesis-generating. In addition, the practical implementation of the proposed framework in real educational settings remains theoretical at this stage and has not yet been prospectively evaluated. The Den-SOFA framework should be viewed as an exploratory proof-of-concept, and future studies using larger, multi-institutional datasets with external and temporal validation are necessary to determine its stability, generalizability, and practical applicability.

Conclusion

This study presents the Den-SOFA framework as an exploratory, proof-of-concept application of ML and explainable AI for predicting academic outcomes in restorative dentistry. The findings suggest that structured academic variables (including term progression, cumulative GPA, and prior course achievement) may offer useful predictive information within the studied cohort. However, given that these models were developed and evaluated within a limited single-institution dataset without external or temporal validation, the results must be interpreted as preliminary. The reported performance metrics should not be considered evidence of robust generalizability or readiness for operational educational deployment. Furthermore, while SHAP analysis provides insight into model-identified associations, this retrospective study did not evaluate whether such information improves remediation planning, counseling, or student outcomes in practice. Consequently, Den-SOFA should be viewed as a hypothesis-generating framework rather than a validated decision-support system. Its practical implementation in educational settings remains theoretical at this stage; further research using large-scale, multi-institutional, and prospective datasets is required to establish its stability, fairness, and true educational utility.

Abbreviations

AI

Artificial Intelligence

ML

Machine Learning

XAI

eXplainable Artificial Intelligence

SHAP

SHapley Additive exPlanations

LR

Logistic Regression

ANN

Artificial Neural Network

RF

Random Forest

XGB

eXtreme Gradient Boosting

CatBoost

Categorical Boosting

Spe

Specificity

Acc

Accuracy

F1

F1-measure

AUC-ROC

Area Under the Curve of Receiver Operating Characteristic

SMOTE

Synthetic Minority Over-Sampling Technique

RUS

Random Under-Sampling

MCC

Matthews Correlation Coefficient

PPV

Positive Predictive Value

TP

True Positive

TN

True Negative

FP

False Positive

FN

False Negative

Authors’ contributions

S.O.M.: Conceptualization; Methodology; Data Curation; Data Interpretation; Writing – Original Draft. E.Sh.: Conceptualization; Methodology; Validation; Writing – Original Draft; Project Administration; Supervision. F.J.: Conceptualization; Methodology; Validation; Data Curation; Writing – Original Draft. M.J.: Coding; Validation; Formal Analysis; Data Interpretation; Visualization; Writing – Original Draft. T.D.: Conceptualization; Methodology; Coding; Validation; Formal Analysis; Data Interpretation; Visualization; Writing – Original Draft; Project Administration; Supervision. S.E.: Conceptualization; Methodology; Validation; Writing – Original Draft; Project Administration; Supervision. T.D. accepts full responsibility for the integrity of the work as a whole.

Funding

This research received no external funding.

Data availability

The source code and implementation files are publicly available at: https://github.com/smartdxgen/Den-SOFA.

Declarations

Ethics approval and consent to participate

The study protocol was reviewed and approved by the Research Ethics Committee of AJA University of Medical Sciences, Tehran, Iran (Ethics Code: IR.AJAUMS.REC.1403.230). All procedures performed in this study were conducted in accordance with the ethical standards of the institutional research committee and with the 1964 Declaration of Helsinki and its later amendments.

Consent for publication

All patient data included in this manuscript have been fully anonymized. No identifying images or clinical details are presented.

Competing interests

The authors declare no competing interests.