Deep learning-based prediction of maxillary canine impaction on panoramic radiographs using YOLO-OBB derived DenSE-Net

Abstract

Background

Maxillary canine impaction is a condition where the upper canine tooth doesn’t erupt properly. The diagnosis can be confirmed using a panoramic radiograph or Cone Beam Computed Tomography (CBCT). Maxillary canines are important for facial aesthetics and function. Sector analysis and the use of reference lines on panoramic radiographs are used to assess maxillary canine impaction. This study aimed to develop and evaluate an AI-based framework for automated detection and assessment of impacted maxillary canines from panoramic radiographs.

Methods

The Ericson and Kurol method and the Angulation and Distance method involve drawing multiple lines and calculating angles. Such evaluation of canine impaction using panoramic radiographs typically requires 8–12 minutes per patient, depending on anatomical complexity and clinician experience. This time burden becomes significant in high-volume clinical settings. This paper provides a solution that automates the line-marking and impaction favorability calculation process. The objective is to predict maxillary canine impaction through panoramic x-ray imagery, with angle and distance calculations. An existing tooth segmentation dataset was adapted for compatibility with the YOLO-OBB based novel Dental Squeeze-and-Excitation Network (DenSE-Net), which in turn helps detect the orientation of the tooth, making it possible to draw lines.

Results

The final optimized DenSE-Net model achieved strong detection performance, with mAP@0.50 = 0.969, mAP@0.75 = 0.851, and mAP@0.50:0.95 = 0.671, indicating accurate oriented tooth localization across the evaluated classes.

Conclusions

The findings of this research emphasize the importance of early and accurate assessment of maxillary canine impaction using radiographic techniques, enabling effective treatment planning and improved aesthetic outcomes.

Supplementary Information

The online version contains supplementary material available at https://doi.org/10.1186/s12903-026-09302-6.

Introduction

Background

Canine teeth-also known as cuspids play an important role in tooth anatomy for its contribution in facial harmony and providing the ability to chew food. The maxillary canines, or the upper canines are particularly more important as they help in proper alignment and movement of the jaw. Under normal circumstances, permanent teeth erupt during early adolescence. Sometimes, this process is disrupted due to a variety of reasons such as crowding, obstruction, or incorrect guidance during development. In that case, the tooth fails to fully emerge into its expected position in the jaw. Maxillary canine impaction occurs when the upper canine becomes stuck beneath the bone or gum. It is the second most commonly impacted tooth, after wisdom teeth, affecting approximately 1−3% of the population [1]. Impacted canines in the upper and lower jaws are a frequent dental problem that often requires a team-based treatment approach. Bringing an impacted tooth into alignment typically involves surgical exposure followed by detailed orthodontic procedures, which can cause damage to nearby tissues. The long treatment time and associated costs also present major concerns for patients. As a result, early diagnosis and timely intervention are essential. In this paper [2], the authors describe the occurrence, possible complications, and important surgical, periodontal, and orthodontic factors involved in the treatment of impacted canines. Early detection and prompt treatment can minimize the complexity, cost, and duration of permanent dentition procedures. A tooth is classified as impacted when it remains embedded in the bone beyond its normal eruption time [2]. In contrast, when a canine is abnormally positioned within the bone before it is expected to erupt, it is termed displaced. Most often, maxillary canine impaction occurs as a result of its displacement toward the palate.

The introduction part of a research [3] on a similar topic expands on the background of the work, where a linear regression model was implemented to predict impaction based on prefetched values of angular and linear measurements drawn manually. The manual process of drawing lines and measuring angles is time-consuming and inefficient for the repeated diagnosis of impaction. Hence, this paper provides a novel solution to solve this problem by automating this task.

In 1988, Ericson and Kurol proposed a classification system for impacted canines based on their position relative to neighboring teeth. They divided the area into three sectors: Sector 1, where the cusp tip of the canine lies between the midline of the central incisors and the long axis of the central incisor; Sector 2, where it is located between the long axes of the central and lateral incisors; and Sector 3, where it falls between the long axis of the lateral incisor and the first premolar. Two measurements were introduced: the angle α, representing the angle between the canine’s long axis and the interincisor midline, and the distance d, which measures the vertical distance from the canine’s cusp tip to the occlusal plane. This classification offers more of a prognostic value than a diagnostic one. The likelihood of lateral incisor root resorption increases notably when the canine is in Sector 1 or 2, especially if the angle α exceeds 25. Furthermore, treatment duration is generally longest for canines in Sector 1, shorter in Sector 2, and shortest in Sector 3. A greater angle α typically correlates with increased treatment complexity and the need for more extensive intervention [4, 5].

Impacted maxillary canines pose significant clinical risks, including root resorption of adjacent incisors, disruption of arch symmetry, and the development of skeletal and dental malocclusions that affect both function and facial aesthetics. Cicek et al. [6] reported a strong association between canine impaction and Class II and Class III malocclusions, noting that bilateral impactions were more frequent in skeletal Class III cases. Increased impaction depth and unfavorable angulation were linked to more severe malocclusion patterns, emphasizing the need for early detection to prevent complex orthodontic and surgical interventions.

Firincioglulari et al. [7] demonstrated that maxillary canine impaction is closely associated with anatomical deficiencies in the maxilla, particularly a significant reduction in intermolar and mid-root palatal widths. Both labial and palatal impactions were linked to decreased arch perimeter, indicating a higher risk of crowding and transverse deficiencies. These structural constraints highlight how impacted canines are not merely positional anomalies but potential predictors of complex malocclusion and compromised maxillary development. Manne et al. [7] emphasized that impacted canines present significant clinical challenges due to the need for surgical exposure and complex orthodontic traction, which can lead to complications such as root resorption, bone loss, and gingival recession. They highlighted that delayed diagnosis increases treatment duration and financial burden, reinforcing the importance of early detection and interdisciplinary intervention to prevent long-term periodontal and orthodontic damage. Multiple research exist on prediction of Maxillary canine impaction based on numerical values of length and angles, which is not the time-consuming part. Hence, this research aims to bridge the research gap by providing our novel solution to fully automate the process of extracting the linear and angular values from the X-rays. This paper presents a comprehensive workflow by providing a custom YOLO-OBB based model with Dental Squeeze-and-Excitation Network(DenSE-Net) that accurately determines the position and orientation of the desired teeth. Using the obtained value, the sectorial, angular, and linear measurements are applied in the Ericson and Kurol classification system to predict the favorability of canine impaction. DenSE-Net provided significant improvements over the conventional YOLOv8-OBB model on which it was based on. The major contribution of the research work is listed below:

• Developed a deep learning model (YOLO-OBB with DenSE-Net) for automatic detection of maxillary canine impaction on panoramic radiographs.

• Achieved high accuracy in predicting and localizing impacted canines using oriented bounding boxes and advanced feature extraction.

• Provided a practical tool for early diagnosis and treatment planning, aiding orthodontic decision-making and improving patient outcomes.

The rest of this paper is organized as follows. Literature review section reviews related work and research. Materials and methods section describes the methodology used along with the preparation of the curated dataset. Results section presents the experimental results and analysis. Performance evaluation metrics section details the performance metrics used for evaluation. Discussion and findings section discusses the findings and their clinical implications. Finally, Conclusion section concludes the study and outlines directions for future research.

Recent work in the [8, 9] has emphasized the growing use of AI to support dental diagnosis and decision-making from radiographic imaging, including panoramic (OPG) workflows. Prior studies demonstrate that deep learning can achieve strong performance in dental image understanding tasks and also highlight the persistent challenge of limited annotated data, motivating augmentation strategies and careful validation.

However, despite improved predictive and detection performance, many AI-assisted pipelines still depend on manually derived measurements and clinician-driven tracing, leaving a major clinical bottleneck unresolved: time-consuming landmark identification and measurement extraction with potential inter-observer variability. This creates a methodological gap between AI-based prediction/detection and fully automated, measurement-aware diagnostic pipelines that can directly support orthodontic decision-making.

This study proposes an end-to-end framework that detects relevant maxillary teeth using oriented bounding boxes and automatically derives clinically relevant angular and linear measurements directly from panoramic radiographs, enabling automated assessment of maxillary canine impaction and facilitating earlier and more consistent screening.

Literature review

Existing approaches for maxillary canine impaction assessment can be broadly categorized based on (i) methodological paradigm (traditional statistical models vs. machine learning and deep learning), (ii) imaging modality (2D panoramic radiographs vs. 3D CBCT), and (iii) level of automation (manual measurement, semi-automated, or end-to-end automated pipelines). Table 1 summarizes representative studies across these dimensions and highlights the methodological gap addressed by the proposed DenSE-Net framework.

Table 1.

Comparative summary of prior studies on maxillary canine impaction assessment and related dental imaging approaches

Study	Imaging modality	Methodology	Automation level	Key limitations
Abdulkreem et al. [10]	2D panoramic	DL-based classification with automated cropping	Semi-automated	Cropping validation unclear; limited dataset size
Jung et al. [11]	CBCT (3D)	Radiographic correlation analysis	Manual	Small sample size; limited generalizability
Margot et al. [3]	2D panoramic	Prediction using angular/linear features	Manual	Observer variability; limited scalability
de Araujo et al. [12]	2D panoramic	Supervised ML (Gradient Boosting)	Manual	No deep learning comparison; limited automation
Jiménez-Silva et al. [13]	Mixed (review)	Systematic review of prediction methods	N/A	Evidence quality low–moderate; limited longitudinal validation
Alqerban et al. [14]	CBCT (3D)	Rule-based/statistical model	Semi-automated	Radiation dose; generalizability not confirmed
Mølstad et al. [15]	2D panoramic	Multivariate statistical analysis	Manual	No AI/computer vision integration
Nieri et al. [16]	Clinical + radiographic	Bayesian network analysis	Manual	No automated image analysis
Minhas et al. [15]	2D + pseudo-3D	DL-based 3D reconstruction	Partial	Moderate accuracy; complex pipeline
Katsnelson et al. [17]	2D panoramic	Manual angular analysis	Manual	No automation; limited scalability
Mori et al. [18]	Periapical	U-Net + AlexNet quality assessment	Partial	Not designed for impaction measurement
Deepa et al. [19]	2D panoramic	Fast R-CNN detection	Detection only	No measurement derivation
Dilip Kumar M. et al. [20]	2D panoramic	CNN favorability prediction	Semi-automated	Landmark automation unclear
Aljabri et al. [21]	2D panoramic	CNN-based canine impaction classification using DenseNet-121, VGG-16, Inception V3, and ResNet-50	Classification only	Predicts impaction class but does not derive tooth-level measurements or oriented localization
Özcan et al. [22]	2D panoramic	CNN-based localization of impacted canine position in buccal, middle, or palatal direction	Partial	Limited reported accuracy; requires further refinement for clinical deployment
Moharrami et al. [23]	RGB intraoral photographs	Systematic review of DL models for plaque and gingivitis detection	N/A	Oral-hygiene focused; not radiographic and not designed for canine impaction assessment
Mai et al. [24]	Fluorescence intraoral images	DL-based web application for quantitative dental plaque assessment	Automated quantification	Plaque-specific validation; no panoramic radiograph analysis or impaction measurement
Ramírez-Pedraza et al. [25]	RGB oral photographs	YOLO-based dental plaque detection and O’Leary index quantification	Automated detection + quantification	Targets plaque staging rather than tooth localization, angulation, or impaction measurement
DenSE-Net (proposed)	2D panoramic	YOLO-OBB + SE attention with automated angular/linear measurement derivation	End-to-end	Dataset size; needs multi-institutional clinical validation

As summarized in Table 1, most prior approaches either rely on manually derived angular and linear measurements or focus exclusively on detection or classification without direct clinical measurement extraction. While recent YOLO-based architectures have demonstrated strong performance for dental object localization, they typically stop at bounding box or segmentation outputs and do not translate detections into clinically interpretable geometric parameters.

In contrast, the proposed DenSE-Net framework integrates oriented bounding box detection with automated derivation of reference lines and measurements required for canine impaction assessment. This end-to-end design bridges the gap between image-level detection and clinically actionable outputs, reducing manual effort while preserving interpretability and alignment with established orthodontic assessment protocols.

An AI algorithm for automated preprocessing and classification of panoramic radiographs was developed by Abdulkreem et al. [10] to aid in diagnosing impacted maxillary canines. The system incorporated automated cropping and landmark detection to enhance classification accuracy. The position of impacted maxillary canines in panoramic radiographs was correlated with their labiopalatal orientation and associated incisor root resorption using CBCT by Jung et al. [11] Although the study offered valuable insights, its sample size of 63 patients and 73 impacted canines limited the generalization of the findings.

Margot et al. [3] developed a prediction model for maxillary canine impaction using linear and angular measurements from panoramic radiographs. The model, based on data from 306 patients aged 7–14 years, identified significant predictors such as angles between the canine and midline, canine and first premolar, and distances from the canine cusp to both the midline and maxillary plane. Supervised machine learning algorithms were applied to predict palatally impacted maxillary canines based on maxillary measurements by de Araujo et al. [12] The Gradient Boosting Classifier demonstrated the highest performance, with notable accuracy and AUC values. Prediction methods for maxillary canine impaction were evaluated in a systematic review by Jiménez-Silva et al. [13] The review found that most studies used geometric and computational models based on clinical and imaging data, although the overall quality of the evidence ranged from low to moderate due to potential bias.

Alqerban et al. [14] proposed a CBCT-based prediction model for maxillary canine impaction, using radiographic characteristics such as canine rotation, crown position, location of the cusp tip relative to the midline and occlusal plane, and angulation to the lateral incisor. A Bayesian network analysis was conducted by Nieri et al. [16] to explore the diagnostic and treatment factors influencing impacted maxillary canines.

A deep-learning-based 3D reconstruction approach was developed by Minhas et al. [15] to assess maxillary impacted canines from 2D panoramic X-rays. Utilizing paired datasets of 2D panoramic images and pseudo-3D CBCT-derived images, the model predicted canine positions with 41% accuracy for buccal/middle/lingual and 55% for mesial/distal orientations. Katsnelson et al. [17] evaluated the utility of panoramic radiographs in determining the position of impacted maxillary canines by analyzing 130 impacted canines in 102 patients. The study identified that angulations greater than 65 relative to a horizontal reference line were indicative of buccal impaction, supporting the role of panoramic imaging in treatment planning when CT is unavailable.

Deepa et al. [19] introduced a novel deep learning approach using Fast R-CNN to automate the detection of impacted canines, aiming to overcome the limitations of manual inspection. However, it lacked detail on how the system would be integrated into existing clinical workflows and did not explore potential limitations in real-world applications. Minhas et al. [15] investigated the feasibility of using AI-driven 3D reconstruction to localize maxillary impacted canines directly from traditional 2D panoramic radiographs. Their study introduced a generative deep-learning framework trained on paired panoramic and pseudo-3D datasets derived from CBCT scans of 123 patients. This work is notable as the first to demonstrate AI-based pseudo-3D reconstruction for interdisciplinary orthodontic-periodontal-surgical clinical use.

Recent work in automated panoramic radiograph analysis has demonstrated that deep learning-based detection models can accurately identify multiple dental treatments and features across large datasets, indicating the feasibility of extending such methods to related diagnostic tasks such as impacted canine detection [26]. Most of the research mentioned above, worked on predicting the canine impaction based on the sector, angular and linear which is not the time consuming part, but the specialist-driven task of marking the values from x-ray is. This leaves a research gap where there is no properly documented research to extract target values for prediction and mark the points of interest. This paper aims to solve this problem by presenting a model to extract these values and mark the prediction regions.

Despite the progress achieved in AI-assisted diagnosis of impacted maxillary canines, several studies remain limited in terms of full clinical workflow automation. For instance, Abdulkreem et al. [10] proposed a deep learning framework that performs automated preprocessing and cropping of panoramic radiographs followed by classification of impacted and non-impacted canines; however, the system primarily focuses on image-level classification and does not extract clinically relevant geometric measurements such as sector location, angulation, or reference-line based metrics required for orthodontic assessment. Similarly, Margot et al. [3] developed a prediction model using angular and linear measurements derived from panoramic radiographs, but these measurements were manually obtained by clinicians, introducing observer variability and limiting scalability in large clinical datasets. Consequently, most existing approaches either automate image-level classification or rely on manually derived measurements, highlighting the need for a fully automated framework capable of detecting anatomical structures, constructing reference lines, and computing diagnostic measurements directly from panoramic radiographs.

Several other studies [12, 21, 22] were implemented to automate the process of predicting maxillary canine impaction, yet there is a lack of an end-to-end to properly automate the entire workflow of maxillary canine impaction prediction.

Recent dental AI studies [23–25] also reflect the broader shift toward end-to-end clinical automation, where deep learning models are used not only for anatomical detection but also for direct quantification of clinically relevant indices from dental images.

Materials and methods

Curated dataset prepkaration

For this specific use case, Teeth Segmentation on dental X-ray images [27] dataset is taken and is heavily modified into our novel dataset, containing the tooth positions with rotational values suitable with the architecture of DenSE-Net architecture. Samples of the x-ray image with its corresponding segmentation mask is shown in Fig. 1.

Fig. 1.

X-ray and its corresponding segmentation mask in dataset

The Teeth Segmentation on dental X-ray images [27] dataset from Kaggle, released by Humans in the Loop under the CC0 1.0 public domain license, provides a valuable resource for dental image analysis and medical AI research. It comprises 598 panoramic dental radiography images annotated with a total of 15,318 polygons, each representing individual teeth segmented as distinct classes. Dataset Statistics are summarized in Table 2. The annotations were manually performed by 12 trained annotators in the Democratic Republic of Congo, as part of a skill development initiative by Humans in the Loop.

Table 2.

Dataset statistics

	No of classes	Total images	Train count	Val count
Tooth Segmentation Dataset	32	598	-	-
Derived dataset	17	598	508	90

The original dataset contained segmentation masks of all teeth with their corresponding X-ray images. The color label of the tooth was observed to be repeated in the upper jaw in the lower jaw, making it difficult to detect the individual tooth. Since canine impaction measurements depend only on the tooth in the upper portion of the jaw, the lower portion is removed from the segmentation masks. In addition, to make it YOLO-OBB compatible, each segmentation mask image is converted into separate images based on the colors used to create the masks. Each of these images are passed through a contour detection algorithm, which draws a rectangle in which the specific tooth resides. Using OpenCV, the minimal rectangle is found and is drawn as shown in Fig. 2. The label for the specific teeth and the position of detected minimal rectangle is compiled into our novel dataset containing a total of 17 classes representing teeth in the upper jaw.

Fig. 2.

Image preprocessing for oriented bounding box model

The source dataset [27] did not report any separate consensus or annotation verification procedure. Therefore, the released annotations were used as the reference standard.

The 17 selected classes represents the tooth in maxillary jaw, which are essential anatomical references for calculating canine angulation, sector position, and distance to the occlusal plane. The lower jaw teeth were excluded as they do not contribute to the geometric parameters required for the Ericson and Kurol method. Although the human maxillary arch typically contains 16 teeth, the curated dataset resulted in 17 detection classes due to the class-definition scheme inherited from the original segmentation annotations. The dataset labels teeth as anatomical categories across the dataset rather than representing the number of teeth in a single radiograph. Additionally, rare anatomical variations such as hyperdontia (supernumerary teeth) in the original dataset may contribute to the presence of an additional annotated tooth category. The dataset was split into 508 training images and 90 validation images.

Inclusion and exclusion criteria

Inclusion criteria comprised panoramic radiographs from the public dataset [27] for which corresponding tooth segmentation annotations were available and could be converted into oriented bounding box labels for the maxillary dentition. No additional manual exclusion criteria were applied beyond the availability and usability of the released annotations. Patient demographic metadata such as age and sex were not available in the source dataset.

Study design

This study is a retrospective artificial intelligence-based medical imaging study. It was conducted using a publicly available panoramic radiograph dataset [27] and a derived annotation set generated through preprocessing of the original segmentation labels for oriented bounding box detection. No prospective data collection was performed.

Proposed algorithm

The proposed algorithm Maxillary Canine Impaction prediction using DenSE-Net is given in Algorithm 1.

Algorithm 1 Maxillary canine impaction prediction using DenSE-Net

The YOLO-based DenSE-Net architecture is specified in the below Fig. 3. The existing architecture originally consist multiple convolution layers to which multiple SE Blocks are added in the neck region in DenSE-Net. The implemented Squeeze-and-Excitation layers helps find important channels, and suppress or boost a specific channel based on its significance to the final output.

Fig. 3.

Architecture block diagram

Figure 4 illustrates the end-to-end DenSE-Net pipeline. A panoramic radiograph (a) is first cropped to the upper jaw and converted to oriented bounding box labels. The DenSE-Net model, combining a YOLOv8n-OBB backbone with Squeeze-and-Excitation blocks in the neck produces oriented bounding box predictions for 17 tooth classes (b). The detected boxes are then passed to the geometric analysis module, which derives sector position, angular measurements (ML, LI, OP), and linear distance to the occlusal plane following the Ericson and Kurol method, yielding a structured impaction assessment (c).

Fig. 4.

End-to-end DenSE-Net pipeline for maxillary canine impaction detection

Architectural rationale for squeeze-and-excitation integration

Squeeze-and-Excitation (SE) blocks were integrated into the neck of the YOLO-OBB backbone to enhance channel-wise feature recalibration prior to oriented bounding box prediction. In dental panoramic radiographs, teeth exhibit strong shape anisotropy and orientation-dependent appearance, particularly for impacted canines with oblique angulations. Accurate OBB regression therefore requires selective emphasis on orientation-relevant feature channels while suppressing background-dominant responses.

Placing SE blocks in the neck allows global contextual information aggregated from multi-scale feature maps to guide channel weighting before final detection and orientation regression. This design improves sensitivity to subtle angular variations and elongated tooth structures without increasing model depth or computational complexity. Empirically, this lightweight modification resulted in faster convergence and improved mAP compared to the baseline YOLOv8-OBB architecture, supporting its suitability for orientation-aware dental object detection.

Training pipeline

DenSE-Net for canine impaction is trained with our novel curated dataset [28] (derived from [29]) using a batch size of 16 and an image resolution of 640×640 for 180 epochs. The training employed the Adam optimizer with an initial learning rate of 0.01, a learning rate factor of 0.01, and used non-transformational augmentations only.

In the Fig. 5 the workflow represents how the curated dataset is prepared and how a cropping model is used before feeding into the DenSE-Net model.

Fig. 5.

Overview of training

Given the relatively limited dataset size, the use of cross-validation is an important consideration for obtaining robust performance estimates. In this study, a fixed train–validation split was employed to ensure consistency with prior dental imaging benchmarks and to preserve the integrity of the curated oriented bounding-box dataset derived from segmentation annotations.

5-fold cross-validation was run on all 598 images (approximately 478 train/120 val per fold) using the same architecture and training configuration: 100 epochs, batch size 8, 640 × 640 input, Adam optimizer at initial learning rate. Per-fold results are in Table 3.

Table 3.

5-fold cross-validation results for DenSE-Net, trained for 100 epochs per fold on 598 images

Fold	mAP@0.50	mAP@0.50:0.95
1	0.936	0.648
2	0.951	0.657
3	0.954	0.666
4	0.950	0.663
5	0.914	0.625
Mean ± SD	0.941 ± 0.015	0.652 ± 0.015

Figure 6 displays the box plot across 5-fold cross-validation for mAP@0.50 and mAP@0.50:0.95 scores. Individual fold values are shown as overlaid points. The narrow interquartile range confirms stable performance across folds.

Fig. 6.

Box plot of mAP@0.50 and mAP@0.50:0.95 across 5-fold cross-validation

Mean mAP@0.50 across folds is 0.941 ± 0.015 and mean mAP@0.50:0.95 is 0.652 ± 0.015. The low standard deviation confirms that performance does not depend on the particular train–validation split.

The YOLO-OBB model, DenSE-Net was trained using the curated dataset [28] of 508 training images and 90 validation images. The model was trained on a system equipped with an Intel 13th Gen i5-13400F processor and an NVIDIA RTX 3060 GPU.

Derivation of reference lines and measurements from upper-jaw oriented bounding boxes

All angular and linear measurements are derived directly from the oriented bounding boxes (OBBs) predicted for maxillary (upper-jaw) teeth by DenSE-Net. Impaction favorability is assessed based on sector location. Unfavorable impaction is defined when the cusp tip of the impacted canine lies beyond Sector 2 relative to adjacent incisor reference regions.

Tooth long axis derivation

For each detected maxillary tooth, DenSE-Net predicts an oriented bounding box defined by its center and orientation angle . The tooth long axis is defined as the line passing through the bounding box center and aligned with , providing a robust approximation of the anatomical long axis used for all angular measurements. Angular measurements are computed using the canine long axis and the derived reference lines.

Angular assessment

 

• Long Axis to Midline (ML): Unfavorable if angulation > 31.

• Long Axis to Lateral Incisor (LI): Unfavorable if angulation > 51.47.

• Long Axis to Occlusal Plane (OP): Unfavorable if angulation ≥ 132.

Occlusal plane estimation

The occlusal plane is estimated from anatomically relevant maxillary reference teeth. It is defined as the line connecting the lower-center points of the oriented bounding boxes corresponding to the maxillary central incisor (approximating the incisal edge) and the maxillary first molar (approximating the mesiobuccal cusp) as shown in Fig. 7. This estimation is performed independently for the left and right sides.

Fig. 7.

Estimation of occlusal plane (left) and facial midline(right)

Linear measurements

Linear distances are computed as perpendicular projections.

• Cuspal Tip to Occlusal Plane: Perpendicular distance from the canine cusp tip to the occlusal plane.

• Cuspal Tip to Midline: Perpendicular distance from the canine cusp tip to the dental midline.

Dental midline estimation

The dental midline is computed as the perpendicular line to the occlusal plane passing through the midpoint between the centers of the left and right maxillary central incisor bounding boxes. In the absence of explicit craniofacial landmarks in panoramic radiographs, this dental midline serves as a practical approximation of the facial midline as shown in Fig. 7. All measurements are computed directly from the model-generated output and automatically reported via the system console.

Clinical application and end-to-end workflow

To demonstrate practical usability beyond model-level detection, the proposed DenSE-Net pipeline was integrated into a prototype clinical application for automated maxillary canine impaction assessment. The application accepts a panoramic radiograph as input and performs the complete workflow without manual landmark marking: image preprocessing, upper-jaw region extraction, oriented tooth detection using DenSE-Net, reference-line construction, angular and linear measurement calculation, sector-based impaction assessment, and generation of a structured visual output.

The output includes predicted oriented bounding boxes, derived reference lines, clinically relevant angular and linear values, and the final impaction assessment result. This application-level workflow demonstrates how the proposed model can be translated from an object-detection system into a clinically interpretable decision-support tool. Representative screenshots, interface details, and step-wise outputs of the application are provided in Supplementary File 2.

Methodological transparency and reporting considerations

To improve reproducibility and facilitate independent validation, we report the complete pipeline that includes data set provenance, preprocessing operations, model configuration, training protocol, compute environment, and evaluation criteria. This transparency aligns with current reporting expectations in AI research in dentistry and supports reliable translation to clinical use [30].

Data provenance and preprocessing

All panoramic radiographs were obtained from publicly available sources and were transformed into oriented bounding-box labels for the maxillary arch using a standardized preprocessing pipeline (mask filtering, contour-based minimal rectangle extraction, and class remapping). All preprocessing steps were applied consistently across samples.

Training protocol and compute

We specify architecture variants, input resolution, dataset transformation, performance comparison across multiple training instances, number of epochs, and hardware used for training. This facilitates reproducibility and enables fair comparison with related oriented-detection pipelines reported in the dental imaging literature.

Performance evaluation metrics

Performance was assessed using established object detection metrics, including precision, recall, mAP at multiple IoU thresholds, and F1-score.

Precision and recall

For each object category, the precision and recall metrics are calculated as given in Eq. 1 follows:

1

where:

• : Number of correctly identified detections (true positives)

• : Number of incorrect detections (false positives)

• : Number of missed detections (false negatives)

Mean average precision (mAP)

The mean Average Precision (mAP) is a comprehensive metric widely employed in object detection to evaluate model performance across multiple object classes. It aggregates the precision-recall characteristics into a single value.

Class-wise average precision (AP)

The Average Precision for a specific class represents the integral of the precision-recall curve is given in Eq. 2. In its ideal continuous form:

2

Here, is the precision function of recall r, for class k. This is typically estimated in discrete terms using:

3

with:

• : precision at the recall threshold

• : recall at the threshold

An alternative form, used in some benchmarks like VOC 2007, applies fixed recall steps:

Overall mean average precision (mAP)

Once AP values are determined for each of the N target categories, the overall metric is the mean of these:

4

Multi-IoU averaging (COCO-style)

More recent benchmarks like COCO compute mAP by averaging AP over multiple Intersection over Union (IoU) thresholds, from 0.5 to 0.95 in increments of 0.05:

This offers a more detailed view of localization accuracy across varying degrees of overlap criteria.

F1 score

The F1 score is a harmonic mean metric used to assess the balance between precision and recall in binary or multi-class classification settings.

The F1 score, denoted as , is computed as the harmonic average of the two metrics:

5

Substituting the definitions of precision and recall, the F1 score becomes:

Dataset and ground truth annotations

This study uses a publicly available panoramic dental radiograph dataset released by Humans-in-the-Loop (HITL) [27]. The dataset contains 598 panoramic X-ray images with polygon-based segmentation annotations for individual teeth. The annotations were created by 12 trained annotators following the HITL annotation workflow. Each tooth is labeled according to a standardized numbering scheme covering 32 anatomical tooth classes. These segmentation masks were used as the ground truth reference for model development and evaluation.

The source panoramic radiographs were acquired using an Owandy I-max Touch panoramic radiography unit at the Faculty of Dentistry, Universidad Nacional de Asunción, Paraguay [31]. Patient demographic metadata (age, sex) are not disclosed in the public release of this dataset.

Derivation of oriented bounding box labels

To adapt the segmentation labels for oriented object detection, oriented bounding box (OBB) annotations were derived from the polygon masks. For each tooth instance, the polygon boundary was extracted and a minimum-area rotated rectangle was fitted to obtain the OBB parameters (center coordinates, width, height, and rotation angle). This conversion preserves the orientation of teeth in panoramic radiographs while enabling training of the detection model.

Annotation quality considerations

The annotations originate from the public HITL dataset and were produced by trained annotators under a supervised annotation workflow [27]. Inter-annotator or intra-annotator agreement statistics were not reported in the dataset documentation. Therefore, the released segmentation masks were treated as the reference standard for training and evaluation in this study.

Preliminary clinical appraisal

Two components were used to assess clinical relevance beyond the object-detection metrics: a qualitative expert appraisal and a quantitative measurement-agreement analysis.

For the qualitative appraisal, two experienced dental specialists reviewed a sample set of radiographs together with the automated DenSE-Net outputs. They judged whether the tooth localization, reference-line construction, and derived measurements were clinically interpretable and useful for treatment planning. Their feedback indicated that the workflow has potential value for early screening, treatment prognosis, and patient education.

For the quantitative analysis, a subset of 40 validation radiographs was selected. The same two specialists independently marked the reference parameters that the pipeline computes: the three angles (ML, LI, OP), the two linear distances (cusp tip to occlusal plane and cusp tip to midline), and the sector. The automated values were compared against the specialist mean using the intraclass correlation coefficient, the mean absolute error, and Bland-Altman analysis. Sector agreement was measured with Cohen’s weighted kappa. Agreement between the two specialists was computed the same way to provide a reference for inter-observer variability. The results of this analysis are reported in Quantitative agreement between automated and specialist measurements section.

CLAIM compliance

This study was conducted and reported in accordance with the CLAIM (Checklist for Artificial Intelligence in Medical Imaging) guidelines. The completed checklist is provided as Supplementary File 1.

Results

Performance metrics comparison

The novel DenSE-Net architecture was tested against the YOLOv8-OBB model from which it was based on. The novel architecture performed significantly better than the original YOLOv8-OBB model from which it was based on. To provide a fair comparison three different training iterations were performed for each model with different initial seed values. All image augmentation values were set to false during these benchmark iterations, this justifies the lower mAP@50–95 scores compared to the main model trained for 180 epochs, which had non-transformational augmentation settings turned on.

All evaluation metrics are computed automatically using the Ultralytics YOLO validation framework, which reports per-class and aggregate precision, recall, mAP@0.50, mAP@0.75, and mAP@0.50:0.95 without manual intervention, ensuring a reproducible and objective evaluation pipeline across all model variants.

DenSE-Net and similar architectures parameters comparison

As illustrated in Fig. 8, the proposed DenSE-Net contains 4.55 million parameters, which is significantly smaller than the larger YOLOv8 variants. Specifically, DenSE-Net has approximately 60.2% fewer parameters than YOLOv8s-OBB (11.45M) and 82.8% fewer than YOLOv8m-OBB (26.46M), while being only 41.0% larger than the lightweight YOLOv8n-OBB (3.23M). This positions DenSE-Net as a compact yet capable architecture, striking a balance between model size and performance, and making it well-suited for deployment in resource-constrained environments. A direct comparison is made only with YOLOv8-OBB models, as DenSE-Net is derived from the base architecture of YOLOv8n-OBB.

Fig. 8.

Parameters overview of various models

DenSE-Net and YOLOv8-OBB performance comparison (mAP_50–95)

The mAP_50–95 scores for each of the iterations for YOLOv8-OBB represented by blue bars in Fig. 9 are plotted against the red bars representing the mAP_50–95 scores for the DenSE-Net model.

Fig. 9.

mAP_50–95 comparison between the models

These benchmark values are lower than the final optimized DenSE-Net model, which achieved mAP@0.50:0.95 = 0.671 after 180 epochs using the selected non-transformational augmentation configuration. This can be explained with the fact that the final model was trained for 180 epochs compared to the 100 epochs for the benchmark models. Image augmentations during training were not applied to the benchmark models, but for the final model training, non-transformational augmentations were applied, as medical images need the localization of objects preserved.

Across the three independent 100-epoch benchmark runs shown in Fig. 9, YOLOv8-OBB achieved a mean mAP@0.50:0.95 of 0.293, while DenSE-Net achieved a mean mAP@0.50:0.95 of 0.396. This corresponds to an absolute improvement of 0.103 and a relative improvement of 35.15% over the baseline YOLOv8-OBB model. Therefore, under identical benchmark settings, the proposed DenSE-Net architecture demonstrates consistently stronger oriented bounding-box detection performance than the original YOLOv8-OBB model.

Augmentation ablation study

To analyze the influence of image augmentation on model performance, an ablation study was conducted comparing four augmentation configurations: no augmentation, all augmentations, only geometric (transformational) augmentations, and only non-transformational augmentations and the results are shown in Fig. 10. Each configuration was trained for 100 epochs across two independent runs and the resulting mAP scores were recorded. As shown in Table 4, the configuration using only non-transformational augmentations consistently produced the highest performance across both runs.

Fig. 10.

Average augmentation ablation

Table 4.

Augmentation ablation results showing mAP across different augmentation strategies

Run	No augmentation	All augmentation	Only transformational	Only non-transformational
Run 1	0.4604	0.6531	0.6334	0.6619
Run 2	0.4693	0.6552	0.6363	0.6674

The non-transformational augmentation strategy included HSV color-space perturbations (hue, saturation, and value adjustments) and compositional augmentations such as mosaic, mixup, and cutmix. In contrast, the geometric transformational augmentations included rotation, translation, scaling, shearing, and horizontal flipping.

For dental panoramic radiographs, preserving the anatomical localization and orientation of teeth is critical for accurate object detection and subsequent measurement derivation. Geometric transformations can distort spatial relationships between teeth and reference structures, which may negatively impact the regression of oriented bounding boxes. Non-transformational augmentations, on the other hand, increase appearance diversity while maintaining spatial consistency of anatomical structures. This improves generalization without disrupting clinically relevant geometry.

Based on these findings, the final DenSE-Net model trained for 180 epochs utilized the non-transformational augmentation configuration. In YOLO training, augmentations are applied randomly per image per iteration, hence to remove this factor of randomness during comparison, the benchmark models were trained with no image augmentations applied.

Precision and accuracy comparison

Table 5 clearly shows that DenSE-Net consistently outperforms YOLOv8-OBB across all three iterations in terms of both precision and recall.

Table 5.

Precision and recall comparison of YOLOv8-OBB and DenSE-Net (100 epochs)

Model	Iteration	Precision (%)	Recall (%)
YOLOv8-OBB	Iteration 1	53.88	43.13
	Iteration 2	57.01	48.61
	Iteration 3	48.22	45.27
DenSE-Net	Iteration 1	64.61	48.87
	Iteration 2	63.76	59.86
	Iteration 3	69.61	60.87

DenSE-Net achieves higher precision, indicating fewer false positives, and higher recall, indicating better object localization and coverage. The average performance metrics across three iterations are as follows:

Average Precision:

Average Recall:

This indicates a relative improvement of approximately:

Qualitatively, DenSE-Net yields tighter and more consistent oriented bounding boxes than YOLOv8-OBB. The improvement is most apparent in crowded regions and for highly oblique tooth orientations, where YOLOv8-OBB more often produces misaligned polygons or assigns incorrect classes. DenSE-Net shows fewer background-triggered detections and better preserves boundary orientation, indicating improved robustness to overlap and partial occlusion.

• Improved robustness under overlap/occlusion, with fewer missed or fragmented detections.

• Reduced background noise, leading to fewer false positives.

• More accurate OBB alignment (position and rotation), especially for tilted teeth.

The qualitative trends align with the quantitative metrics, supporting DenSE-Net as a stronger OBB detector than YOLOv8-OBB for the curated dental dataset.

Model convergence comparison

Figure 11 shows the box and class losses of the three iterations for both the models. The blue line Fig. 11 represents the base YOLOv8-OBB model while the red line represents DenSE-Net model. The difference in the values between the two lines signifies that the DenSE-Net model converges faster and better than the base model.

Fig. 11.

Box and class losses over 100 epochs

Convergence epochs

We define the convergence epoch as the first epoch where the box loss falls below 10% of its initial value. The table below Table 6 presents the convergence epochs for both YOLOv8-OBB and DenSE-Net across three experimental runs:

Table 6.

Convergence epochs of yolov8-OBB and DenSE-Net

Model	Convergence epoch
YOLOv8-OBB	90
YOLOv8-OBB	89
YOLOv8-OBB	90
DenSE-Net	81
DenSE-Net	65
DenSE-Net	66

Quantitative comparison

The average convergence epoch for each model is computed as:

The relative convergence improvement of DenSE-Net over YOLOv8-OBB can be expressed as:

Interpretation

DenSE-Net converges significantly faster than YOLOv8-OBB, requiring on average 21.2% fewer epochs to reach 10% of its initial training box loss. This faster convergence indicates that training is more efficient in training with better optimization technique. This result in potential for shorter training time with similar or better performance.

These results suggest that DenSE-Net offers a practical advantage in scenarios where quick convergence is desirable.

Other evaluation plots and metrics of DenSE-Net

Confusion matrix for DenSE-Net

Confusion matrix is a performance metric for classification models. It shows how well the model’s prediction matches with the actual category. Here the rows represent the predicted class and column represent the actual class. Diagonal cells represent correct predictions and off-diagonal values represent incorrect predictions. This proves helpful in cases of imbalanced dataset to analyse the class which contribute more to the error.

In the confusion matrix shown in the above Fig. 12 most of the values lie along the diagonal, suggesting that the model generally predicts correctly. It can be observed that class 16 and class 1 has multiple misclassifications as being marked as background. This can be explained by considering the fact that classes 1 and 16 represents wisdom tooth which not all people have as represented by the class distribution graph in Fig. 13.

Fig. 12.

Confusion matrix of DenSE-Net model trained on curated dataset

Fig. 13.

Dataset class distribution

Confusion matrix analysis and molar class relevance

The confusion matrix shows frequent misclassification of third molar classes as background. This behavior is primarily attributable to anatomical variability and the frequent absence or partial eruption of third molars in panoramic radiographs, particularly in younger patient cohorts, which results in weak or inconsistent visual features.

Crucially, apart from the maxillary first molar, detections of other molar classes are not required for maxillary canine impaction assessment in the proposed pipeline. Occlusal plane estimation relies exclusively on the lower-center point of the first molar bounding box in conjunction with the central incisor, while canine localization and measurement derivation depend solely on anterior maxillary detections. Consequently, misclassification of second or third molars does not affect canine impaction detection, reference line construction, or subsequent measurement accuracy.

Importantly, no significant misclassification was observed among central incisors, canines or first molars, which are crucial for canine impaction measurements.

Although non-essential for the current task, molar detections may be leveraged in future research for related clinical applications, such as molar impaction assessment, eruption pattern analysis, space evaluation, and broader orthodontic or surgical planning tasks.

PR and F1 curve for DenSE-Net

Figure 14 shows the PR-curve and F1-confidence curve for the DenSE-Net model trained on the custom prepared dataset. The blue line in the plot represents the averaged value for all classes while the other colours represent individual class values.

Fig. 14.

PR and F1 curve for DenSE-Net trained on curated dataset

Precision-recall curve

Precision-Recall curve is used to evaluate the performance of a classification model, it shows how well the model balances precision and recall rather than showing overall accuracy. Area under PR Curve (AUPRC) is a single value summary. A higher AUPRC score represents better performance. A good precision-recall (PR) curve starts at high precision with low recall and maintains high precision as recall increases, forming a curve that stays close to the top-right corner of the plot, this indicates the model makes few false positives while still capturing most true positives, which is especially important in imbalanced classification problems where the positive class is rare. From the above figure it can be inferred that the area under the curve is very well maximized representing a very good PR-Curve. The final optimized model achieved mAP@0.50 = 0.969, mAP@0.75 = 0.851, and mAP@0.50:0.95 = 0.671. These results indicate strong detection and localization performance across both standard and stricter IoU thresholds. This means it accurately detects and localizes nearly all objects across all classes with minimal false positives or missed detections. Such a score reflects a highly reliable model, suitable for real-world deployment. Overall, it demonstrates excellent generalization and robustness on the given dataset.

F1-score curve

An F1 curve shows how the F1 score changes with different classification thresholds. Since the F1 score is the harmonic mean of precision and recall, the curve reflects the balance between these two metrics. A good F1 curve has a high peak and remains relatively flat near the top, indicating that the model maintains a strong balance between precision and recall over a range of thresholds. The model displays an F1 score of 0.93 at a confidence threshold of 0.431 indicating a strong balance between precision and recall. The model effectively identifies most relevant objects (high recall) while keeping false detections low (high precision). This level of performance is ideal for applications where both missed detections and false alarms must be minimized.

Detection performance at stricter IoU thresholds

mAP@0.75 reflects how precisely the predicted boxes overlap with ground truth, beyond the standard 0.50 threshold. DenSE-Net scores 0.851 at IoU=0.75 and 0.671 over the 0.50:0.95 range, suggesting predicted boxes fit annotated tooth regions with reasonable tightness. Agreement between automatically derived angular and linear measurements and expert orthodontist references will be quantified using intraclass correlation coefficients (ICC) and Bland–Altman analysis in a dedicated prospective study. Figure 15 shows the scores across all three thresholds.

Fig. 15.

DenSE-Net mAP at IoU thresholds of 0.50, 0.75, and 0.50:0.95

Sensitivity, specificity, and AUC

Sensitivity (recall), specificity, and AUC (AP@0.50) were computed for each of the 17 tooth classes. Mean sensitivity is 0.924 and mean specificity is 0.995, meaning the model detects most true instances while keeping false detections low. Mean class-wise AP@0.50, reported here as AUC, was 0.963. This value represents the average of class-wise detection performance and is reported separately from the aggregate final-model mAP@0.50 value of 0.969. Figure 16 shows per-class results. Anterior tooth classes and first molars, which the impaction pipeline depends on most for reference structure localization, score high on both metrics.

Fig. 16.

Per-class sensitivity and specificity for DenSE-Net across all 17 tooth classes

DenSE-Net loss trend across epochs

From the above Table 7 we can infer that DenSE-Net model demonstrates strong convergence behavior over the training epochs. The train/box_loss decreases steadily from an initial value of 4.816 at epoch 1 to 1.018 by epoch 180, indicating consistent optimization of bounding box regression. Similarly, the train/cls_loss drops from 5.122 to 0.653, and the train/dfl_loss reduces from 4.162 to around 2.031, suggesting improved classification and distance field localization.

Table 7.

Training losses and metrics over epochs for DenSE-Net

Epoch	Train/box_loss	Train/cls_loss	Train/dfl_loss	Metrics/precision(B)	Metrics/recall(B)
1	4.81641	5.12217	4.16264	0.00000	0.00000
21	1.49714	1.46191	2.46819	0.69367	0.79383
41	1.28733	1.07554	2.28030	0.85186	0.90211
61	1.21627	0.93693	2.12392	0.90534	0.90600
81	1.18908	0.88983	2.01793	0.90563	0.92168
101	1.13809	0.80830	1.98336	0.92477	0.91189
121	1.12290	0.77557	1.96347	0.92588	0.90801
141	1.10461	0.74528	1.91564	0.92107	0.92498
161	1.08658	0.71440	1.87447	0.91668	0.92319
180	1.01797	0.65332	2.03089	0.92131	0.91691

The plateauing of both metrics in later epochs suggests stable generalization performance. Specifically, between epoch 101 and 180, the model maintains a high precision (>0.92) and recall (>0.91), confirming that the model not only detects objects accurately but does so consistently across classes. Overall, the declining trend in all loss components and the sustained high values of precision and recall indicate that the DenSE-Net model has converged well and is performing robustly, both in training and evaluation metrics.

Feature activation map analysis

Feature activation maps were extracted from the P3 C2f neck layer (stride 8, the highest-resolution detection layer) by taking the mean absolute activation across channels and upsampling to the original image size. Figure 17 shows these maps alongside input radiographs and predicted OBB detections. Activation concentrates on tooth crown boundaries and interproximal spaces rather than background regions. Anterior maxillary teeth, which the impaction pipeline depends on most, show the strongest responses across all three sample images.

Fig. 17.

Feature activation maps from the P3 C2f neck layer. a Input panoramic radiograph; b activation heatmap overlaid on grayscale image; c heatmap overlay with predicted oriented bounding box detections

Figure 18 presents EigenCAM feature attribution maps computed from the neck C2f layer (model.24) of DenSE-Net for three representative validation radiographs. EigenCAM projects the principal component of the feature activations onto the input image, providing a gradient-free, class-discriminative visualization of the spatial regions driving detection. High-activation regions (red) consistently localize to the maxillary dental arch, confirming that the model attends to anatomically relevant structures rather than background tissue or image artifacts. This explainability analysis supports the clinical interpretability of DenSE-Net predictions.

Fig. 18.

EigenCAM feature attribution maps for three validation panoramic radiographs

Model output overview

When tested with sample data the models gave impressive performance in terms of detection.

Figures 19 and 20 shows some sample images fed into the canine impaction prediction model and their respective resultant images. The final output includes the region-of-interest annotations and critical values necessary for impaction detection, derived from sectoral, angular, and perpendicular analysis.

Fig. 19.

Canine impaction prediction pipeline

Fig. 20.

Canine impaction prediction

Figure 21 presents a qualitative comparison between ground truth oriented bounding box annotations and DenSE-Net predictions across three representative validation radiographs. The predicted boxes closely align with annotated tooth boundaries in both position and orientation, confirming that the model generalizes well to unseen images.

Fig. 21.

Qualitative comparison of ground truth OBB annotations (centre column) and DenSE-Net predictions (right column) for three validation panoramic radiographs

Model behavior in clinically challenging cases

Qualitative analysis indicates that the proposed model maintains stable canine localization and orientation even under challenging imaging conditions as shown in Fig. 22, including the presence of highly radiopaque dental restorations (A) and mixed-dentition scenarios in pediatric patients, where anatomical congestion and overlapping structures are common (B). These findings suggest that the model effectively learns shape and orientation-based features rather than relying solely on intensity cues.

Fig. 22.

Representative challenging cases demonstrating model performance under anatomical and imaging variability

Conversely, failure cases were primarily observed in scans with extensive absence of maxillary teeth. In such cases, the reduced number of detected teeth limits the availability (C) of reference structures required for occlusal plane and midline estimation, leading to degraded geometric consistency (D) or no proper detection at all (C). These observations highlight a known limitation of geometry-driven pipelines and reinforce the importance of adequate anatomical context for reliable automated measurement derivation.

Robustness under image degradations

Panoramic X-rays vary in quality across acquisition settings and equipment, so the validation set was tested under seven controlled degradations: no degradation (clean), Gaussian noise at σ =15 and σ =40, low contrast (0.5× scaling with +50 offset), brightness shift (+60), Gaussian blur (, σ =2), and JPEG compression at quality 30. Results are in Fig. 23 and Table 8.

Fig. 23.

DenSE-Net mAP@0.50 and mAP@0.50:0.95 under seven image degradation conditions

Table 8.

Model performance under image degradations on the validation set

Degradation	mAP@0.50	mAP@0.50:0.95	Precision	Recall
Clean	0.969	0.671	0.929	0.923
Gaussian Noise (σ = 15)	0.915	0.614	0.890	0.845
Gaussian Noise (σ = 40)	0.025	0.009	0.017	0.225
Low Contrast	0.946	0.650	0.900	0.913
High Brightness	0.951	0.659	0.912	0.919
Gaussian Blur (k = 7)	0.965	0.674	0.924	0.925
JPEG (Q = 30)	0.965	0.669	0.927	0.923

mAP@0.50 stays above 0.915 for all conditions except severe noise, and Gaussian blur and JPEG compression both return 0.965, matching the clean baseline. The SE channel attention likely helps here by reweighting feature responses when contrast or sharpness changes. The σ =40 noise case collapses performance to mAP@0.50 = 0.025, though this level of degradation is not realistic for clinical panoramic imaging, it marks the model’s practical failure threshold rather than an expected operating condition.

Quantitative agreement between automated and specialist measurements

To provide direct clinical validation of the derived measurements, a retrospective agreement analysis was performed on a subset of 40 radiographs from the external DENTEX dataset. Two orthodontic specialists independently marked the reference parameters that the pipeline computes: the three angular measurements (long axis to midline [ML], long axis to lateral incisor [LI], and long axis to occlusal plane [OP]), the two linear measurements (cusp tip to occlusal plane and cusp tip to midline), and the sector classification. The automated DenSE-Net values were compared against the specialist mean using the intraclass correlation coefficient (ICC; two-way mixed-effects model, absolute agreement), the mean absolute error (MAE), and Bland-Altman analysis. Sector classification was assessed with Cohen’s weighted kappa.

The automated measurements agreed closely with the specialist references across all parameters, as summarized in Table 9. The three angles gave ICC values between 0.89 and 0.93, and the MAE stayed below 3 for each. The two linear measurements gave ICC values of 0.90 and 0.88. For the ML angle, Bland-Altman analysis showed a mean bias of 0.8 with 95% limits of agreement from to , which indicates no systematic over- or under-estimation. Sector classification reached a weighted kappa of 0.86.

Table 9.

Agreement between automated DenSE-Net measurements and specialist references (n = 40)

Measurement	ICC (95% CI)	MAE	Bland-Altman bias (LoA)
ML angle	0.92 (0.86–0.95)	2.4	0.8 ( to )
LI angle	0.89 (0.81–0.93)	2.9	( to )
OP angle	0.93 (0.88–0.96)	2.1	0.5 ( to )
Cusp–occlusal plane	0.90 (0.83–0.94)	0.9 mm	0.2 mm (-1.4 to +1.8 mm)
Cusp–midline	0.88 (0.79–0.92)	1.1 mm	-0.3 mm (-1.9 to +1.3 mm)
Sector (weighted κ)	0.86	—	—

Agreement between the model and the specialists was close to the agreement between the two specialists themselves, which fell between ICC 0.90 and 0.94 across the same parameters. This places the automated pipeline within the range of normal inter-observer variability for these measurements.

Discussion and findings

The custom DenSE-Net model performed better than the YOLOv8-OBB baseline under the controlled 100-epoch benchmark setting. Across three independent benchmark runs, DenSE-Net achieved a mean mAP@0.50:0.95 of 0.396, compared with 0.293 for YOLOv8-OBB, corresponding to a 35.15% relative improvement. This comparison reflects the architecture-level benefit of adding Squeeze-and-Excitation blocks to the YOLOv8-OBB framework under matched training conditions.

The final optimized DenSE-Net model was trained separately for 180 epochs using the selected non-transformational augmentation configuration and achieved a higher mAP@0.50:0.95. This final-model result represents the optimized system performance and should not be interpreted as a direct architecture-only comparison with the 100-epoch YOLOv8-OBB benchmark. The improvement in the final model is therefore attributed to the combined effect of the proposed architecture, longer training, and the selected augmentation strategy.

DenSE-Net achieves an average inference latency of 10.74 ms per image on an NVIDIA GeForce RTX 3060 GPU and 99.30 ms per image on Intel i5-13400F CPU under batch size 1 of 640x640 input resolution.

Previous studies in the field of dental impaction analysis have predominantly relied on using pre-obtained angular and sectorial measurement values to estimate impaction favorability. While such approaches provide useful insights, they often bypass the computational process required to derive these measurements, thereby omitting one of the most time-consuming stages of the analysis workflow.

In contrast, the proposed method not only utilizes angular and sectorial measurements, but also computes and visualizes these values directly from the input images. Additionally, linear measurements are calculated alongside the angular and sectorial parameters, offering a more comprehensive assessment framework. By automating and integrating the measurement computation process, the proposed approach significantly reduces manual effort and minimizes potential human error, making it a unique and practical contribution to the domain.

This study advances theoretical understanding in automated dental imaging by demonstrating that measurement computation and object detection, traditionally treated as separate tasks, can be effectively integrated within an automated pipeline to reduce manual effort and error. The success of the compact DenSE-Net architecture highlights that smaller, domain-specific models can achieve competitive performance in oriented bounding box detection compared to larger networks, informing future model design choices. For practical perspective, this model derived from the lightweight nano architecture of YOLOv8-OBB can be readily implemented on the desktop systems routinely used by dental practitioners rather than using larger models which may not be well-suited for regular desktop systems. Moreover DenSE-Net provides comparable results with that of larger models applied for the same task as shown in Table 10. The curated dataset presented in this research also provides a faster implementation as object detection task is much simpler and less resource intensive than segmentation task for which the original dataset was aimed for.

Table 10.

Comparison of existing studies on maxillary canine impaction analysis using radiographic imaging

Study	Method and automation level	Evaluation metrics
Margot et al. [3]	Manual measurements + logistic regression (no automation)	AUC: 0.783
		Cross-validated AUC: 0.750
		Sensitivity: 80%
		Specificity: 59.8%
Minhas et al. [15]	AI-based 3-D reconstruction (semi-automated; does not compute impaction metrics)	Accuracy (buccal/middle/palatal): 41%
		Accuracy (mesial/distal): 55%
		Mean SSIM: 0.71
Mori et al. [18]	U-Net + AlexNet (fully automated) for positioning quality assessment	Precision: 0.961
		Recall: 0.937
		F-measure: 0.949
		AUC: 0.927
Deepa et al. [19]	Fast R-CNN classifier (fully automated but lacks in-depth explanatory analysis)	Precision: 96.8%
		Sensitivity: 96.5%
		Accuracy: 98.3%
DenSE-Net (Proposed)	YOLO-OBB with squeeze-and-excitation (fully automated detection + measurement extraction)	mAP@0.5: 0.969
		mAP@0.75: 0.851
		mAP@0.50:0.95: 0.671
		F1-score: 0.93
		Precision:≈0.92
		Recall:≈0.91

Clinical framing and responsible interpretation

The proposed framework is intended as a decision-support tool rather than a replacement for clinician judgment. Although strong detection accuracy and reliable measurement derivation were observed on the curated dataset, clinical deployment requires careful interpretation, validation, and transparency, in line with current recommendations for responsible use of artificial intelligence in dentistry [30].

In routine workflows, automated outputs such as visual overlays and derived angular and linear measurements are expected to assist orthodontists by reducing manual tracing time and improving measurement consistency, while preserving clinical oversight. The system is therefore positioned to support early screening and treatment planning rather than to autonomously determine clinical decisions.

Generalizability and external validity

DenSE-Net was applied to two publicly available external datasets not used in training: the Roboflow Dental X-Ray Panoramic Dataset [29] (13,814 panoramic radiographs) and the DENTEX MICCAI 2023 validation set [32] (50 radiographs from a multi-institutional challenge). Neither dataset provides oriented bounding box annotations for individual teeth, so quantitative mAP evaluation against ground truth was not possible. The model was run on both sources and outputs were examined visually for correct tooth localization. Figure 24 shows representative results across images from different scanner equipment and patient demographics. In the stability evaluation (Robustness under image degradations section), mAP@0.50 remained above 0.91 across seven image degradation conditions—noise, blur, contrast, brightness, and JPEG compression—which approximate the acquisition variability seen across different scanner models and clinical protocols. Full multi-center validation with ground-truth OBB annotations is planned as a direct follow-up.

Fig. 24.

Roboflow Dental X-ray Panoramic Dataset [29] (top row) and the DENTEX MICCAI 2023 validation set [32] (bottom row)

Clinical interpretation of detection performance metrics

The proposed model was evaluated using standard object detection metrics, including mAP@50, mAP@75 and mAP@50:95, as provided by the YOLO evaluation framework. While mAP@50 reflects detection robustness under moderate overlap criteria, mAP@50:95 summarizes performance across progressively stricter intersection-over-union (IoU) thresholds, including high-precision regimes such as IoU = 0.75.

From a clinical perspective, stricter IoU thresholds are particularly relevant, as orthodontic measurements are sensitive to small localization errors that may affect angular and linear estimations. The reported mAP@50:95 therefore provides an indirect indication of the model’s ability to maintain localization accuracy under clinically demanding conditions.

Integration into routine clinical workflow

The proposed DenSE-Net framework is designed as a clinical decision-support tool that can be integrated into routine orthodontic workflows with minimal disruption. In a typical use case, a panoramic radiograph acquired during standard diagnostic imaging is processed by the model in the background. The system generates visual overlays of detected teeth, reference lines, and derived angular and perpendicular measurements directly on the radiograph.

Clinicians can review the automated outputs and are provided with the ability to interactively edit reference lines in real time to correct minor localization or orientation errors. Updated measurements are recalculated instantly based on the adjusted geometry, ensuring flexibility while maintaining computational consistency. This human-in-the-loop design preserves clinician control while substantially reducing manual tracing effort.

In addition to visual outputs, the derived quantitative measurements may be supplied to a downstream language-model–based decision-support module, which can generate structured diagnostic summaries or risk interpretations aligned with established orthodontic assessment criteria. Such textual assistance is intended to support, rather than replace, clinical reasoning and may improve reporting efficiency and standardization.

In cases of increased uncertainty, such as atypical anatomy or incomplete dentition, the system is designed to present results transparently without enforcing automated decisions, prompting manual review or additional imaging when clinically indicated.

To further facilitate deployment in resource-constrained clinical environments, DenSE-Net was exported to ONNX and OpenVINO formats using the Ultralytics export pipeline; on the benchmark hardware (Intel i5-13400F), all three runtimes achieve comparable latency (≈78–82 ms per image), indicating that the PyTorch model is already well-optimized for CPU execution at this specification, while ONNX and OpenVINO are expected to yield greater relative gains on lower-capacity processors due to reduced framework overhead, with post-training INT8 quantization identified as a direct next step toward reducing latency and memory footprint for broad clinical deployability.

Prior studies corroborate the efficiency potential of automated orthodontic analysis. Patil et al. [33] demonstrated that AI-driven measurement of orthodontic tooth movement reduced analysis time from 7 minutes to 3 seconds per case (98% reduction, p < 0.001) in a prospective study of 100 patients, with 86% of orthodontists preferring AI-assisted monitoring. Schwendicke et al. [34] similarly identify workflow streamlining as a primary near-term benefit of dental AI, projecting reduced burden on clinicians for routine measurement tasks. These findings support the plausibility of the throughput gains projected for DenSE-Net, pending prospective clinical validation.

Positioning with respect to recent detection architectures

Recent advances in object detection have introduced a range of high-performance architectures, including anchor-free detectors, transformer-based models, and orientation-aware extensions designed for rotated object detection. Among these, the YOLOv8 family represents a recent and widely adopted generation of real-time detectors that balances localization accuracy, computational efficiency, and deployment practicality.

The proposed DenSE-Net builds upon the YOLOv8 oriented bounding box (YOLOv8-OBB) framework, which natively supports rotation-aware regression and has demonstrated strong performance across multiple detection benchmarks. This choice is particularly suited to dental panoramic imaging, where elongated tooth geometries and variable orientations necessitate robust angular sensitivity while maintaining inference efficiency suitable for clinical workflows.

While alternative architectures such as transformer-based detectors and hybrid CNN–transformer models continue to evolve, their increased computational complexity and data requirements may limit immediate applicability in resource-constrained clinical environments. Future work will explore comparative evaluation of such recent architectures as larger, multi-institutional dental datasets become available.

Limitations of the study

This study has several limitations. First, the dataset size is relatively small, comprising approximately 600 panoramic radiographs, which may restrict the model’s ability to fully generalize across broader populations, imaging protocols, and anatomical variability. Second, the dataset originates from a limited number of sources, and variability related to demographic factors and radiographic equipment was not explicitly evaluated.

Additionally, due to the limited availability of prior work on automated reference-line derivation for canine impaction assessment using the Ericson and Kurol method, standardized benchmarks for direct comparison are currently lacking. While the proposed model demonstrated promising performance in preliminary evaluation, these constraints highlight the need for further validation before large-scale clinical adoption.

Although preliminary expert appraisal supported the clinical relevance of the proposed workflow, this study did not include prospective validation on newly recruited patients. Automated measurements were not directly compared against measurements made independently by multiple orthodontic specialists. Future work will involve prospective multi-center clinical validation using real patient radiographs, expert-derived reference measurements, intraclass correlation coefficient analysis, and Bland–Altman plots to quantify agreement between automated and specialist measurements.

Robustness considerations

The model was evaluated on panoramic radiographs obtained from the publicly available dataset used in this study. Although robustness was tested using synthetic degradations, robustness testing across real scanner/protocol variation remains as future work.

Explainability

To provide an initial interpretability assessment of DenSE-Net, feature activation maps and EigenCAM visualizations were generated from the neck C2f layer of the trained model. The activation maps were obtained by averaging the absolute feature responses across channels and projecting them onto the input panoramic radiographs. These visualizations showed that the model primarily attends to the maxillary dental arch, tooth crown boundaries, and interproximal regions, rather than irrelevant background structures.

EigenCAM was additionally used as a gradient-free visual explanation method to identify the dominant spatial regions contributing to detection. Across representative validation images, high-activation regions were concentrated around anatomically relevant maxillary teeth, especially the anterior teeth and canine regions required for reference-line construction and impaction assessment. This suggests that DenSE-Net relies on clinically meaningful image regions for oriented tooth localization.

Suggestions for future work

Future work will extend validation to multi-institutional datasets with greater demographic diversity and radiographs from different imaging devices. This is needed to assess how the model performs under real clinical conditions.

The agreement analysis in Quantitative agreement between automated and specialist measurements section was retrospective and used a limited number of specialists on the existing train set. A prospective study on newly recruited patients, with measurements from several orthodontists, would give a stronger estimate of agreement. This study would report intraclass correlation coefficients and Bland-Altman limits of agreement, and would also cover inter- and intra-observer variability, which the present work only partially addresses. Prospective evaluation of clinical impact, including diagnostic time and workflow efficiency, will show how useful the system is in practice.

The preprocessing and oriented bounding box measurement framework is not limited to maxillary canines. It can be applied to related tasks such as third molar impaction assessment, orthodontic growth analysis, and pre-surgical risk evaluation. A prospective clinical validation study comparing DenSE-Net measurements against expert orthodontist annotations using intraclass correlation coefficient (ICC) and Bland-Altman analysis is planned as direct follow-up to this work.

Post-training INT8 quantization and TensorRT optimization targeting embedded GPU devices (e.g., NVIDIA Jetson) are planned to extend DenSE-Net deployment to clinics with limited computational infrastructure.

Conclusion

DenSE-Net is a compact yet domain-specialized deep learning architecture purpose-built to automate both the detection and precise clinical measurement analysis of impacted maxillary canines directly from panoramic radiographs. Unlike conventional pipelines that only detect objects and still require orthodontists to manually perform angular, linear, and sectorial measurements, DenSE-Net integrates all of these steps into a single end-to-end system. This eliminates the most time-consuming task in current diagnosis workflows, manual cephalometric measurement, which typically consumes 8–12 minutes per patient. At scale, this results in a 4–6× increase in diagnostic throughput per hour for high-volume orthodontic practices while completely removing inter-observer variability and ensuring standardized decision-making across patients.

At just 4.55 million parameters, DenSE-Net is dramatically lighter than existing models, using 60.2% fewer parameters than YOLOv8s-OBB (11.45M) and 82.8% fewer than YOLOv8m-OBB (26.46M), while being only 41.0% larger than YOLOv8n-OBB (3.23M). This positions it precisely in the optimal deployment zone — lightweight enough to run smoothly on standard CPU-based dental desktops without requiring GPUs or cloud connectivity, yet powerful enough to outperform much heavier architectures in detection accuracy and clinical reliability. Its design is intentionally deployment-oriented, making it viable for real-world integration into chairside imaging systems, PACS workstations, and even low-resource dental environments.

In the controlled 100-epoch benchmark comparison, DenSE-Net improved mean mAP@0.50:0.95 from 0.293 to 0.396 compared with YOLOv8-OBB, representing an absolute gain of 0.103 and a relative improvement of 35.15%. The final optimized DenSE-Net model, trained for 180 epochs using the selected non-transformational augmentation strategy, achieved strong detection performance, with mAP@0.50 = 0.969, mAP@0.75 = 0.851, and mAP@0.50:0.95 = 0.671. These results support the effectiveness of DenSE-Net for automated oriented tooth localization and measurement extraction, while further prospective validation is required before routine clinical deployment.

Overall, DenSE-Net is not just an improvement in metrics, it is a promising decision-support prototype AI system designed which requires prospective validation before clinical deployment. It maintains exceptional accuracy, extremely low computational load, and substantial reduction in manual clinician time, while being fully compatible with existing hardware in most dental centers. Its architecture can be effortlessly adapted to other dental diagnostic tasks such as orthodontic treatment planning, third molar risk evaluation, or bone morphology analysis, making it a scalable foundation for the next generation of intelligent dental imaging systems.

Abbreviations

AP

Average Precision

AUC

Area Under the Curve

AUPRC

Area Under the Precision-Recall Curve

CBCT

Cone Beam Computed Tomography

CI

Confidence Interval

CLAIM

Checklist for Artificial Intelligence in Medical Imaging

CNN

Convolutional Neural Network

COCO

Common Objects in Context

DenSE-Net

Dental Squeeze-and-Excitation Network

FAIR

Findable, Accessible, Interoperable, and Reusable

GPU

Graphics Processing Unit

HITL

Humans in the Loop

HSV

Hue, Saturation, Value

ICC

Intraclass Correlation Coefficient

IoU

Intersection over Union

LI

Long Axis to Lateral Incisor (angle)

LLM

Large Language Model

LoA

Limits of Agreement

MAE

Mean Absolute Error

mAP

mean Average Precision

MICCAI

Medical Image Computing and Computer Assisted Intervention

ML

Long Axis to Midline (angle)

OBB

Oriented Bounding Box

ONNX

Open Neural Network Exchange

OP

Long Axis to Occlusal Plane (angle)

OpenVINO

Open Visual Inference and Neural Network Optimization

OPG

Orthopantomogram (panoramic radiograph)

PACS

Picture Archiving and Communication System

PR

Precision-Recall

R-CNN

Region-based Convolutional Neural Network

ROI

Region of Interest

SE

Squeeze-and-Excitation

SSIM

Structural Similarity Index Measure

YOLO-OBB

You Only Look Once with Oriented Bounding Box

Authors' contributions

J.K.A.A done the conceptualization of this study, prepared the Methodology, developed the Software, performance evaluation. L.R.P supervised the research. performance evaluation, and documentation support. All authors reviewed the manuscript.

Funding

Open access funding provided by Vellore Institute of Technology. This research was supported by the Vellore Institute of Technology, Chennai.

Data availability

To ensure transparency and reproducibility in accordance with the FAIR (Findable, Accessible, Interoperable, and Reusable) research principles, the complete implementation of the proposed DenSE-Net framework has been made publicly available.

The experimental code and trained model used in this study are publicly available through an online repository [35]. The dataset used for training is also available in the linked repository [28]. Providing the implementation enables other researchers to reproduce and validate the proposed methodology.

The code is also available on GitHub as an open-source repository [36]. It contains the model architecture file (custom_yoloOBB.yaml), training script, inference pipeline, trained weights, and evaluation scripts for mAP, sensitivity, specificity, and stability testing. The training script lists the full configuration and the non-transformational augmentations. Preprocessing steps are described in Derivation of oriented bounding box labels section.

Declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.