Reliability and reproducibility of CBCT assessment of craniomaxillary changes before and after treatment for Class III growing patients – a cautious or critical approach

Abstract

Background

To evaluate reliability and reproducibility of 3-dimensional (3D) assessment of maxillary protraction treatment using voxel-based superimposition of cone-beam computed tomography (CBCT) models of the anterior cranial base in growing patients with skeletal class III malocclusion.

Methods

CBCT scans were performed before and after maxillary protraction treatment for Class III malocclusion. Three observers independently constructed 162(27*2*3) 3D virtual models from CBCT scans, which had been reoriented 3D models before treatment to natural head posture, of 27 patients in software. The anterior cranial base was used to register the 3D models pre- and post- treatment. Three observers independently identified 9 landmarks(Including those in the contralateral side)and recorded in three-dimensional coordinates in the 3D models. each observer performed this three times on the pre- and post-treatment model. The mean value of the 3 sets of coordinates at different times was taken as the coordinates for each landmark. The intraobserver reliability and inter-observer reproducibility of the method for craniomaxillary changes were analyzed.

Results

The reliability for intraobserver assessments, with ICC > 0.90 for 25 (92.6%) of intraobserver assessments. 24 of 27 (88.9%) cases showed that the accuracy of the measurement method were below 0.3 mm. 21 of the 27 cases showed that the inter-bserver reproducibility errors were below 0.3 mm; only 6 cases (22.2%) showed error ≥ 0.3 mm.

Conclusions

The reliability andreproducibility of the method for assessment of maxillary protraction treatment in growing patients with skeletal Class Ⅲmalocclusion were judged to be excellent.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12887-025-06049-x.

Introduction

Skeletal Class III malocclusions are the abnormal relationship between maxilla and mandible due to uneven growth of the two bony structures. About 42% to 63% of patients belong to the maxillary hypoplasia with the mandible being of normal size or slight prognathic type [1]. Maxillary protraction is recommended for such skeletal Class III patients with maxillary deficiency.

Traditionally the cephalograms pre-and post- treatment were compared by superimposing stable structures such as the anterior cranial base, to evaluate the effect of maxillary protraction and maxillofacial changes [2, 3]. However, the simple sagittal assessments using 2-dimensional (2D) lateral cephalogram are prone to errors where accurate determination of some landmarks is concerned, because of failure to the anatomic structures [4] there. Also,as patients with skeletal malocclusion often are associated with facial asymmetry [5], 2D assessment cannot be accurate or reliable. With the advent of imaging technology, cone-beam computed tomography (CBCT) has been preferred as the method of choice for maxillofacial imaging,and as a valuable tool for orthodontic treatment planning and clinical research. CBCT would provide a comprehensive analysis of any three-dimensional (3D) changes in the size, shape and position of the maxillofacial structures [6].In particular, it affords a more accurate assessment of the labiopalatal direction and the contact relationship between adjacent teeth,when compared to 2D radiography [7]. Superimposition of virtual models is necessary for 3D measurements, a process that is much more complex than with 2D images. Identification of landmarks the first step, which is also prone to errors. Despite the difficulty, some researchers have made progress and achieved good reliability [8–11]. For instance, Nada RM [12] proposed the zygomatic arches and the anterior cranial base for obtaining planes to standardize the 3D image orientation in non growing subjects. However, it is essential to identify stable markers in children, and ensuring that these markers are in close proximity to the facial region. Given that the landmarks are situated remotely from the facial region, any inconsistencies are heightened during the mapping of craniomaxillary features into the coordinate framework. Subsequently, the error associated with the 3D superposition measurements may be coamplified, potentially attaining clinical significance.

Studying the growth of the craniomaxillary structure relies on measurements made to stable locations in the skull, such as that anterior cranial base [13]. The cribriform ethmoid plate growth ceased by the age of 2.That is, the pubertal growth spur has much less pronounced effect on the anterior cranial base [14]. With different methodologies (longitudinal cephalometry [15], histology [14]and dry-skull measurements [16]), three groups of researchers reported that the sphenoethmoid synchondrosis growth stand-by age 7. Therefore, the pre-sphenoid region (the plane surface on the sphenoid bone, in front of sella turcica),which is considered stable after 7 years of age [17], were noted as stable in craniomaxillary region of growing individuals. They may then be used as stable reference, as well as treatment changes.

Voxel-based registration of changes with 3D CBCT images has been used in the past few years to evaluate overall facial changes relative to the cranial base [18]. But then, regional craniomaxillary registrations were still controversial. This present study, aimed to evaluate the 3-dimensional craniomaxillary changes in growing patients, who had received orthodontic treatment for Class III malocclusion, and to examine the intraobserver reliability and inter-observer reproducibility of a voxel-based superimposition method.

Materials and methods

Corresponding pairs of pre- and post-treatment CBCT scans of 27 children, aged 8–11 years before treatment, who had received protraction therapy for their Class III malocclusion at the Department of Orthodontics of the University of Hong Kong-Shenzhen Hospital were included in this study. All patients or their parents gave consent to participate in this study, after hearing a thorough explanation of the research objectives and procedures. The two scans were taken on average 24 to 26 months apart. The scan was included in the analysis based on: cervical vertebral maturation (CS1-CS3); clear and legible 3D images; Class III malocclusion The Exclusion criteria were: discernible craniofacial asymmetry; discernible mandibular asymmetry; temporomandibular joint disorders; history of maxillofacial trauma or surgery in the region; some systemic disease; and previous orthodontic treatment. The study protocol was approved by the Medical Ethics Commission of the University of Hong Kong-Shenzhen Hospital.

The CBCT imaging parameters were configured as follows: 120 kVp, 18.54 mA, a field of view measuring 23 × 17 cm, a voxel size of 0.3 mm, and a scanning duration of 8.9 s. The acquired data were subsequently exported in DICOM format and imported into a 3D imaging software application, namely InvivoDental software version 5.1.3 (manufactured by Anatomage Inc., San Jose, USA), where a three-dimensional model of the patient was generated within a 3D coordinate framework. The spatial positioning and orientation of the digital 3D model within this framework were contingent on the patient's head posture during CBCT acquisition. Within the"section"module, the orientation of the 3D virtual model was facilitated by utilizing axial, coronal, and sagittal perspectives on the left side of the screen, while on the right side, the model was positioned: in the lateral view, structures such as the orbits, external auditory canals, and other relevant features were aligned to achieve maximum overlap, and the Frankfurt horizontal plane was orientationally adjusted parallel to the ground Fig. 1.

Fig. 1.

Reorientation of the 3D model. In the lateral view, bilateral structures, such as the orbits, external auditory canals and other structures as much as possible overlap, and the Frankfort horizontal plane was oriented horizontally

The software provided a module entitled"superimposition"to facilitate fully automated voxel-based superimposition of pre- and post-treatment scans, necessitating the initial delineation of a stable anatomical area. This process was initiated by the activation of the"voxel registration"feature, which prompted the appearance of three dialogue boxesoriented in the sagittal, coronal, and axial perspectives,allowing the observer to pinpoint the stable region to direct the superimposition. These boxes facilitated the selection of anatomical structures along the anterior cranial base within the 3D models, as illustrated in Fig. 2. Subsequent to the selection of the"start"option, the soft are automatically initiated the voxel-based superimposition of the 3D images, utilizing a stable reference structure to guide the process. The comprehensive registration and superimposition procedure was approximately 3 min in duration.

Fig. 2.

The superimposition region selected in the CBCT volumes. Axial, sagittal, and coronal slice views of the volumes were used to select the anatomical structures of the anterior cranial base

A comprehensive total of nine landmarks (or six, if the left and right landmarks are considered as a singular entity) were delineated, along with specific criteria for each. Each of the nine landmarks was independently pinpointed by individual observers on every 3D surface rendering as presented in Table 1 and Fig. 3. The landmark identification process involved the observers navigating through various perspectives, including sagittal, coronal, and axial, in order to select the most pertinent section for registration purposes. Subsequently, the three-dimensional coordinates of each landmark were automatically logged by the software suite. In instances where the delineation of certain landmarks proved challenging within specific views, the observers had the capability to activate the surface-rendered model to facilitate the localization process, as illustrated in Fig. 4.

Table 1.

Landmarks selected for the study

Landmark name	Anatomic region	Lateral view	Axial view	Anteroposterior view
1. Nasion (N)	Frontonasal suture	Anterior-most point	Middle-anterior–most point on the anterior contour	Middle point
2. Anterior nasal spine (ANS)	Median, sharp bony process of the maxilla	Point on the tip	Anterior-most point	Middle point in the anteroposterior slice determined by the lateral and axial view
3. A point (A)	Premaxilla	Posterior-most point on the curve of the maxilla between the anterior nasal spine and supradentale	Middle-anterior–most point on the tip of the premaxilla	Middle point in the anteroposterior slice determined by the lateral and axial views
4. Right upper incisal alveolar ridge(rUIAR)	Premaxilla alveolar	Anterior-inferior–most point	Anterior-most point	Middle point in the anteroposterior slice determined by the lateral and axial view
5. Left upper incisal alveolar ridge(lUIAR)	Premaxilla alveolar	Anterior-inferior–most point	Anterior-most point	Middle point in the anteroposterior slice determined by the lateral and axial view
6. Right pyriform aperture(rPA)	pyriform aperture	Anterior-most point in the Lateral slice determined by the Anteroposterior view and axial view	Anterior-most point	Lateral–most poin
7. Left pyriform aperture(lPA)	pyriform aperture	Anterior-most point in the Lateral slice determined by the Anteroposterior view and axial view	Anterior-most point	Lateral–most poin
8. Right Zygomatic suture (rZS)	Zygomaticomaxillary suture	Anterior-inferior–most point	Anterior-most point	Lateral-inferior–most point
9. Left Zygomatic suture (lZS)	Zygomaticomaxillary suture	Anterior-inferior–most point	Anterior-most point	Lateral-inferior–most point

Fig. 3.

Landmarks displayed in the 3D virtual surface model

Fig. 4.

Example of identification of the ANS point in the 3 planes of space. The software allows tracking of the cursor with display of all 3 planes of space and 3D rendering in the same software window to verify landmark location

The training and calibration phase involved three individuals with expertise in orthodontics, including an orthodontist, an orthodontics master, and a dental radiologist. They were provided with a separate set of 30 CBCT scans for the purpose of completing the entire process of superimposing measurements. In this study, these scans were not included. Following this, each observer independently identified nine specific anatomical landmarks (Table 1) within the superimposed CBCT volume for both pre- and post-treatment images of each patient. The X, Y, and Z coordinates corresponding to each landmark were then recorded from sagittal, coronal, and axial views respectively. These values were subsequently exported into Microsoft Excel for later analysis purposes. To ensure accuracy and consistency, each observer repeated the superimposing measurement three times at five-day intervals resulting in a total of 90 sets per observer.

The statistical analyses were conducted using a statistical software package, SPSS version 21.0 (SPSS, Chicago, IL, USA). Initially, the differences in the coordinates'values (i.e., the alterations resulting from the treatment and growth) across all landmarks were computed by subtracting the pre-treatment from the post-treatment coordinate values for each respective landmark. Subsequently, the intra-observer agreement was evaluated by calculating the Intraclass correlation coefficient(ICCs). The distribution and variance of the data were meticulously evaluated using the Shapiro–Wilk normality test to ascertain the Gaussian distribution and the Levene's homogeneity test to ensure the equality of variances. For inter-observer comparisons, both the independent T-test and the non-parametric Mann–Whitney U-test were employed. Meanwhile, the paired T-test was utilized to determine the intra-observer reliability.

Results

Voxel-based superimposition visually demonstrated, the morphological skeletal changes of the craniomaxillary region for these Class III patients after protraction therapy:

The intraobserver agreement, as was estimated by the ICCs for the coordinate difference of each landmark, was listed in Tables 2 and 3. Generally, the results indicated excellent reliability for intraobserver assessments, with ICC > 0.90 for 25 (92.6%) of intraobserver assessments.

Table 2.

Assessment of intraobserver reliability: Intraclass correlation Coefficient(ICC) and confidence interval for repeated measurements (from the triplicate evaluation)

Landmark	Intraclass Correlation			95% Confidence Interval
	X	Y	Z	X	Y	Z
Nasion (N)	0.96	0.99	0.97	0.89–0.99	0.94–1.00	0.91–1.00
Anterior nasal spine (ANS)	0.96	0.98	0.98	0.90–0.99	0.92–1.00	0.92–1.00
A point (A)	0.97	0.92	0.99	0.92–0.99	0.81–0.96	0.95–1.00
Right upper incisal alveolar ridge(rUIAR)	0.95	0.98	0.97	0.86–0.98	0.90–1.00	0.90–1.00
Left upper incisal alveolar ridge(lUIAR)	0.94	0.98	0.98	0.85–0.98	0.92–1.00	0.92–1.00
Right pyriform aperture(rPA)	0.95	0.97	0.78	0.85–0.99	0.90–0.99	0.62–0.84
Left pyriform aperture(lPA)	0.96	0.97	0.81	0.89–0.99	0.91–1.00	0.74–0.89
Right Zygomatic suture (rZS)	0.91	0.96	0.94	0.80–0.95	0.89–0.99	0.85–0.98
Left Zygomatic suture (lZS)	0.90	0.95	0.94	0.80–0.94	0.86–0.99	0.83–0.98

Table 3.

Assessment of intraobserver reliability: ICC values of changes in madibular measurements difference in X, Y, and Z coordinates

Range	Coordinate difference
	X		Y		Z		Total
	n	(%)	n	(%)	n	(%)	n	(%)
ICC ≥ 0.95	6	(66.7)	8	(88.9)	5	(55.6)	19	(70.4)
0.90 ≤ ICC < 0.95	3	(33.3)	1	(11.1)	2	(22.2)	6	(22.2)
0.85 ≤ ICC < 0.90	0	0	0	0	0	0	0	0
0.80 ≤ ICC < 0.85	0	0	0	0	1	(11.1)	1	(3.7)
ICC < 0.80	0	0	0	0	1	(11.1)	1	(3.7)
Total	9	(100.0)	9	(100.0)	9	(100.0)	27	(100.0)

Regarding the ICCs for the coordinate difference,low ICC scores were obtained for 2 bilateral landmarks indicating a relatively low reliability.They were the X coordinate of the right and left zygomatic suture, and the Z coordinate of the right and left pyriform aperture.

Table 4 showed the frequency counts of the difference in mean values of the coordinate (i.e. difference arising from treatment and growth for each landmark). This was used to assess the reproducibility of this measurement method. The precision of the measurement was better than 0.3 mm in 21 cases (77.8%). For the difference in mean values for the X, Y, and Z coordinates used to estimate interobserver reproducibility, similar ICC results were obtained.The interobserver reproducibility errors were < 0.3 mm in 21 of the 27 cases; only 6 cases (22.2%) showed error ≥ 0.3 mm.

Table 4.

Frequency of the difference in mean values for difference among measurement (mm) and for reproducibility of the measurement method (mm)

Range (mm)	Among measurement								Reproducibility of the measurement method
	Coordinate difference								Coordinate difference
	X		Y		Z		Total		X		Y		Z		Total
	n	(%)	n	(%)	n	(%)	n	(%)	n	(%)	n	(%)	n	(%)	n	(%)
≥ 0.3	2	−22.2	1	−11.1	3	−33.3	6	−22.2	2	−22.2	1	−11.1	3	−33.3	6	−22.2
0.15 ≤ x < 0.3	7	−77.8	7	−77.7	6	−66.7	20	−74.1	7	−77.8	7	−77.7	6	−66.7	20	−74.1
≤ 0.15	0	0	1	−11.1	0	0	1	−3.7	0	0	1	−11.1	0	0	1	−3.7
Total	9	−100	9	−100	9	−100	27	−100	9	−100	9	−100	9	−100	27	−100

Discussion

Melsen [15] confirmed that by the age of 7, the growth of sphenoethmoidal and sphenofrontal suture of the anterior cranial base usually stops. The authors also pointed out that after 5 years of age, changes in sella turcica were most likely, to some degree, due to resorptive activity in the lower half of the posterior wall and the floor of the sella turcica. On the other hand, the anterior part of the sella turcica was the most stable, and resting (inactive) bone was observed in almost all subjects. The brain almost stops growing at 7–8 years of age, after which the anterior cranial base continues to grow and contributes to facial development. That growth occurs almost entirely due to increased pneumatization of the frontal and ethmoid bones [19]. This present study used the voxel based superimposition of CBCT data of the anterior cranial base in growing patients, to examine the coordinate difference in position between the pre- and post- treatment for skeletal class III malocclusion. Our patients’ age was 8–11, and hence choice of stable reference was approriate. The three-dimensional analysis was carried out in a five-step process: initial model construction, subsequent reorientation, voxel-based alignment, precise landmark delineation, and concluding quantitative assessments. The coordinate systems were automatically generated by the software during the construction of the models. Although the coordinates of various structures on the reconstructed models might differ, that difference could usually be resolved by reorienting the models, that is, the coordinates of the pre- and post- treatment landmarks were very little affected. The superimposition methods were fully automated, with voxel-wise rigid registration of the anterior cranial base structures that have completed growth and did not change further with age [20–22]. The bilateral structural landmarks were involved for this study.: the nasion (N), the anterior nasal spine (ANS), A point (A), the right upper incisal alveolar ridge (rUIAR), the left upper incisal alveolar ridge (lUIAR), the right pyriform aperture (rPA), the left pyriform aperture (lPA), the right zygomatic suture (rZS), and the left zygomatic suture (lZS). The N, rPA, lPA, rZS, and lZS together reflected the changes in the upper part of the maxilloface. The ANS, A, rUIAR, and lUIAR together reflected the changes in the lower part of the maxilloface. In additional, the rUIAR and lUIAR reflected the changes in the premaxilla alveolar.

Errors may arise in 4 subsequent steps of the data processing procedure: a) model reorientation, b) voxel-based superimposition, c) landmark identification (localization), and d) quantitative measurement. Firstly, even if the head for CBCT scans were consistently positioned according to the protocol, scan data with slight variations in head position would still occur due to such factors as varying body positions and neck curvatures. This could result in inter-observer variability in cranial base registration.

In this research, proper automated voxel registration need these two CBCT scanings approximated as far as possible, Two CBCTS with significant differences cannot be performed voxel registrationand. And different operators, using the same patient scans, the same references for registration, same software may or may not achieve proper registration. However, as the subjects’ craniofacial characteristics were basically symmetrical, the method error could be minimized by model reorientation. Secondly, the voxel-based superimposition was semi-automated, requiring observers to select the relevant area before the computer executed the superimposition automatically. The error in this process was contingent upon the area chosen by the observers [23]. If there was a discrepancy in the region selected on the pre- and post-treatment models during the initial superimposition, it could introduce errors into the superimposition results. To address this issue, we refrained from finalizing the superimposition until the results were consistent. Thirdly, two sources of error were identified during the landmark identification process in this study: a) Although it was comparatively straightforward to delineate landmarks in the 3D virtual surface models, the determination of the optimal slice or region for the localization of the selected landmarks posed a significant challenge [24]; b) The three spatial planes are intrinsically interconnected. An alteration in the slice position within one plane will inevitably lead to the displacement of the reference line in another plane. Consequently, a certain level of proficiency on the part of the observers was indispensable, and in preparation for this, training was conducted utilizing an additional 30 sets of scan data for the observers. In the current investigation, despite the three observers possessing distinct professional backgrounds, this appeared to exert a minimal influence on the variance in measurement errors. The training and calibration of the observers or assessors is of paramount importance and cannot be overstated in scholarly inquiries. Additional exploration is necessitated to delve into other variables that are pertinent to the precision and repeatability of three-dimensional (3D) measurements, including parameters such as voxel size, scanning duration, and the extent of the scanning range [25]. Within the scope of this research, a voxel resolution of 0.3 mm was employed, which contrasts with previous CBCT studies that have utilized resolutions ranging from 0.5 mm to as large as 3 mm. Recent scholarly contributions [26, 27] have suggested that diminutive voxel sizes are indicative of enhanced measurement accuracy and reduced measurement discrepancies.

Despite the general reliability demonstrated by the coordinate discrepancies presented in Table 2 for the right and left zygomatic sutures, the method yielded results with enhanced intraobserver consistency and inter-observer reproducibility. The moderate reproducibility may be attributed to ambiguous criteria for definition, including the inconsistent selection of the optimal perspective and section. Furthermore, Table 3 indicates that the reliability of Z-coordinate definition was comparatively inferior to that of the X and Y coordinate definitions, which could be associated with the reduced visibility of certain landmarks in the Z-coordinate. Consequently, the reproducibility of landmarks is contingent upon their distinct characteristics. The selection of landmarks exerts a significant influence on the reliability and reproducibility of measurement outcomes [28].

In the present study, we introduced a novel 3D quantitative measurement approach for evaluating maxillary protraction treatment outcomes in children with skeletal Class III malocclusion. These coordinate systems were automatically generated by the software when constructing the models. Orthodontists typically allocate a duration of approximately 5 min to superimpose pre- and post-treatment CBCT scans, thereby visually illustrating the therapeutic efficacy to patients and their parents [29]. This process enables orthodontists to graphically delineate the anatomical localization and scope of treatment-induced modifications, which in turn enhances patient understanding [30]. The landmarks chosen for this investigation extensively represented the maxillofacial alterations. As indicated in Table 4, the intrarater ICC values exceeded 0.90 for 25 (92.6%) of the assessments. The method's precision was achieved at a threshold of < 0.3 mm in 21 (77.8%) instances. I think this level of precision is brought about by the functions of 3D software and the anatomical structures of the anterior cranial base selected as excellent overlapping area. Furthermore, Table 4 revealed that interrater reproducibility errors were maintained at < 0.3 mm in 21 out of the 27 cases evaluated. Collectively, the method exhibited exceptional reliability and reproducibility. It is imperative to propose and validate additional landmarks to ascertain the applicability of this method for assessing other growth-related or treatment modalities.

Conclusion

Overall, the assessment of craniomaxillary changes via voxel-based superimposition of CBCT models demonstrates a high level of intraobserver reliability and interobserver reproducibility. This method is particularly effective when measuring the anterior cranial base in growing patients with skeletal Class III malocclusion. Furthermore, the application of this technique facilitates the explanation of the therapeutic process and the visualization of improvements in craniomaxillary positions to patients following maxillary protraction.

Abbreviations

3D

Three-dimensional

2D

Two-dimensional

CBCT

Cone-beam computed tomography

ICC

Intraclass correlation coefficient

ANS

Anterior nasal spine

rUIAR

Right upper incisal alveolar ridge

lUIAR

Left upper incisal alveolar ridge

rPA

Right pyriform aperture

lPA

Left pyriform aperture

rZS

Right Zygomatic suture

lZS

Left Zygomatic suture

DICOM

Digital Imaging and Communications in Medicine

XiaoYing Hu

Orthodontist. There are about 6000 outpatient visits per year. She have finished the correction of over 1000 cases with Class III Malocclusion. During the management of these patients, she noted the deleterious effect of bad tongue habit on the development of Class III Malocclusion. The bad tongue habit also caused narrowing of the upper airway even in deciduous period of about 1-2years of age. So she consulted two experts (Gary Shun Pan Cheung and FuSheng Dong) to investigate the topics. The data show the craniofacial and upper airway characteristics with Class III malocclusion in 4 to5year-old children, and reflect the outcome of the early intervention quatitatively She also wish obtained to find a way to decrease the recurrence of Class III malocclusion.

Authors’ contributions

*XiaoYing Hu 1** made substantial contributions to design of the work,and the acquisition, analysis and interpretation of data, and revise the manuscript. *YiYang Zhang2* involved in the interpretingand drafting the manuscript. *Gary Shun Pan Cheung3* contributed to the conception and analysis of data, and revise the manuscript. *RuoNan Sun 4* drafted and performed the work and drafted the manuscript. *FuSheng Dong 5* have made substantial contributions to the conception, design of the experimental work and analysis of data.

Funding

This work was supported by grants from the Program of Key Science and Technology Research, Health Commission of Hebei Province (Project No. 20191074).

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. Data are stored in controlled access data storage at The University of Hong Kong-Shenzhen Hospital due to sensitivity considerations.

Declarations

Ethics approval and consent to participate

All experiments were performed in accordance with relevant guidelines and regulations (Declarations of Helsinki).

Study protocol was approved by the Medical Ethical Commission of the University of Hong Kong-Shenzhen Hospital.

The informed consents from all subjects and/or their legal guardian(s) have been obtained for study participation.

Consent for publication

The informed consents from all subjects and/or their legal guardian(s) have been obtained for publication of identifying information/images in an online open-access publication.

Competing interests

The authors declare no competing interests.