A semi-supervised learning approach for automated 3D cephalometric landmark identification using computed tomography

Abstract

Identification of 3D cephalometric landmarks that serve as proxy to the shape of human skull is the fundamental step in cephalometric analysis. Since manual landmarking from 3D computed tomography (CT) images is a cumbersome task even for the trained experts, automatic 3D landmark detection system is in a great need. Recently, automatic landmarking of 2D cephalograms using deep learning (DL) has achieved great success, but 3D landmarking for more than 80 landmarks has not yet reached a satisfactory level, because of the factors hindering machine learning such as the high dimensionality of the input data and limited amount of training data due to the ethical restrictions on the use of medical data. This paper presents a semi-supervised DL method for 3D landmarking that takes advantage of anonymized landmark dataset with paired CT data being removed. The proposed method first detects a small number of easy-to-find reference landmarks, then uses them to provide a rough estimation of the all landmarks by utilizing the low dimensional representation learned by variational autoencoder (VAE). The anonymized landmark dataset is used for training the VAE. Finally, coarse-to-fine detection is applied to the small bounding box provided by rough estimation, using separate strategies suitable for the mandible and the cranium. For mandibular landmarks, patch-based 3D CNN is applied to the segmented image of the mandible (separated from the maxilla), in order to capture 3D morphological features of mandible associated with the landmarks. We detect 6 landmarks around the condyle all at once rather than one by one, because they are closely related to each other. For cranial landmarks, we again use the VAE-based latent representation for more accurate annotation. In our experiment, the proposed method achieved a mean detection error of 2.88 mm for 90 landmarks using only 15 paired training data.

1 Introduction

Cephalometric analysis is commonly used by dentists, orthodontists, and oral and maxillofacial surgeons to provide morphometrical guidelines for diagnosis, surgical planning, growth analysis, and treatment planning by analyzing dental and skeletal relationships in the craniofacial complex [1]. It is based on cephalometric landmarks, which serve as proxy to the skull morphological data pertaining to craniofacial characteristics [2]. Conventional cephalometric analysis uses two-dimensional (2D) cephalometric radiographs (lateral and frontal radiographs), which have drawbacks including geometric distortions, superimpositions, and the dependence on correct head positioning [3]. Due to recent advances in image processing techniques and the need for accurate craniofacial analysis, a three-dimensional (3D) approach to the cephalometric landmarks obtaining 3D computerized tomography (CT) images is gaining preference over the conventional 2D techniques [4–6].

Recently, there have been many studies conducted on automated cephalometric landmark identification that aims to find the landmarks and enable immediate cephalometric analysis, because manual landmarking and cephalometric analysis are labor-intensive and cumbersome tasks even for the trained experts. Due to recent advances in deep learning techniques, the automated annotation of 2D cephalometric landmarks may now be used for clinical application [7, 8]. Conversely, automated 3D cephalometric tracing (for 90 landmarks) may not yet be utilized in clinical applications, wherein the required average error is commonly designated to be less than 2 mm [9–13]. The high dimensionality of the input data (e.g., 512 × 512 × 512) and limited number of training data are the main factors that hinder the training of deep learning networks for learning the 3D landmark positional vectors from 3D CT data. Moreover, due to the current legal and ethical restrictions on medical data, it is very difficult to utilize CT data from patients.

To overcome the above-mentioned learning problems caused by the high input dimensions and training data deficiencies, the method proposed in this study utilizes semi-supervised learning that takes advantage of a large number of anonymized landmark dataset (without using the corresponding CT dataset) which have been used in surgical planning and treatment evaluation. We use these landmark dataset to obtain their low dimensional representations, reducing the dimensions of the total landmark vectors (270 = 90 × 3 dimension) to only 9 latent variables via a variational autoencoder (VAE) [14]. For training the VAE, a normalized landmark dataset is used to efficiently learn skull shape variations while ignoring unnecessary scaling factors. With this dimensionality reduction technique, the positions of all 90 landmarks can be roughly estimated by identifying a small number of easy-to-find reference landmarks (10 landmarks), which can be accurately and reliably identified via a simple deep learning method [11].

The rough estimation of all landmarks is used to provide a small 3D bounding box for each landmark in the 3D CT images. Following this, we apply convolutional neural networks (CNNs) to these small bounding boxes to enable the accurate placement of landmarks. Our fine detection strategy is divided into two parts; mandible and cranium. It is desirable to accurately capture the morphological variability of the mandible because the shape of the mandible can be affected by a variety of factors, including the masticatory occlusal force, muscular force, functional activity such as breathing and swallowing, and age [15]. Noting that landmarks on the mandible represent morphological features of a 3D mandibular surface geometry, we apply 3D CNN to a segmented image of the mandible (separated from the cranium). We follow a recent study [16] for a segmentation method to separate the mandible from the cranium.

Because several landmarks around the condyle are closely related to each other, it is better to detect these landmarks all at once. For the landmarks on the midsagittal plane, it is better to further reduce the dimensionality of the input by using a partially integrated 2D image of the midsagittal plane. For the remaining landmarks lying on the cranium, we again use the anonymized landmark dataset to obtain a more accurate latent representation of all landmarks on the cranium, due to its rigidity. The proposed approach achieved a mean detection error of 2.88 mm for 90 landmarks, which nearly meets the clinically acceptable precision standard. It should be emphasized that this accuracy has been achieved using a very small amount of training data.

2 Method

We begin by introducing the following notations. Five easy-to-find reference landmarks (CFM, Bregma, Na, and Po (L/R)) are used as the basis for constructing a coordinate system to determine the midsagittal and axial planes, and they are utilized for data normalization (methods for obtaining these five reference landmarks will be described in Section 2.1).

• x denotes a 3D CT image, which is defined on a voxel grid Ω ≔ {v = (v₁, v₂, v₃):v_j = 1, ⋯, 512 for j = 1, 2, 3}. Here, we set v₁ as the normal direction of the midsagittal plane.

• x_b denotes a binarized CT image of x (i.e., skull segmentation from the CT image), defined by

  $$\begin{array}{c}\hfill {\mathit{x}}_b=\{\begin{array}{cc}{\mathit{x}}_b(v_1,v_2,v_3)=1& \text{if}\mathit{x}(v_1,v_2,v_3)\ge \rho \hfill \\ {\mathit{x}}_b(v_1,v_2,v_3)=0& \text{otherwise}\hfill \end{array}\end{array}$$ (1)

  where ρ is a thresholding value. In our experiment, the value of ρ was consistently chosen as ρ = 500HU, which is known as an effective choice for thresholding-based bone segmentation [17].
• x^mid denotes a partially integrated 2D image of x_b in the normal direction of the midsagittal plane, defined by

  $$\begin{array}{c}\hfill {\mathit{x}}^{\text{mid}}=\sum _{v_1=a}^b{\mathit{x}}_b(v_1,v_2,v_3)\end{array}$$ (2)

  where [a, b] determines the truncated volume of x_b.

• ${\mathfrak{R}}^{\text{cr}}\in {\mathbb{R}}^{138(=46\times 3)}$ and ${\mathfrak{R}}^{\text{md}}\in {\mathbb{R}}^{132(=44\times 3)}$ denote the concatenated vectors of 46 cranial and 44 mandibular 3D landmarks, respectively. The entirety of the landmarks $\mathfrak{R}\in {\mathbb{R}}^{270(=90\times 3)}$ is defined by $\mathfrak{R}≔[{\mathfrak{R}}^{\text{cr}},{\mathfrak{R}}^{\text{md}}]$. See S1 Table for more detailed information of the landmarks.

• ${\mathfrak{R}}_{♯}^{\text{cr}}\in {\mathbb{R}}^{24(=8\times 3)}$ denotes a concatenated vector of the landmarks (Bregma, CFM, Na, ANS, Or (L/R), and Po (L/R)) in the cranium and ${\mathfrak{R}}_{♯}^{\text{md}}\in {\mathbb{R}}^{6(=2\times 3)}$ denotes a concatenated vector of the landmarks (MF (L/R)) in the mandible. A reference landmark vector ${\mathfrak{R}}_{♯}\in {\mathbb{R}}^{30(=10\times 3)}$ is defined by ${\mathfrak{R}}_{♯}=[{\mathfrak{R}}_{♯}^{\text{cr}},{\mathfrak{R}}_{♯}^{\text{md}}]$.

We mention here details of reference landmarks; Bregma is the point of junction of the coronal and sagittal sutures of the skull. (i) CFM, (ii) Na, (iii) ANS, (iv) Or, (v) Po, and (vi) MF are the abbrevations of (i) the center of foramen magnum, (ii) nasion, (iii) anterior nasal spine, (iv) orbitale, (v) porion, and (vi) mental foramen, which are defined by (i) the center of an opening for spinal cord, (ii) the center of the midline bony depression between the eyes, where the frontal and two nasal bones meet, just below the glabella, (iii) the projection formed by the fusion of the two maxillary bones at the intermaxillary suture, (iv) lower most point on the lower margin of the left or right orbit, (v) the most superior point of the upper margin of each ear canal, and (vi) a bilateral opening in the vestibular portion of the mandible through which nerve endings, such as the mental nerve, emerge. See Fig 1.

Fig 1. Reference landmarks.

These are easy-to-find through CNN with input of the illuminated images because they have strong geometric cues that can be revealed in illuminated 2D images.

The 3D cephalometric landmarking aims to develop a function $f:\mathit{x}\mapsto \mathfrak{R}$ that maps a 3D CT image x to all landmarks $\mathfrak{R}$. To learn the landmark detection map f, deep learning techniques can be used. Unfortunately, due to the legal and ethical restrictions on medical data, a few paired data are available. This severe shortage of paired data makes it difficult to obtain an accurate and reliable map $f:\mathit{x}\mapsto \mathfrak{R}$ in the following supervised learning framework:

$$\begin{array}{c}\hfill f=\underset{f\in \mathbb{N}\text{et}}{\text{argmin}}\frac{1}{N_p}\sum _{i=1}^{N_p}{\Vert f\left({\mathit{x}}^{\left(i\right)}\right)-{\mathfrak{R}}^{\left(i\right)}\Vert }_2^2,\end{array}$$ (3)

where N_p is the small number of paired training data, $\{({\mathit{x}}^{\left(i\right)},{\mathfrak{R}}^{\left(i\right)}):i=1,\cdots ,N_p\}$ is a paired dataset, $\mathbb{N}\text{et}$ is a deep learning network, and ‖ ⋅ ‖ is the standard Euclidean norm. In our study, only 15 paired data are available (i.e., N_p = 15). Even with a certain amount of paired data, the learning process (3) of the direct detection map f can be difficult because the dimension of the input image is very large (greater than 10⁸).

The proposed method attempts to address this problem by taking advantage of a semi-supervised learning framework that permits the utilization of the N_l number of anonymized landmark data ${\left\{{\mathfrak{R}}^{(N_p+i)}\right\}}_{i=1}^{N_l}$ whose corresponding CT data are not provided. In this research, specifically, 229 anonymized data (i.e., N_l = 229) are utilized.

As shown in Fig 2, the proposed method comprises the following three main steps: (i) To obtain easy-to-find reference landmarks ${\mathfrak{R}}_{♯}$, we apply CNN with 2D illuminated images generated from a binarized CT image x_b and normalize the output with respect to the cranial volume. (ii) A rough estimation of entire landmarks $\mathfrak{R}$ is obtained using the partial knowledge ${\mathfrak{R}}_{♯}$ and a VAE-based low dimensional representation of $\mathfrak{R}$. (iii) Using this estimation, coarse-to-fine detection for $\mathfrak{R}$ is conducted, wherein separate strategies are utilized for the mandibular and cranial landmarks. For the mandibular landmarks, the landmarks are accurately identified by applying 3D patch-based CNNs to capture the morphological features on a 3D surface geometry associated with the landmarks, wherein an input patch is extracted based on the coarse estimation. For cranial landmarks, we first detect three landmarks lying on the midsagittal plane by applying a 2D CNN whose input is an extracted 2D patch from a partially integrated image x^mid in basis of the coarse estimation. By utilizing the three finely-detected landmarks and cranial reference landmarks ${\mathfrak{R}}_{♯}^{\text{cr}}$ as the partial information of ${\mathfrak{R}}^{\text{cr}}$, the remaining cranial landmarks are finely annotated via a VAE-based local-to-global estimation utilizing the same method in the previous step.

Fig 2. Schematic diagram of the proposed method for the 3D landmark annotation system.

Each of these steps is described in detail as follows.

2.1 Detection of easy-to-find reference landmarks and uniform scaling for skull normalization with respect to the cranial volume

The first step of the proposed method is to find 10 reference landmarks ${\mathfrak{R}}_{♯}$ from a given x. Initially, a CT image x is converted into a binarized image x_b by (1). From x_b, 2D illuminated images are generated by manipulating various lighting and viewing directions (see Fig 1). By applying VGGNet [18] to these illuminated images, the reference landmarks ${\mathfrak{R}}_{♯}$ are accurately and automatically identified. This detection method is based on that presented in the recent study [11].

Using these identified reference landmarks, data normalization is conducted for efficient feature learning of skull shape variations in further steps. By applying a uniform scaling with respect to the cranial volume, the landmark vector ${\mathfrak{R}}_{♯}$ is normalized, wherein the cranial volume is defined via a product of the distance between the v₁-coordinate of Po (L) and Po (R) (cranial length), the distance between the v₂-coordinate of Po (L) and Na (depth), and the distance between the v₃-coordinate of CFM and Bregma (height). This data normalization minimizes the positional dependencies of the landmarks on the translation, rotation, and overall size of the skull; therefore, shape information of the skull (regarding facial deformities) can be effectively learned in further VAE-based steps. From here on, we will denote all landmark vectors as normalized vectors (e.g., $\mathfrak{R}$ and ${\mathfrak{R}}_{♯}$ are normalized vectors for total landmarks and reference landmarks).

2.2 Rough estimation of all landmarks from reference landmarks using VAE-based low dimensional representation

This subsection provides a method for roughly estimating all landmarks $\mathfrak{R}$ from the reference landmarks ${\mathfrak{R}}_{♯}$ that are accurately annotated in the previous step. Based on the method in [13], we build a bridge that connects ${\mathfrak{R}}_{♯}$ and $\mathfrak{R}$ by taking advantage of a low-dimensional representation of $\mathfrak{R}$ learned by the variational autoencoder (VAE) [14].

The rough estimation obtained from ${\mathfrak{R}}_{♯}$, denoted by ${\mathfrak{R}}_{◇}$, is given by

$$\begin{array}{c}\hfill {\mathfrak{R}}_{◇}=\mathcal{D}\circ \Phi \left({\mathfrak{R}}_{♯}\right)\end{array}$$ (4)

where $\mathcal{D}\circ \Phi :{\mathfrak{R}}_{♯}\mapsto {\mathfrak{R}}_{◇}$ is a local-to-global landmark estimation map as described in Fig 3. The map is constructed as follows: First, we train VAE that consists of an encoder ($\mathcal{E}$) and a decoder ($\mathcal{D}$), to learn low dimensional representation of $\mathfrak{R}$. Afterwards, we train the nonlinear map Φ that provides $\Phi \left({\mathfrak{R}}_{♯}\right)\approx \mathcal{E}\left(\mathfrak{R}\right)$ so that $\mathcal{D}\circ \Phi \left({\mathfrak{R}}_{♯}\right)\approx \mathfrak{R}$.

Fig 3. Initial estimation of all 90 landmarks $\mathfrak{R}$ using the knowledge of 10 reference landmarks ${\mathfrak{R}}_{♯}$.

This is possible because all landmarks $\mathfrak{R}$ can be roughly represented by only 9 latent variables.

Specifically, the VAE learns an encoder $\mathcal{E}:\mathfrak{R}\mapsto \mathbf{z}$ and a decoder $\mathcal{D}:\mathbf{z}\mapsto \mathfrak{R}$, where $\mathbf{z}\in {\mathbb{R}}^d$ is a d-dimensional latent variable (d ≪ 270). The maps $\mathcal{E}$ and $\mathcal{D}$ learn landmark patterns by leveraging dataset ${\left\{{\mathfrak{R}}^{\left(i\right)}\right\}}_{i=1}^{N_t}$ that consists of the unpaired landmark dataset as well as the paired dataset. The training is achieved via the following energy minimization sense:

$$\begin{array}{c}\left(\mathcal{E},\mathcal{D}\right)=\underset{\left(\mathcal{E},\mathcal{D}\right)\in \mathbb{V}\mathbb{A}\mathbb{E}}{\text{argmin}}{\displaystyle \sum }_{i=1}^{N_t}(\Vert \mathcal{D}\circ \mathcal{E}\left({\Re }^{\left(i\right)}\right)-{\Re }^{\left(i\right)}{\Vert }_2^2+D_{KL}\left(\mathcal{N}\left({\mu }^{\left(i\right)},{\Sigma }^{\left(i\right)}\right)\Vert \mathcal{N}\left(0,I\right)\right))\end{array}$$ (5)

where N_t = N_p + N_l is the total number of training landmark data, $\mathbb{V}\mathbb{A}\mathbb{E}$ is a class of functions in the form of a given VAE network, $\mathcal{N}({\mu }^{\left(i\right)},{\Sigma }^{\left(i\right)})$ is a d-dimensional normal distribution with a mean μ⁽ⁱ⁾ and a diagonal covariance matrix Σ⁽ⁱ⁾ = diag((σ⁽ⁱ⁾(1))², ⋯, (σ⁽ⁱ⁾(d))²), $\mathcal{N}(0,I)$ is a standard normal distribution, and the last term in the loss function is the Kullback-Leibler (KL) divergence defined by

$$\begin{array}{cc}& D_{KL}(\mathcal{N}({\mu }^{\left(i\right)},{\Sigma }^{\left(i\right)})\Vert \mathcal{N}(0,I))\hfill \\ & =\frac{1}{2}\sum _{l=1}^d({\mu }^{\left(i\right)}{\left(l\right)}^2+{\sigma }^{\left(i\right)}{\left(l\right)}^2-\text{log}{\sigma }^{\left(i\right)}\left(l\right)-1)\hfill \end{array}$$ (6)

Here, μ⁽ⁱ⁾ = (μ⁽ⁱ⁾(1), ⋯, μ⁽ⁱ⁾(d)) and σ⁽ⁱ⁾ = (σ⁽ⁱ⁾(1), ⋯, σ⁽ⁱ⁾(d)) are the mean and standard deviation vectors obtained in the interim of the encoding process of an i-th training data ${\mathfrak{R}}^{\left(i\right)}$ (i.e., $\mathcal{E}\left({\mathfrak{R}}^{\left(i\right)}\right)$).

The encoder $\mathcal{E}$ can be expressed in the following nondeterministic form:

$$\begin{array}{c}\hfill \mathcal{E}\left(\mathfrak{R}\right)=\mathbf{z}≔\mu +\sigma \odot {\mathbf{h}}_{\text{noise}}\end{array}$$ (7)

where h_noise is a noise sampled from $\mathcal{N}(0,I)$, ⊙ is the Hadamard product (i.e., element-wise product), and vectors μ and σ are given by

$$\begin{array}{cc}& \mu ={\mathbf{E}}_4^{\mu }\mathbf{h},\sigma ={\mathbf{E}}_4^{\sigma }\mathbf{h}\hfill \\ & \mathbf{h}=\text{ReLU}\left({\mathbf{E}}_3\text{ReLU}\left({\mathbf{E}}_2\text{ReLU}\left({\mathbf{E}}_1\mathfrak{R}\right)\right)\right)\hfill \end{array}$$ (8)

Here, the matrices $\{{\mathbf{E}}_1,{\mathbf{E}}_2,{\mathbf{E}}_3,{\mathbf{E}}_4^{\mu },{\mathbf{E}}_4^{\sigma }\}$ represent fully-connected layers and ReLU is an element-wise activation function defined by ReLU(t) = max(t, 0). The decoder $\mathcal{D}$ is the reverse process of the encoder $\mathcal{E}$, which can be represented by

$$\begin{array}{c}\hfill \mathcal{D}\left(\mathbf{z}\right)={\mathbf{D}}_1\text{ReLU}\left({\mathbf{D}}_2\text{ReLU}\left({\mathbf{D}}_3\text{ReLU}\left({\mathbf{D}}_4\mathbf{z}\right)\right)\right)\end{array}$$ (9)

where the matrices {D₁, D₂, D₃, D₄} represent fully-connected layers. The detailed network architecture is described in the red box of Fig 3.

After pretraining the VAE, the nonlinear map $\Phi :{\mathfrak{R}}_{♯}\to \mathbf{z}$ is learned, which connects reference landmarks ${\mathfrak{R}}_{♯}$ with a latent variable $\mathbf{z}=\mathcal{E}\left(\mathfrak{R}\right)$. The architecture of the map Φ is a fully-connected neural network as described in Fig 3. The training is achieved by the following minimization sense:

$$\begin{array}{c}\hfill \Phi =\underset{\Phi }{\text{argmin}}\sum _i{\Vert \Phi \left({\mathfrak{R}}_{♯}^{\left(i\right)}\right)-{\mathbf{z}}^{\left(i\right)}\Vert }_2^2\end{array}$$ (10)

Here, we remark that Φ relies on the pretrained VAE, whose encoder ($\mathcal{E}$) is used only for training and decoder ($\mathcal{D}$) is the component of the local-to-global detection map.

The resultant map $\mathcal{D}\circ \Phi$ estimates all landmarks $\mathfrak{R}$ from the partial knowledge ${\mathfrak{R}}_{♯}$, based on the learned patterns of landmarks by VAE.

2.3 Coarse-to-fine detection

This subsection explains coarse-to-fine detection obtained using the initial estimation ${\mathfrak{R}}_{◇}$. We put a final touch on ${\mathfrak{R}}_{◇}$ by utilizing CT image data. The coarse-to-fine detection is based on suitable strategies that rely on the landmark locations (i.e. on the mandible or cranium). The details are explained in the following subsections.

2.3.1 Mandible-cranium segmentation

In the binarized skull image x_b, we segment the mandible and cranium separately using the connected component labeling (CCL) technique [19]. Among all connected components generated from the CCL method, the largest component and the second largest are the cranium and the mandible respectively. The segmented cranium and mandible images are denoted as ${\mathit{x}}_b^{\text{cr}}$ and ${\mathit{x}}_b^{\text{md}}$ (as shown in Fig 2). Using these images and the rough estimation ${\mathfrak{R}}_{◇}$, the following fine detection processes are conducted.

2.3.2 Detection of mandibular landmarks

For the landmarks on the mandible being articulated to the skull, a patch-based 3D CNN is applied to capture the morphological variability of the 3D mandibular surface geometry associated with the landmarks.

Let ${\mathfrak{R}}_{◇}^j\in {\mathbb{R}}^3$ be a roughly estimated position of a landmark with index j in ${\mathfrak{R}}_{◇}$. See S1 Table for the details of the landmark index. For each mandibular landmark (i.e., j ∈ {49, ⋯, 90}), we extract a 3D image patch ${\left({\mathit{x}}_b^{\text{md}}\right)}_{(\eta ,{\mathfrak{R}}_{◇}^j)}$ that is defined by a cube with edge length of η and center of ${\mathfrak{R}}_{◇}^j$. By using 3D CNN, we obtain a map $f_j^{\text{md}}:{\left({\mathit{x}}_b^{\text{md}}\right)}_{(\eta ,{\mathfrak{R}}_{◇}^j)}\mapsto {\widetilde{{\mathfrak{R}}_{◇}}}^j$, where ${\widetilde{{\mathfrak{R}}_{◇}}}^j$ is an accurate positional estimation for the landmark with index j (i.e., ${\widetilde{{\mathfrak{R}}_{◇}}}^j\approx {\mathfrak{R}}^j$).

To learn the fine detection map $f_j^{\text{md}}$, we generate training dataset by using the paired dataset ${\left\{({\left({\mathit{x}}_b^{\text{md}}\right)}^{\left(i\right)},{\mathfrak{R}}^{\left(i\right)})\right\}}_{i=1}^{N_p}$: For a given landmark index j, we generate

$$\begin{array}{c}\hfill \{({\left(x_b^{\text{md}}\right)}_{(\eta ,{\left({\mathfrak{R}}_{◇}^{\left(i\right)}\right)}^j)}^{\left(i\right)},{\left({\mathfrak{R}}^{\left(i\right)}\right)}^j):i=1,\cdots ,N_p\}\end{array}$$ (11)

The 3D CNN is trained by the dataset in the following sense:

$$\begin{array}{c}\hfill f_j^{\text{md}}=\underset{f_j^{\text{md}}}{\text{argmin}}\sum _i{\Vert f_j^{\text{md}}\left({\left(x_b^{\text{md}}\right)}_{(\eta ,{\left({\mathfrak{R}}_{◇}^{\left(i\right)}\right)}^j)}^{\left(i\right)}\right)-{\left({\mathfrak{R}}^{\left(i\right)}\right)}^j\Vert }_{{\ell }^2}^2\end{array}$$ (12)

In practice, data augmentation using translation and horizontal flip is applied. The architecture of the 3D CNN is a modified version of VGGNet [18], which is described in Fig 4.

Fig 4. Mandibular landmarks detection.

Patch-based 3D CNN is applied to the segmented image of the mandible (separated from the maxilla), in order to capture 3D morphological features of mandible associated with the landmarks. For six landmarks on condyle, we detect them all at once, instead of one by one, because they are positionally related to each other.

In practice, several landmarks are identified in a group at once. We simultaneously identify six landmarks on the condyle (COR, MCP, LCP, Cp, Ct-in, and Ct-out) that are positionally related to one another, as well as landmarks with bilaterality (e.g. left/right mandibular foramen) that are associated with the symmetric structure of the mandible. For this group detection, we construct a 3D CNN to produce a concatenated vector of all landmark positions on the same group from one 3D image patch. Here, COR is the abbreviation of coronoid point, defined by the anterior process of the superior border of the ramus of the mandible. (i) MCP, (ii) LCP, (iii) Cp, (iv) Ct-in, and (v) Ct-out are the abbreviations of (i) medial condylar point, (ii) lateral condylar point, (iii) posterior condylar point, (iv) medial temporal condylar point, and (v) lateral temporal condylar point.

2.3.3 Detection of cranial landmarks

Landmarks on the cranium that demonstrates rigidity have less variability between subjects. According to [13], cranial landmarks have smaller variance compared to mandibular landmarks with the normalization presented in Section 2.1. Moreover, our empirical experiment shown in Fig 7 demonstrates that the rough local-to-global estimation achieved using the VAE-based low dimensional representation provides more accurate annotations for cranial landmarks. Therefore, we again utilize a VAE-based low dimensional representation in the same manner as in Section 2.2 by using only the cranial landmarks ${\mathfrak{R}}^{\text{cr}}$. To increase the detection accuracy, we enrich the partial knowledge of ${\mathfrak{R}}^{\text{cr}}$ by accurately detecting three additional cranial landmarks lying near the midsagittal plane (MxDML, Od, and PNS). Here, (i) MxDML, (ii) Od, (iii) PNS are the abbrevation of (i) maxillary dental midline, (ii) odontoid process, and (iii) posterior nasal spine, which are defined by (i) the midsagittal line and point of the maxillary central incisors, usually defined by the junctional line and point of right and left incisal edge and medial surface on maxillary central incisors, (ii) a protuberance (process or projection) of the axis (second cervical vertebra), and (iii) a part of the horizontal plate of the palatine bone of the skull. The overall process is illustrated in Fig 5.

Fig 7. VAE-based local-to-global estimation errors (mm) of 90 landmarks over 9 test data.

From 10 reference landmarks obtained in the first step, 90 landmarks are roughly estimated. Blue and brown dots denote errors for the cranial and mandibular landmarks, respectively. Red, blue, and brown lines represent average error over all, cranial, and mandibular landmarks.

Fig 5. 3D cranial landmark detection using VAE-based low dimensional representation combined with easy-to-find landmarks.

Here, the entire cranial landmarks ${\mathfrak{R}}^{\text{cr}}$ are estimated directly from the knowledge of the reference landmarks ${\mathfrak{R}}_{♯}^{\text{cr}}$ and three landmarks ${\mathfrak{R}}_{♯}^{\text{mid}}$ on midsagittal plane that are obtained by 2D CNN.

First, we compute a partially integrated image x^mid from ${\mathit{x}}_b^{\text{cr}}$ using (2) so that the center of the truncated volume of ${\mathit{x}}_b^{\text{cr}}$ lies on the midsagittal plane. Next, a 2D patch ${\left({\mathit{x}}^{\text{mid}}\right)}_{(\eta ,{\mathfrak{R}}_{◇}^j{|}_{v_2,v_3})}$ is extracted, which is defined by a square with edge length of η and center of ${\mathfrak{R}}_{◇}^j{|}_{v_2,v_3}$. Here, ${\mathfrak{R}}_{◇}^j{|}_{v_2,v_3}$ is a vector eliminating the v₁ component in the ${\mathfrak{R}}_{◇}^j$ and j ∈ {24, 25, 26}. Using a 2D CNN, we learn a function $f_j^{\text{cr}}$ that infers an accurate position of a landmark ${\mathfrak{R}}^j$ in v₂- and v₃-coordinates (${\mathfrak{R}}^j{|}_{v_2,v_3}$) from the 2D image patch ${\left({\mathit{x}}^{\text{mid}}\right)}_{(\eta ,{\mathfrak{R}}_{◇}^j{|}_{v_2,v_3})}$. The landmark position in the v₁-coordinate is determined by the location of the midsagittal plane.

In the similar manner as in (11), the following training dataset is generated:

$$\begin{array}{c}\hfill \{({\left(x^{\text{mid}}\right)}_{(\eta ,{\left({\mathfrak{R}}_{◇}^{\left(i\right)}\right)}^j{|}_{v_2,v_3})}^{\left(i\right)},{\left({\mathfrak{R}}^{\left(i\right)}\right)}^j{|}_{v_2,v_3}):i=1,\cdots ,N_p\}\end{array}$$ (13)

With the training dataset, the 2D CNN is trained as follows:

$$\begin{array}{c}\hfill f_j^{\text{cr}}=\underset{f_j^{\text{cr}}}{\text{argmin}}\sum _i\Vert f_j^{\text{cr}}\left({\left(x^{\text{mid}}\right)}_{(\eta ,{\left({\mathfrak{R}}_{◇}^{\left(i\right)}\right)}^j{|}_{v_2,v_3})}^{\left(i\right)}\right)-{\left({\mathfrak{R}}^{\left(i\right)}\right)}^j{{|}_{v_2,v_3}\Vert }_{{\ell }^2}^2\end{array}$$ (14)

Here, data augmentation using translation is applied. The architecture of the 2D CNN is modified from VGGNet [18], as illustrated in Fig 4.

Let ${\mathfrak{R}}_{♯}^{\text{mid}}$ be a concatenated positional vector with cranial reference landmarks ${\mathfrak{R}}_{♯}^{\text{cr}}$ and three finely detected landmarks obtained by $f_j^{\text{cr}}$. Using this partial knowledge ${\mathfrak{R}}_{♯}^{\text{mid}}$, we find accurate cranial landmark positions $\widetilde{{\mathfrak{R}}_{◇}^{\text{cr}}}$ via

$$\begin{array}{c}\hfill \widetilde{{\mathfrak{R}}_{◇}^{\text{cr}}}={\mathcal{D}}^{\text{cr}}\circ {\Phi }^{\text{cr}}\left({\mathfrak{R}}_{♯}^{\text{mid}}\right)\end{array}$$ (15)

where ${\Phi }^{\text{cr}}:{\mathfrak{R}}_{♯}^{\text{mid}}\mapsto {\mathbf{z}}^{\text{cr}}$ is a nonlinear map and ${\mathcal{D}}^{\text{cr}}:{\mathbf{z}}^{\text{cr}}\mapsto {\mathfrak{R}}^{\text{cr}}$ is a decoder of VAE. Here, ${\mathbf{z}}^{\text{cr}}\in {\mathbb{R}}^{d^{\text{cr}}}$ is a $d^{\text{cr}}$-dimensional latent variable given by ${\mathbf{z}}^{\text{cr}}=\mathcal{E}\left({\mathfrak{R}}^{\text{cr}}\right)$ and ${\mathcal{E}}^{\text{cr}}:{\mathfrak{R}}^{\text{cr}}\mapsto {\mathbf{z}}^{\text{cr}}$ is an encoder of VAE. The maps $({\mathcal{E}}^{\text{cr}},{\mathcal{D}}^{\text{cr}})$ and ${\Phi }^{\text{cr}}$ are trained in the same manner of the method presented in Section 2.2 using cranial landmarks ${\mathfrak{R}}^{\text{cr}}$. The detailed architectures of $({\mathcal{E}}^{\text{cr}},{\mathcal{D}}^{\text{cr}})$ and ${\Phi }^{\text{cr}}$ are illustrated in Fig 5.

3 Result

3.1 Dataset and experimental settings

Our experiment used a dataset containing 24 paired data (multi-detector CT images and landmark data) and 229 anonymized landmark data. This dataset was provided by Yonsei University, Seoul, Korea. The paired dataset was obtained from normal Korean adult volunteers (9 males and 15 females; 24.22±2.91 years old) with skeletal class I occlusion and was approved by the local ethics committee of the Dental College Hospital, Yonsei University (IRB number: 2–2009-0026). All informed consents were obtained from each subject. Among the 24 paired data, we used 15 data pairs for training (i.e., N_p = 15) and 9 data pairs for testing. The anonymized landmark dataset with 3D landmark coordinates was acquired in an excel format from 229 anonymized subjects with dentofacial deformities and malocclusions (i.e., N_l = 229). Manual landmarking for both dataset was performed by one of the authors (S.-H. Lee) who is an expert in 3D cephalometry with more than 20 years of experience.

Our deep learning method was implemented with Pytorch [20] in a computer system with 4 GPUs (GeForce RTX 1080 Ti), two Intel(R) Xeon(R) CPU E5–2630 v4, and 128GB DDR4 RAM. In the training process, the Adam optimizer [21] was consistently adopted, which is known as an effective adaptive gradient descent method. In our experiment, optimal values of all learning parameters (epoch and learning rate) were empirically selected via cross validation. For image-based methods, 15-fold cross validation was applied, where 15 paired training data were split into 1-fold for validation and the others for training. For the training of VAE parts, 5-fold cross-validation was applied to 244 training data (229 unpaired and 15 paired ones). Fold values were empirically selected, depending on the amount of available training data.

The nonlinear map Φ in (10) can be trained by pairs of ${\mathfrak{R}}_{♯}$ and z, where ${\mathfrak{R}}_{♯}$ is an estimated vector of 10 reference landmarks in the first step and z is a latent variable obtained by $\mathit{z}=\mathcal{E}\left(\mathfrak{R}\right)$. Here, $\mathcal{E}$ is the encoder of the pretrained VAE and $\mathfrak{R}$ is the corresponding global landmark. Due to the limited number of CT data, only 15 outputs ${\left\{{\mathfrak{R}}_{♯}^{\left(i\right)}\right\}}_{i=1}^{15}$ were provided in the first-step. Hence, an additional dataset ${\{{\mathfrak{R}}_{♯}^{\left(i\right)},{\mathit{z}}^{\left(i\right)}\}}_{i=16}^{244}$ was generated from the unpaired landmark dataset, where ${\mathfrak{R}}_{♯}^{\left(i\right)}$ and z⁽ⁱ⁾ are given by ${\mathfrak{R}}_{♯}^{\left(i\right)}={\mathcal{S}}_{\text{ub}}\left({\mathfrak{R}}^{\left(i\right)}\right)$ and ${\mathit{z}}^{\left(i\right)}=\mathcal{E}\left({\mathfrak{R}}^{\left(i\right)}\right)$. Here, ${\mathcal{S}}_{\text{ub}}$ denotes a subsampling operator of 10 reference landmarks. This dataset was used in our implementation. Likewise, the map Φ^cr in (15) was trained as the same manner.

For quantitative evaluation, we used mean detection error (MDE) computed as follows: Let ${\left\{{\mathfrak{R}}_{\text{est}}^{\left(i\right)}\right\}}_{i=1}^{N_{eval}}$ be a set of N_eval landmarks output to be evaluated. The MDE is computed by

$$\begin{array}{c}\hfill \text{MDE}={\displaystyle \frac{1}{N_{eval}}}\sum _{i=1}^{N_{eval}}\Vert {\mathfrak{R}}_{\text{est}}^{\left(i\right)}-{\mathfrak{R}}_{\text{label}}^{\left(i\right)}\Vert \end{array}$$ (16)

where ${\mathfrak{R}}_{\text{label}}^{\left(i\right)}$ is the corresponding ground truth for ${\mathfrak{R}}_{\text{est}}^{\left(i\right)}$ and $\Vert {\mathfrak{R}}_{\text{est}}-{\mathfrak{R}}_{\text{label}}\Vert$ is defined by

$$\begin{array}{c}\hfill \Vert {\mathfrak{R}}_{\text{est}}-{\mathfrak{R}}_{\text{label}}\Vert ={\displaystyle \frac{1}{N_{lmk}}}\sum _{j=1}^{N_{lmk}}\Vert {\mathfrak{R}}_{\text{est}}^j-{\mathfrak{R}}_{\text{label}}^j\Vert \end{array}$$ (17)

Here, ${\mathfrak{R}}_{\text{est}}^j$ is the vector corresponding to j-th landmark in ${\mathfrak{R}}_{\text{est}}$, N_lmk is the number of landmarks contained in ${\mathfrak{R}}_{\text{est}}$ (e.g., N_lmk = 90 for entire global landmarks), and $\Vert {\mathfrak{R}}_{\text{est}}^j-{\mathfrak{R}}_{\text{label}}^j\Vert$ is given by

$$\begin{array}{c}\hfill \Vert {\mathfrak{R}}_{\text{est}}^j-{\mathfrak{R}}_{\text{label}}^j\Vert =\sqrt{{\displaystyle \sum _{k=1}^3}({\mathfrak{R}}_{\text{est}}^j{|}_{v_k}-{\mathfrak{R}}_{\text{label}}^j{{|}_{v_k})}^2}\end{array}$$ (18)

where ${\mathfrak{R}}^j{|}_{v_k}$ denotes the v_i-coordinate of ${\mathfrak{R}}^j$.

3.2 Results of reference landmark detection

The detection of the 10 reference landmarks (${\mathfrak{R}}_{\#}$) provided very accurate and robust results (see Table 1 and Fig 6). These results almost meet clinical requirements, while the intra-observer repeatability has a precision of less than 1 mm and the overall median inter-observer precision is approximately 2 mm in the 3D landmarking system [22].

Table 1. Mean detection error for 10 reference landmarks.

Most of the landmarks are annotated almost within clinical requirements.

Landmark	Mean (mm)	Landmark	Mean (mm)
ANS	1.2	Na	1.7
Bregma	1.9	Or (R)	1.3
CFM	2.32	Po (R)	1.63
Or (L)	1.6	MF (L)	1.96
Po (L)	2.21	MF (R)	1.72

Fig 6. Final localization errors (mm) of 90 cephalometric landmarks after coarse-to-find detection over 9 test data.

Blue and brown dots denote errors for cranial and mandibular landmarks, respectively. Red, blue, and brown lines represent average error over all, cranial, and mandibular landmarks.

By using reference landmarks, we normalized the landmark data via uniform scaling by fixing the cranial volume of each subject as the average value of the cranial volume for the training dataset.

3.3 Results of VAE

For the local-to-global detection, the VAE was trained using 45000 epochs, a full batch-size, and a learning rate of 0.001. Here, the full batch-size indicates that our dataset was not divided into several batches in the training process. These learning parameters were empirically chosen by comparing validation errors, which were obtained by varying parameters when training VAEs.

To investigate the effect on the dimension of the latent space, we trained VAE with varying the latent space size. The latent dimension is preferred to be as small as possible compared to that of the vector of reference landmarks (${\mathbb{R}}^{30}$) as well as global landmarks (${\mathbb{R}}^{270}$). Taking this into account, the latent dimension (9) and epochs (45000) were chosen as empirical optimal values based on the validation error. Table 2 shows the variation of the averaged test error for the epoch and latent space dimension. The error tendency for the test set was almost the same as that for the validation set.

Table 2. Mean detector error (mm) of VAE over 9 test data.

This is obtained by varying the number of epochs and latent space dimension.

dim/epoch	35000	40000	45000	50000	55000
3	4.93	5.11	5.20	5.19	5.23
5	4.18	4.29	4.26	4.41	4.51
7	3.23	3.32	3.35	3.41	3.36
9	3.04	3.27	3.06	3.21	3.18
11	3.55	3.34	3.30	3.23	3.27
13	3.09	3.19	3.19	3.10	3.15
15	3.12	3.16	3.14	3.00	3.08

The averaged representation errors of VAE for 9 test data were 2.89 mm, 3.11 mm, and 3.06 mm for the cranial, mandibular, and all landmarks.

3.4 Results of the initial local-to-global detection

The nonlinear map Φ was trained with 5400 epochs, a full batch-size, and a learning rate of 0.0001. For each landmark, Fig 7 shows the performance evaluation achieved using 9 test data with respect to the averaged error in the sense of (17). The mean detection error was 3.27 mm for the cranial landmarks, 3.90 mm for the mandibular landmarks, and 3.59 mm for all landmarks. The error of the cranial landmark estimation was much smaller than that of the mandibular landmark estimation.

The reference landmark detection outputs were selected as the estimation results instead of VAE outputs. This is because the 2D CNN is specially designed to detect the reference landmarks that are placed on a point with a morphologically distinct feature, whereas the VAE-based estimation focuses on capturing global landmark patterns within acceptable tolerance rather than on accurately detecting the specific landmarks. This also applied for the 2D CNN-based cranial landmark detection (in Section 2.3.3).

3.5 Result for coarse-to-fine detection

3.5.1 Mandibular landmark detection

For fine detection of the mandibular landmarks, 3D image patches were extracted with size of 80 × 80 × 80 voxels (≈ 4 × 4 × 4 cm³). To generate the training data in (11), the center location of patch was varied to cover 2 times the maximum error of the initial estimation of ${\mathfrak{R}}_{◇}$ for the training data. Using the parameters of 20000 epochs, a full batch size, and a learning rate of 0.0005, nine 3D CNNs were trained.

Figs 6 and 8(b) show the quantitative and qualitative results of the 3D CNNs. The mean distance error decreased to 2.68 mm from the initial detection error of 3.90 mm. The proposed method achieved an error range of 1 to 4 mm for the detection of most landmarks. In addition, as shown in Fig 10(b), the proposed method significantly reduced the mean and variance of error for the test subjects, compared to the initial detection.

Fig 8. Qualitative evaluation of detection for (a) cranial landmarks and (b) mandibular landmarks.

The red and green dots denote the ground truth and detected output landmarks respectively.

3.5.2 Cranial landmark detection

To generate the partially integrated image x^mid, we set the interval for the truncated volume as ± 7.5 mm v₁-directionally from the midsagittal plane. Next, 2D image patches were cropped into sizes of 80 × 80 pixels (≈ 4 × 4 cm²). For training the 2D CNNs, we used the learning parameters of 5000 epochs, a full batch-size, and a learning rate of 0.0001.

In Fig 9 and Table 3, qualitative and quantitative evaluations of the 2D CNN-based detection of three cranial landmarks on the midsagittal plane are provided. The detection achieved relatively accurate annotation on the three target landmarks.

Fig 9. Results of coarse-to-fine landmark detection on 2D patch.

Yellow dot is the output of the coarsely detected VAE. Green dot is the output of detection using patch-based CNN. Red dot is the ground truth.

Table 3. Error evaluation of the landmarks on the midsagittal plane.

Initial error and 2D CNN error are presented in the table. The errors are reduced after the 2D CNN is applied.

Landmark name	Error before 2D CNN (mm)	Error after 2D CNN (mm)
MxDML	2.81	1.18
Od	3.85	2.65
PNS	2.70	1.49

For the estimation of all cranial landmarks, VAE $({\mathcal{E}}^{\text{cr}},{\mathcal{D}}^{\text{cr}})$ was trained with 80000 epochs, a full batch, and a learning rate of 0.001. The map Φ^cr was trained with 23000 epochs, a full batch-size, and a learning rate of 0.0001. The latent dimension was empirically set to 15.

Figs 6 and 8(a) show the final cranial landmark estimation results in quantitative and qualitative formats, respectively. The mean detection error for all cranial landmarks was 3.08 mm, decreasing from the initial estimation error of 3.27 mm (Fig 10(a)). The error for most cranial landmarks fell within the range of 1 to 4 mm.

Fig 10. The mean detection error of the cranial, mandibular, and all landmarks for each of the 9 test data.

Blue bars represent the error from the initial estimation, red bars represent the error from the final estimation, and the lines represent the averaged error over test subjects. (a) Cranial Landmarks. (b) Mandibular Landmarks. (c) Entire Landmarks.

In terms of all landmarks, as described in Fig 10(c), our proposed method achieved an error of 2.88 mm (Fig 6), which is much lower than the initial detection error of 3.59 mm (Fig 7).

4 Discussion and conclusion

This article proposes a fully automatic landmarking system for 3D cephalometry in 3D CT. The proposed method provides the accurate and reliable identification of cephalometric landmarks that can be used in subsequent clinical studies, such as in the development of morphometrical guidelines for diagnosis, surgical planning, and the treatment of craniofacial diseases. The proposed semi-supervised method is designed to use many anonymized landmark dataset to address the severe shortage of training CT data. Currently, only 24 CT data pairs are available due to the legal and ethical restrictions on medical data, while approximately 200 anonymized landmark data are available.

The proposed method is based on the benchmark model [13], which provides 3.63 mm error for annotation of 90 landmarks. This model motivated the backbone structure of the coarse estimation step. The proposed method reduces the average detection error from 3.63 mm to 2.88 mm by employing the coarse-to-fine detection, where appropriate strategies for mandibular and cranial landmarks were considered for their different properties. We expect the detection accuracy to be further improved with increasing amount of available training data.

While it may be possible to directly learn the map from the partial knowledge to the global landmarks, the use of VAE is an effective approach for obtaining a meaningful latent representation in terms of skull morphology while learning the local-to-global estimation map. Human skull morphology follows certain patterns and the positions of landmarks are closely interrelated. A previous study [13] provided empirical evidence that VAE can learn a low-dimensional representation that is strongly associated with the factors determining facial skeletal morphology. It is also a well-known advantage of VAE that the learned latent space is dense and smooth [13, 14, 23]. Hence, it is expected that the VAE-based local-to-global estimation map not only provides the connection between partial and global landmarks but also follows the merits of VAE for latent representation.

Landmarks on the cranium have smaller variability between subjects compared to those on mandible due to the rigid property of the cranium; therefore, cranial landmarks are relatively suitable for the effective estimation in the sense of finding certain common patterns over the training dataset. Moreover, the use of 3D CNN-based fine annotation for all landmarks requires high computational memory consumption and power budget due to the increased use of 3D networks. Hence, the VAE-based approach can be regarded as an effective strategy to finely detect cranial landmarks with a sufficient level of accuracy. Meanwhile, the positional estimation of the summit position of the cranium (SC) obtained from the relation learned via VAE exhibited the lowest accuracy (see Fig 6). This appears to have occurred because the SC may weakly depend on the positions of other landmarks. A rigorous factor analysis using VAE may be undertaken in future research.

The proposed method has the potential to alleviate the experts’ hectic workflow by introducing an automated cephalometric landmarking with high accuracy. In clinical practice, our method allows all 3D landmarks to be estimated from partial information obtained via 3D CT data. Although the error level of some landmarks does not meet the requirement of clinical applications (less than 2 mm), the proposed method may still aid in decisions of clinicians in determining landmark positions, thereby improving their working processes.

Recently, as concerns about the radiation doses have increased, there have been attempts to use dental cone-beam CT for cephalometric analysis instead of the conventional multi-detector CT because cone-beam CT utilizes a much lower radiation dose than multi-detector CT. The investigation of an automated 3D landmarking system for cone-beam CT will therefore be a topic of our future research.

Supporting information

S1 Table. About 90 cephalometric landmarks.

(ZIP)

Data Availability

There is high concern about the public availability of data which compromises the subject privacy by exposing the potentially identifying information of them. Ethical and legal restrictions were imposed by Korean national law and IRB of Yonsei University Dental Hospital. Data access can be requested from the Corresponding Author (chammyhyun@yonsei.ac.kr), or Institutional Review Board of Yonsei University Dental Hospital (irb@yuhs.ac), by interested researchers who meet the criteria for access to confidential data. For the long-term data storage and availability, data have been being securely stored in four external hard disks at two independent locations.

Funding Statement

This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HI20C0127).