Automated landmarking for palatal shape analysis using geometric deep learning

Introduction

The palate is interesting from both functional and anatomical points of view. It serves crucial roles in mastication and speech^1,2. It is situated at the interface of the oral and nasomaxillary cavities and its shape may be influenced by the different functional demands acting upon it from adjacent structures. The shape of the palate differs significantly among individuals and is influenced by numerous factors, such as developmental stage, mode of breathing, tongue size and posture, malocclusion types and diseases^1–5. It is also altered by orthodontic treatment such as palatal expansion, tooth extraction and tooth inclination^6,7.

Traditionally the palate, and other anatomical structures, have been quantified using landmarks manually annotated by an expert observer. Manual landmarking is laborious and notoriously difficult to standardize. This necessitates that repeat measurements be taken by multiple observers at multiple times to properly account for inter- and intra-observer error. An automatic approach can reduce or, if the method is fully deterministic, wholly eliminate this variation. This has the potential to save time for individual clinicians and facilitate large-scale studies of shape variation.

Laser scanning a cast of a maxillary impression creates a digital 3D representation of the surface as a ‘mesh’, i.e. a cloud of 3D point locations and their connections that together define the surface, which can be subject to further image processing including automatic landmarking. Several methods have been proposed for positioning dental landmarks on digitized dental casts^8,9. The cephalometric software ‘ViewBox’¹⁰ can be used to automatically position 100s of dense ‘semi-landmarks’ over the palatal surface but requires manual indication of fixed landmarks and curves on the specimen¹. A fully automatic approach to landmarking dental casts is, therefore, valuable.

Over the past decade deep-learning methods, employing convolutional neural nets (CNNs), have gradually supplanted methods based on pattern-matching, active shape models, and hand-crafted image features, such as edges¹¹. CNNs have become the state of the art in machine learning for many applications including automatic landmarking of image data^12–15. In broad terms CNNs learn to extract features from the input image that are most useful for a given task. The features of interest are not decided by the algorithm designer, but instead learned from the raw data and abstractly coded in the network. Features are learned in a hierarchical manner, whereby information in the full image is compressed into a smaller image, retaining important information from the full image and so-on, through multiple layers. The way in which the information is compressed through the layers of the network is refined iteratively with the goal of minimizing an application-dependent error metric, such as landmark prediction accuracy on a set of training images. CNNs are well-developed in applications on 2D and 3D volumetric images and have been applied to landmark prediction on computed tomographic images¹² and lateral cephalograms^13–15. Many follow a two-step approach where a coarse prediction is performed initially, followed by a second localized CNN operating on a subsection of the image, identified in the first step. However, the 3D meshes produced by laser-scanning are fundamentally different from the 2D and 3D volumetric images on which CNNs were first developed. The emerging field of geometric deep learning aims to generalize deep learning techniques to 3D meshes, point-clouds and other non-Euclidean data types¹⁶.

Here we investigate geometric deep learning for automatic landmark prediction on the human palate. We employ a PointNet++-based architecture¹⁷ to hierarchically learn features from 3D point-cloud representations of casts of the palate and upper dentition, for landmark localization. Inspired by previous work on lateral cephalograms^13–15, we develop a two-step procedure wherein a first, coarse, prediction of the landmark location is made from points on the complete shape. A second more refined prediction of each landmark is then made in landmark specific networks focusing on a small patch of the image. The accuracy and repeatability of the approach is assessed.

Materials and Methods

Participants

Data were collected as part of the Pittsburgh Orofacial Cleft (POFC) study of participants with cleft lip and cleft palate, unaffected relatives, and unrelated controls. All participants gave informed written consent or consent was obtained from their legal guardian.

All study sites provided local IRB approval, then the University of Pittsburgh Coordinating Center (Marazita and Weinberg, PD/PIs) received approval from the University of Pittsburgh IRB (FWA # 00006790) for the overall project (IRB approval number CR19080127–001).

To focus this initial investigation on normal-range palates, participants with cleft lip, cleft palate or both were excluded as these participants often show marked deviation from normal palatal morphology. The sample comprised only unaffected relatives of affected cases plus unaffected, unrelated controls with no personal nor family history of craniofacial anomalies. To simplify model development and assessment in this initial investigation, we included only participants with complete dentition up to the first molar. After exclusions, 1045 (646 Female) 3D dental casts from individuals of various ages (Median=29.1, IQR=15.8) were included in this study. The sample was ethnically diverse, including self-reported ‘White’ (N=581), ‘Black’ (N=282), ‘Asian’ (N=86) with the remainder being various other or mixed ethnicity (N=96). As is standard practice in artificial intelligence and machine learning, the data were split into training, validation, and testing sets comprising approximately 70% (N=732), 20% (N=209) and 10% (N=104) of the data respectively. Each serve a distinct purpose in network development and assessment (see below “Training and hyper-parameter tuning”).

Dental cast and image acquisition

Maxillary dental impressions were obtained by having each participant bite into a hydrocolloid material. Once set, the resulting mould was filled with plaster to produce a positive cast of the palate and dentition. Each dental cast was scanned using either an R700 3D desktop laser scanner or ‘TRIOS’ 3 Basic intra-oral scanner. Both scanners were manufactured by 3shape (Copenhagen, Denmark). The resulting digitizations were cleaned and cropped using ‘3Shape Ortho Analyzer’ and exported as a 3D mesh. Co-incident points, in the resulting mesh, were then removed using a routine custom-written in MATLAB. The median number of points comprising the resulting 3D meshes was 39 053 (IQR= 18 413). The median of the median sizes of the triangles on each mesh was 0.093mm² (IQR= 0.040mm²).

Landmarking

A single operator (AE) manually indicated landmarks using ‘3dMD Vultus’ software. For this investigation we used the seven landmarks collected as part of the POFC study (Figure 1 and Table 1), comprising two bilateral landmarks at the canines (CL, CR) and first molars (ML, MR) and three midline landmarks at the positions of the canines (CM) and molars (MM) and at the tip of the incisive papilla (IP). To standardize the viewing angle, when indicating the midline landmarks CM and MM, the casts were viewed perpendicular to the anterior-incisal-occlusal plane. Intra-operator error of this landmarking protocol was assessed in Supplementary Text 1. These landmarks, although not exhaustive, serve to measure important, global dimensions of the palate. The canines are situated at a turning point in the dental arch and the distance between them gives a good indication of anterior arch width. The distance between the first permanent molars indicates the maximum transverse width of the dental arch. The three midline landmarks index the anterior-posterior variation in the palate. The superior displacement of CM and MM, relative to the positions of the canine and molar bilateral landmarks, respectively, indicate variation in palatal height.

Figure 1.

The seven landmarks used in this study.

Table 1.

Definition of landmarks used in this study.

Landmark	Definition
Bilateral Landmarks
CL, CR	The deepest point of the gingival crevice on the palatal surface of the canine. If there was gingival recession and the cemento-enamel junction was visible, this was defined as the deepest point of the cemento-enamel junction. If the canines were malpositioned, the tooth in the correct position of the canine within the dental arch was used.
ML, MR	The deepest point of the gingival crevice or the cemento-enamel junction on the palatal surface of the first molar. If there was gingival recession and the cemento-enamel junction was visible, this was defined as the deepest point of the cemento-enamel junction.
Midline Landmarks
IP	The tip of the incisive papilla.
CM	The point on the palatal raphe between CL and CR landmarks, indicated while viewing the casts perpendicular to the anterior-incisal-occlusal plane.
MM	The point on the palatal raphe, between ML and MR landmarks indicated while viewing the casts perpendicular to the anterior-incisal-occlusal plane.

Automatic landmark prediction

Preprocessing

We used the robust principal curvatures algorithm¹⁸, as implemented in MeshLab¹⁹, to calculate the mean curvature at each vertex of the mesh. The mean curvature was then used to remove the base of the dental casts as explained in Supplementary Text 1. Mean curvature and normal direction were retained to be used as features, augmenting the input to the landmark regression model.

Landmark regression model

We followed a two-step approach for the landmark prediction. In the first ‘holistic’ step we predicted a coarse location for all the seven landmarks, based on the input mesh. In the second ‘refinement’ step, the predicted location of a given landmark was refined in one of seven identically structured, but independently trained, networks taking as input a local patch of the mesh around the coarse prediction. An overview of the ‘holistic’ step and the ‘refinement’ step for one landmark is given in Figure 2. For a complete description of the architecture in PointNet++ notation (see Supplementary Text 1). Code is available on GitHub (https://github.com/Bawaw/palatal-landmark-detector). Before feeding the point cloud in the neural networks the preprocessed input mesh was normalized for size and location. Landmarks, curvature and surface normal directions were mean centered and their variances normalized as described in Supplementary Text 1. The input meshes were not normalized for rotation at any stage, but were scanned in an approximately standard orientation (Supplementary Figure 1).

Figure 2.

2-step landmark prediction with learned hierarchical features. The preprocessed mesh is used as input for the holistic landmark detection step (top row). From this mesh 4096 points are sampled uniformly. Each sampled point is augmented with four features (C₁ = 4), see text. The augmented pointcloud is used as input for the neural network. Point cloud and feature representation size is written in the bottom of each rounded rectangle. Features are represented by the color of each point, which is indexed to the mean value of the features. The first ‘holistic’ step (top row) performs hierarchical feature learning followed by a coarse prediction of each landmark (orange points) using a multilayer perceptron. For the refinement step (bottom row), a localised patch is extracted around each coarse landmark from which 2024 points are sampled and augmented as in the holistic step. The augmented pointcloud is fed into 1 of 7 dedicated neural networks. These are structured in the same way as the holistic network and end in a refined prediction of the landmark. The ground truth manual landmarks are indicated as green points in the bottom left rounded rectangle.

The holistic step took as input the preprocessed mesh of a dental cast and the, previously computed, mean curvature and surface normal direction per point, and output initial predictions of the seven landmarks (top row Figure 2). To improve robustness to input meshes comprising different numbers of points 4096 points were sampled on the mesh by randomly sampling a number of points on the faces proportional to the face’s area²⁰. Mean curvature and normal direction for each sampled point was interpolated linearly from the three closest vertices of the input mesh. This scalar mean curvature and three element vector of the normal direction augmented the input point cloud. Thereby, the number of input features channels (C₁) associated with each point is 4. Higher level features were learned hierarchically, using two PointNet++ blocks¹⁷. Each PointNet++ block reduced the point-cloud size by a factor of 2 while increasing the number of feature channels. The resulting points and learned feature vectors were vectorized using global max pooling. These features were then passed to a regression network consisting of a multilayer perceptron, itself comprising two non-linear hidden layers and a linear output layer. This then predicted the normalized locations of all seven landmarks. The normalizations that had been applied to the landmarks to standardize the input to the network were inverted to return the landmarks to the co-ordinate system of the input mesh.

In the refinement step, the initial coarse prediction of a given landmark was refined based on a local region of the original input mesh (bottom Figure 2). It took as input the preprocessed mesh and the coarse prediction of a single landmark and output a second refined prediction of that landmark. A geodesic patch around the coarse prediction of the landmark, of radius 8mm, was taken. On this patch 2048 points were randomly sampled and corresponding features (mean curvature and normal directions) were interpolated as in the holistic step, and fed into a refinement network of similar architecture. The output of this network was the final normalized landmark coordinates. As at the end of the holistic step, the normalizations that had been applied to the landmarks were inverted to return the landmarks to the co-ordinate system of the input mesh. As a final heuristic correction, the landmarks were projected onto the surface of the dental cast. This was done in MATLAB 2021a and is not part of the released network.

Model training and testing

Training was performed on a NVIDIA GeForce RTX 2080 Ti GPU was completed over 2000 epochs, sampling a different point cloud for every mesh at each epoch. Adam optimizer was used, with a learning rate that reduced when improvement plateaued, with a factor of 0.5, with a patience of 20 epochs, starting at 0.01 and with a minimum learning rate of 1e-4. Performance on the validation data set was used to design the neural network for optimal performance. Performance on the testing data set was used to evaluate the final model. Each cast of the test set was fed into the final trained model to predict the landmark co-ordinates.

As the network takes as input a randomly sampled number of points from the input mesh, running the automatic prediction multiple times will give different results. To assess repeat measurement reliability, landmark prediction was performed five times for each participant in the testing set, each time with a different sample of 4096 points as input to the network.

Statistical analysis

Repeat measurement reliability

All statistical analyses were performed in MATLAB 2021a. The variation among the five landmark indications on each cast, for each landmark, was quantified as the root-mean-squared Euclidean distance of the five landmark indications from their mean. The distributions of these statistics were visualized using box plots. Given the very low repeat measurement error observed (see results), a single (the first) automatic landmarking indication is used for all foregoing analyses.

Accuracy

Automatic landmarking error

For each automatically placed landmark, for each cast, the Euclidean distance to its corresponding manual landmark indication was calculated. This distribution of Euclidean distances for each landmark was visualized using box plots. To assess anisotropy within the errors of each landmark, the three principal directions of landmark error were computed. This used an uncentered principal components analysis of the 3D displacements from manual to automatic landmark indications (see Supplementary Text 1). These directions are visualized on a randomly chosen example dental cast.

Size and shape predictors of landmarking errors

Prior to all foregoing analyses, which employ geometric morphometric descriptions of shape and size, all manual and automatic landmarks were scaled and aligned to their common mean configuration by a combined generalized Procrustes analysis of all configurations. This removes non-shape variation induced by differences in translation, rotation or scale²¹. Palatal size was recorded separately and was measured as the root mean squared distance of all seven landmarks to the centroid of the landmarks.

Unusually shaped or sized palates may be more error-prone than more normally shaped or sized palates. To investigate this, the total error over all landmarks was calculated as the square root of the sum of squared errors for each size. This error was correlated with palatal size and the Procrustes distance to the mean landmark configuration²¹. A larger Procrustes distance indicates a more unusual shape, independent of size. The size and Procrustes distance measurements were based on the manual landmarks. Due to visible skewness and outliers in the distributions, Spearman’s rank correlation was used to assess correlations.

Differences in morphometric size and shape descriptors between manual and automatic landmarks.

We investigated whether using the manual and automatic landmarking methods gave substantially different descriptions of the sample of shapes using geometric morphometric methods^21,22. In other words, we tested if manual and automatic landmarks are equivalent and interchangeable in biological shape analyses. We first compared the mean landmark configurations of the automatic to the manual landmarks by a permutation test on the variance explained (R²) by the difference between the two means²³. We investigated the dispersion around the mean by comparing the Procrustes distance of manual and automatic landmark configurations from their corresponding mean configuration using Bland-Altman plots²⁴. We also compared palatal size obtained by the two methods using a Bland-Altman plot.

Results

Repeat measurement reliability and accuracy of automatic landmarking

Figure 3 shows the principal directions of the automatic landmarking errors, as well as box plots of the repeat measurement errors and the errors of the automatic landmarking. In addition Supplementary Figure 2 compares clinically-relevant, landmark-based distances calculated using the manual and automatic landmarks. Repeat measurement error was below 0.3 mm for all landmarks on all casts with median values ~0.1 mm for all landmarks.

Figure 3.

Accuracy (a and c) and repeat-measurement error (b) of the automatic landmarking method. The box plots show the median (red line) and interquartile range (vertical span of the blue box; IQR), whiskers show the range of values not considered outliers. Observations are plotted as outliers if they are more than 1.5*IQR from the median. The principal directions of error are shown on an example mesh in a). Red, blue and green lines are the first, second and third principal directions respectively.

Comparing automatic to manual landmarks (Figure 3), median automatic landmarking errors for all landmarks ranged from 0.78mm (IP) to 1.45mm (MM). The proportion of the 104 landmarking indications for which the error was less than 2 mm ranged from 0.93 (IP) to 0.68 (MM). 2 mm is a commonly used cut-off for clinically acceptable accuracy of landmark detection (e.g.²⁵). Errors were anisotropic (Figure 3a). For landmarks CM and MM on the roof of the palate, the largest error occurred along the palate midline. For landmarks MR and ML on the second molars and landmarks CL and CR on the canines the greatest error occurred in the direction of the gingival margin. For landmark IP the largest error occurred vertically along the incisive papilla.

Size and shape predictors of landmarking errors

Figure 4 shows scatter plots of the total landmarking error for each cast plotted against the palatal size and Procrustes distance from the mean, based on the manual landmarks. Spearman’s rank correlation showed a significant positive association between total error (median=3.47mm; IQR=2.04mm) and palatal size (median=18.02mm, IQR=1.50mm; ρ(102)=.20, p=.044, two-tailed), and between total error and Procrustes distance (median=0.20, IQR=0.10 dimensionless units; ρ(102)=.54; p<.001, two-tailed). Thus, there is a tendency for the larger palates and the more unusually shaped palates to have larger errors in the automatic landmarking than smaller palates or palates closer to the average.

Figure 4.

Total automatic landmarking error as a function of palatal size (a) and Procrustes distance from the mean (b). Red lines plot the linear regression of error onto size or Procrustes distance.

Comparison of morphometric descriptions between the automatic and manual landmarking

There is no evidence of a difference between mean landmark configurations obtained by the manual or automatic landmarking. The difference between means explains only 0.5% of the variation (p=.379). Figure 5 shows Bland-Altman plots comparing Procrustes distance to the mean shape and size, obtained using manual and automatic landmarking. These tests assess agreement between measures, using the two methods of landmarking. The differences between automatic and manual measurements of size (M=−0.135; SD=0.47) and Procrustes distance (M=−0.048, SD=0.054) are significantly different from zero (one-sample t(103) =−2.92, p=.004 two-tails and t(103)=−8.94, p<.001 two tails respectively). In general both measures tend to be underestimated when using the automatic approach.

Figure 5.

Bland-Altman plots comparing palatal size (a) and Procrustes distance to the mean (b) obtained using the manual and automatic landmarking. The differences between the values from the two methods are plotted on the vertical axis, the mean of the two values is plotted along the horizontal axis. Red lines show the location of the mean difference (bias). Blue lines are located at +/−1.96 standard deviations of differences, indicating the 95% limits of agreement.

Discussion

Manual landmarking is laborious, error-prone, and requires a skilled operator with anatomical knowledge. The principle of deep hierarchical feature learning is to learn implicitly from training data, image features that best serve the landmarking. This removes the need for anatomical knowledge on the part of the developer or the user. In this study, we investigated geometric deep learning as a tool for automatic landmarking of the palate using digitized 3D meshes of maxillary dental casts. Our approach shows promising accuracy and excellent repeat-measurement reliability.

In general, automatic landmarking can standardize individual patient assessments and facilitate large-scale studies of anatomical variation by removing the laborious and error-prone task of manual landmarking. This, in turn, can automate the collection of shape descriptors traditionally used in orthodontic clinics and research, including palatal depth and width. Further, it can facilitate more comprehensive approaches that require anatomical landmarks as input. These include: calculating palatal surface area and volume; semi-landmarking algorithms; and surface registration algorithms for establishing corresponding points over the whole surface of an anatomical structure such as the palate^26,27. Previous work has investigated automatic landmarking of the dentition⁸. However, automatic landmarking of the palate using geometric deep learning, or by any other means, has received little attention.

Repeat-measurement reliability was better than for repeated manual landmarking by a single observer (Supplementary Text 1). The proportion of test dental casts for which error was less than 2 mm ranged from 0.93 (IP) 0.68 (MM). The range of error that is acceptable is application-dependent and must be decided by the individual user. However, 2mm is a commonly used threshold for clinically acceptable accuracy²⁵. Accuracy is reduced for palates of unusual shape (large Procrustes distance) and large size.

To test the method’s utility in geometric morphometric analysis, we compared descriptions of sample central tendency (mean shape) and variation (Procrustes distances from the mean) between the two methods, as well as a measure of palatal size. We found no evidence of a difference between mean landmark configurations produced with the two methods. Size and Procrustes distance to the mean tend to be underestimated using the manual approach. Underestimating Procrustes distance indicates that sample variation is artificially reduced. It remains to be investigated whether the variation that is lost is biologically-driven, scientifically-useful variation or noise produced by scanning or manual landmarking error.

The two midline landmarks (CM and MM) showed the highest median errors. These are type 3 landmarks, defined in part by their relationships to the canines and first molars, respectively. These errors may be partly explained by the fact that, in the refinement step of the network, only a localized patch around the landmark is considered while its spatial relationship to the other landmarks is ignored. The two-step approach may therefore be less suitable for identifying landmarks of type 3. In support of this, midline landmark CM is not significantly improved in the refinement step although MM and all other landmarks are (Supplementary Text 1). In addition, assessment of repeat-measurement reliability of the manual landmarking showed that these landmarks are also more difficult for humans to identify (Supplementary Text 1). Consequently, for these landmarks the ‘ground-truth’ is less well defined, which might have complicated the training of the network and somewhat inflated the errors at these landmarks in the testing phase.

In this study, the development, training, and testing of the network were simplified by including only palates of individuals with complete dentition up to the second molar. Therefore, the data may not reflect the true diversity of individuals of interest to orthodontists. In future work the robustness of the algorithm to missing teeth and juvenile dentition should be assessed appropriately. We chose to focus on 7 palatal landmarks that, in our view, well represent overall palatal morphology. However, this selection is not exhaustive. It is worth noting that the general methodology developed in this work can also be applied to other landmarks on the palate, or to other structures represented by 3D point clouds.

The problem of hierarchical feature learning from meshes is not trivial and geometric deep-learning, in general, is a new field even by the standards of rapidly-moving deep-learning. In this initial work we obtained promising results, but there is scope for improvement. One approach is to explore different learning strategies on graphs and manifolds²⁸. Another approach would be to combine mean curvature and normal direction (which serve as a starting point for the feature learning) with other descriptors, one example being the shape index which arguably can discriminate among local shape variations better than the mean curvature²⁹. Another avenue is to incorporate a statistical shape model of landmark variation as a prior. This model can be learned by traditional methods (e.g. principal components analysis) or employing an auto-encoder.

Conclusions

While the proposed method shows promising accuracy and repeat measurement reliability, it is less accurate on unusually shaped (large Procrustes distance), or on large palates relative to more average-shaped or smaller palates. This method could substantially streamline landmarking of digitized dental casts for clinical and research applications. However, in the current form it should not be used uncritically; visual error checking and correction will still be required. There is substantial scope for improvement, particularly as the field of geometric deep learning evolves. Future work should investigate the robustness of the algorithm to missing teeth, juvenile dentition and other avenues to hierarchical feature learning on meshes. Critical to these future endevours are large-scale and multi-center datasets to enhance the learning process.