Convolutional Bayesian Models for Anatomical Landmarking on Multi-Dimensional Shapes

Abstract

The anatomical landmarking on statistical shape models is widely used in structural and morphometric analyses. The current study focuses on leveraging geometric features to realize an automatic and reliable landmarking. The existing implementations usually rely on classical geometric features and data-driven learning methods. However, such designs often have limitations to specific shape types. Additionally, calculating the features as a standalone step increases the computational cost. In this paper, we propose a convolutional Bayesian model for anatomical landmarking on multi-dimensional shapes. The main idea is to embed the convolutional filtering in a stationary kernel so that the geometric features are efficiently captured and implicitly encoded into the prior knowledge of a Gaussian process. In this way, the posterior inference is geometrically meaningful without entangling with extra features. By using a Gaussian process regression framework and the active learning strategy, our method is flexible and efficient in extracting arbitrary numbers of landmarks. We demonstrate extensive applications on various publicly available datasets, including one brain imaging cohort and three skeletal anatomy datasets. Both the visual and numerical evaluations verify the effectiveness of our method in extracting significant landmarks.

1. Introduction

Statistical shape models are widely used to describe anatomical structures and morphological differences in medical image analysis [4]. The anatomical landmarking (or called saliency detection in some literature) is often implemented as the first step in anatomical shape analyses such as registration [21], segmentation [30, 31] and reconstruction [33]. Hence, inferring reasonable landmarks from shape models is a prerequisite to the success of these applications [32, 23].

The most reliable landmarking is manual labeling by experts. But obviously, this is impossible to be widely implemented. An observation is that a point has a higher likelihood to be a qualified landmark if its geometric features are more significant [5, 7]. For example, as shown in Fig. 1(a)-(c), points on ridges of the teeth and the epiphysis of the bones are more likely to be landmarks than points on flat regions. This insight inspires a thought of acquiring reasonable anatomical landmarks by using quantified geometric features [10]. Most, if not all, of existing methods use existing geometric feature descriptors and take the feature computation as a standalone step before the landmarking. However, this design often shows a feasibility restriction to different dimensional shapes because the classical geometric features are usually limited to certain types of shapes. For example, a weighted Gaussian process landmarking method (W-GP) proposed in [10] uses two curvatures to direct the landmark inference. It works well on surface shapes, but it is ineffective on tetrahedral models because both curvatures have no definitions on interior structures. Another example is the morphometric Gaussian process landmarking method proposed in [8]. This method solves the landmarking on tetrahedral meshes, but is not applicable to surface meshes because of the missing support from interior structures. Furthermore, the calculation of the geometric features increase the computational cost, which impedes applications on large-scale data.

Fig. 1.

shape models used in this paper: (a) Molars; (b) Metatarsals; (c) Distal radii; (a)-(c) are rendered by the normalized Gaussian curvatures. Yellow regions stand for high curvature areas. Landmarks are marked by red spheres. (d) The pial (d-i) and white (d-ii) surfaces of the grey matter. 1500 landmarks and their 50 neighboring vertices are marked in red. (d) A grey matter tetrahedral mesh on (d-i) coronal cutting planes and (d-ii) horizontal cutting planes. The landmarks are colored in green.

In this paper, we introduce a convolutional Bayesian model to address these problems. The inspiration comes from the success of convolutional neural networks (CNN) and the strong reasoning ability of the Gaussian process (GP). The convolutional operation helps to capture significant structural features from different types of vision inputs. The GP is capable of making continuous reasonable inferences from the prior knowledge. This motivates us to integrate them together towards a universal and robust landmarking technique. Our contributions are summarized into two-folds:

• We propose a GP model with a multi-frequency multi-phase periodic diffusion kernel (mmPDK) to solve the problems of anatomical landmarking on multi-dimensional shape models. The multi-frequency multi-phase setting is analogous to convolutional layers with different hyperparameters. Hence, the mmPDK mimics the accumulated results of convolutional operations with different patterns. The utilization of mmPDK guarantees that the high-quality geometric features are encoded into the prior knowledge of a GP, so the posterior inference is geometrically reasonable. Each landmark is determined based on the maximized uncertainty of GP posterior inferences. By using a GP regression framework with an active learning strategy, our method is flexible with demands on arbitrary numbers of landmarks.

• We apply anatomical landmarking to various shape models. We demonstrate extensive applications and experimental results including visualizations of uncertainty maps and landmarks, registrations between surfaces, and classifications with simplified feature space. The results verify the effectiveness of our method. Fig. 1 shows two types of shape models from four datasets used in our work. A reasonable theoretical insight is that our method is feasible to more types of medical image data, such as closed surface meshes and images.

2. Methods

Notations:

Given a shape model $\mathcal{G}=\{\mathcal{V},\mathcal{E},\mathcal{F},\mathcal{T}\}$ in ${\mathbb{R}}^3$, where $\mathcal{V}$ is the indexed vertex set of the size $\mid \mathcal{V}\mid$. $\mathcal{E}$, $\mathcal{F}$ and $\mathcal{T}$ are indexed edge set, face set and tetrahedron set, respectively. $\mathcal{E}$, $\mathcal{F}$ and $\mathcal{T}$ can be empty in some data types. v_n is the n^th vertex. ${\tilde{v}}^n$ is the n^th landmark. t denotes for a temporal variable. Define a zero-mean GP as $\mathcal{G}\mathcal{P}(0,K)$, where K is the kernel. Here, we interchangeably use kernel and covariance function to represent any positive semi-definite matrix. ∥·∥ denotes the distance lag, it can be L1 norm, L2 norm or some other spatial metrics. Some other notations are defined near to the places they are referred.

2.1. Multi-frequency Multi-phase Periodic Diffusion Kernel

The inference ability of a zero-mean GP is uniquely determined by its kernel function. So we start by investigating an effective kernel. In a convolutional neural network, the spatial features of an image is well captured by the patch-wise convolutional operations. This inspires us to construct similar computational structures in a kernel construction. The bridge linking two concepts together is the diffusion process. A diffusion process is a solution to a stochastic differential equation [19]. When the scenario is the heat diffusion problem, the diffusion process stands for a heat distribution at a certain time. The diffusion process of the heat diffusion equation is essentially the result of a convolutional filtering [6]: $f(x,t)={\int }_{\mathcal{X}}{f}_0(x^{\prime })h_t(x,x^{\prime })d{x}^{\prime }=(f_0\ast h_t)(x)$. This theorem has been applied for extracting features on manifold-valued medical shapes [13]. Let T(x, t) denote the heat at position x at time t in region D in ${\mathbb{R}}^d$. Given proper initial boundary conditions, a diffusion process solves this parabolic partial differential equation (PDE) for $T:\frac{\partial T}{\partial t}=\alpha \Delta T+F(t\ge 0)$, where Δ is the Laplace operator; F is a spatial-temporal potential function describing the properties of heat sources, which is also written as the multiplication of a Dirac delta function and a temporal function h(t): F = h(t)δ(x – x′), where constant α is 1. Denoting Green’s function of Δ under the Dirichlet boundary condition as G, a fundamental solution to the above PDE is: $T(x,x^{\prime },t)={\int }_0^{t}G(x,x^{\prime },t-s)h(s)ds$. Green’s function in ${\mathcal{R}}^d$ diffusion problems has the standard form: $G=\frac{e^{-x^2∕4t}}{(4\pi t{)}^{d∕2}}$. A periodic potential gives h(t) = cos(ωt). Then the diffusion process with the standard Green’s function and the periodic potential is: $T={\int }_0^{t}cos\omega (t-s)\frac{e^{-x^2∕4s}}{(4\pi t{)}^{d∕2}}ds$.

The Green’s function of Δ in 3D diffusion problem is a specific realization from the family of functions $f(t)=t^{d-1}{e}^{-\frac{1}{4}at}$ in ${\mathbb{R}}^d$, d =3. The integral Laplace transform of this family of functions is available in Eq. (1). An interesting observation is that the solution of the integral is structurally similar to the popular Matérn kernel:$C(\tau )=\frac{{\sigma }^2}{\Gamma (d)2^{d-1}}(2\sqrt{d}\tau \kappa {)}^d{\mathcal{K}}_d(2\sqrt{d}\tau \kappa )$. Both of them have terms such as the modified Bessel function of the second kind ${\mathcal{K}}_d$ and functions of the same dimension order d. This indicates a solid kernel from Eq. (1).

$${\int }_0^{\infty }{t}^{d-1}{e}^{-\frac{1}{4}at}{e}^{-st}dt=2{\left[(\frac{1}{4}a{)}^{\frac{1}{2}}{s}^{-\frac{1}{2}}\right]}^d{\mathcal{K}}_d(a^{\frac{1}{2}}{s}^{\frac{1}{2}})$$ (1)

However, a practical explicit solution to such an integral is not directly derivable [22]. So we estimate the integral to be the summation of a cosine Fourier transform ${\widehat{f}}_c(\omega )$ and a sine Fourier transform ${\widehat{f}}_s(\omega )$ if t → ∞:

$$\begin{gathered} T=cos(\omega t){\int }_0^{t}cos(\omega s)G(s)ds+sin(\omega t){\int }_0^{t}sin(\omega s)G(s)ds \\ \approx cos(\omega t){\widehat{f}}_c(\omega )+sin(\omega t){\widehat{f}}_s(\omega ) \end{gathered}$$ (2)

The cosine and sine transform is expanded with Euler’s formula from Eq. (1):

$${\widehat{f}}_c(\omega )=2^{-1-d∕2}{\pi }^{-d∕2}\left[(\sqrt{-i\omega }{)}^{d-2}{\Sigma }_1+(\sqrt{i\omega }{)}^{d-2}{\Sigma }_2\right]$$ (3)

$${\widehat{f}}_s(\omega )=-i{2}^{-1-d∕2}{\pi }^{-d∕2}\left[(\sqrt{-i\omega }{)}^{d-2}{\Sigma }_1-(\sqrt{i\omega }{)}^{d-2}{\Sigma }_2\right]$$ (4)

$${\Sigma }_1=(\Vert x\Vert \sqrt{-i\omega }{)}^{1-d∕2}{\mathcal{K}}_{1-d∕2}(\Vert x\Vert \sqrt{-i\omega })$$ (5)

$${\Sigma }_2=(\Vert x\Vert \sqrt{i\omega }{)}^{1-d∕2}{\mathcal{K}}_{1-d∕2}(\Vert x\Vert \sqrt{i\omega })$$ (6)

Substituting Eq. (3)-(6) into Eq. (2):

$$\begin{gathered} T=2^{-1-d∕2}{\pi }^{-d∕2}\left[(\sqrt{-i\omega }{)}^{d-2}{\Sigma }_1+(\sqrt{i\omega }{)}^{d-2}{\Sigma }_2\right]cos(\omega t) \\ -i{2}^{-1-d∕2}{\pi }^{-d∕2}\left[(\sqrt{-i\omega }{)}^{d-2}{\Sigma }_1-(\sqrt{i\omega }{)}^{d-2}{\Sigma }_2\right]sin(\omega t) \end{gathered}$$ (7)

When the data is in ${\mathbb{R}}^3$, d = 3, we get an initial kernel expression from Eq. (7):

$$T(\Vert x\Vert ,\omega ,t)=\frac{1}{4\pi }{e}^{-\Vert x\Vert \sqrt{\frac{1}{2}\omega }}cos(\Vert x\Vert \sqrt{\frac{1}{2}\omega }+\omega t)$$ (8)

For simplicity, we define a frequency term $\lambda =\sqrt{\frac{1}{2}\omega }$ and a phase term ϕ = ωt. Then the initial kernel expression is simplified to be:

$$T(\Vert x\Vert ,\lambda ,\varphi )=\frac{1}{4\pi }{e}^{-\lambda \Vert x\Vert }cos(\lambda \Vert x\Vert +\varphi )$$ (9)

We define the final version of mmPDK as a weighted squared form: K = TWT, where the T and weight matrix W are defined in Eq. (10). The weight matrix W is a diagonal matrix with the absolute sum of each row as the diagonal entry. This design is inspired by the kernel principal component analysis (KPCA) which uses a weighted multiplication of the covariance matrix. We define N frequencies with $\lambda =\sqrt{0.2\pi n}$, n = [1, …, N]. We assign ϕ with the values by dividing [0, $\frac{\pi }{2}$] into N equal line-spaces. T equals to the summation of N different convolutional filterings.

$$T(\Vert x\Vert ,N)=\frac{1}{4\pi N}\sum _{n=1}^N{e}^{-{\lambda }_n\Vert x\Vert }cos({\lambda }_n\Vert x\Vert +{\varphi }_n),W(x)=\sum \mid T(x,\cdot )\mid$$ (10)

The heat kernel (HK) has a similar theoretical background [3]. But HK considers a zero potential while mmPDK is in a dynamic scenario that has more variations. Fig. 2(a) shows five kernels with single frequency and phase, which is analogous to the HK. Fig. 2(b)-(d) show the two to four summations of multi-frequencies and multi-phases kernels. More filtering patterns are observed.

Fig. 2.

Illustrations of periodic diffusion kernels after adding multiple frequencies and phases. (a) original kernels with single frequency and phase. (b)-(d) Multi-frequency and multi-phase Kernels after adding two, three and four sets of frequencies and phases. This is analogous to the results of different convolutional operations.

2.2. Anatomical Landmarking via GP Uncertainty Estimation

The main algorithm for anatomical landmarking is a GP regression process with an active learning strategy. We assume a multivariate Gaussian distribution with mmPDK as the kernel: $\mathcal{G}{\mathcal{P}}_{\mathcal{G}}(0,K)$. Supposing $\mathcal{S}\le \mid \mathcal{V}\mid$ landmarks are needed. For each vertex, we choose $N_{\mathcal{G}}$ neighboring vertices and use their pairwise distance lags to build the kernel. For surface models, we use the fast marching algorithm to calculate the geodesic distance and choose the nearest $N_{\mathcal{G}}$ neighboring vertices. For high dimensional models, we use K-Nearest Neighbor (KNN) algorithm with the Euclidean distance as the spatial metric. The kernel is a sparse matrix. Define the uncertainty map of $\mathcal{G}$ during selecting the s^th landmark as:

$${\Sigma }_{\mathcal{G}}(v)=diag(K{)}_v-K_{v,{\tilde{v}}^{s-1}}^T{K}_{{\tilde{v}}^{s-1},{\tilde{v}}^{s-1}}^{-1}{K}_{v,{\tilde{v}}^{s-1},},s\le \mathcal{S}$$ (11)

$$K_{v,{\tilde{v}}^{s-1}}=\left(\begin{array}{c}\hfill K(v,{\tilde{v}}^1)\hfill \\ \hfill ⋮\hfill \\ \hfill K(v,{\tilde{v}}^{s-1})\hfill \end{array}\right),K_{{\tilde{v}}^{s-1},{\tilde{v}}^{s-1}}=\left(\begin{array}{ccc}\hfill K({\tilde{v}}^1,{\tilde{v}}^1)\hfill & \hfill \cdots \hfill & \hfill K({\tilde{v}}^1,{\tilde{v}}^{s-1})\hfill \\ \hfill ⋮\hfill & \hfill \hfill & \hfill ⋮\hfill \\ \hfill K({\tilde{v}}^{s-1},{\tilde{v}}^1)\hfill & \hfill \cdots \hfill & \hfill K({\tilde{v}}^{s-1},{\tilde{v}}^{s-1})\hfill \end{array}\right)$$ (12)

The uncertainty map is initialized as diag(K). The landmark is the vertex with the largest uncertainty score on the map: $argmax({\Sigma }_{\mathcal{G}})$. From Eq. (11) we can see that the whole process follows an active learning strategy. The uncertainty map is updated by adding the covariance information of the newly-selected landmark to the prior knowledge after each selection. This update keeps happening until $\mathcal{S}$ landmarks are selected. This framework has been successfully implemented in other work [10, 8]. Figure 3 shows the process of selecting the first five landmarks on a pial surface. We render the landmark and its 50 neighboring faces to highlight the region where the landmark comes from. Fig. 1(a) shows the landmarks on skeletal surfaces. The mesh is rendered by normalized Gaussian curvatures. Fig. 1(a) shows 1500 landmarks and their 50 neighbors on pial and white surfaces. Some landmarks in sulci are invisible.

Fig. 3.

A sequential landmarking process. We render the selected vertex and its 100 neighboring vertices in green as the current salient area.

3. Experiments

Common settings:

Wave Kernel Signature (WKS) [2] is used as the vertexwise feature descriptor in registration and classification experiments. Surface-based WKS is computed by regular cotangent form. Tetrahedron-based WKS is computed by using the discretized eigenproblem method in [9]. Python-based GPyTorch [11] and Matlab-based GPML [28] are used as development tools. The tetrahedral mesh is formatted and visualized in the way of TetView [24].

Comparison methods:

(1) Heat kernel GP (HK) ; (2) Spectral mixture kernel GP (SMK) [29]; (3) Periodic kernel GP (PK) [27]; (4) Mesh saliency (MS) [16]. This is a highly cited classical method in saliency detection on meshes; and (5) The W-GP in [10]. The diffusion kernel also has the same theoretic background with mmPDK and HK [15]. But we skip it in this paper because of similar performances with HK. We also test the Matern kernel family [25] but we neither demonstrate the results because the performances are not comparable. The mesh saliency method and W-GP are not applied to three-dimensional models.

3.1. Two-dimensional shape models

The purpose of this set of experiments is to verify the performance of mmPDK on classical 2-dimensional tasks. Implementations and visualizations are easier and more visible on 2-dimensional models. Three datasets released in [5] are used: (i) “Mandibular molars”, or “molar”, contains 116 teeth triangle meshes; (ii) “First metatarsals”, or “metatarsal”, contains 57 models; (iii) Distal radii contains 45 models. One prime reason for employing these datasets is the availability of ground-truth landmarks from experts. The regions of interest in these models are the marginal ridges, teeth crowns and outline contours where distinguishable geometric features are rich in [5, 12]. If taking the normalized Gaussian curvature as the measurement of geometric features, the above regions have high curvatures. Fig. 1(a)-(c) show the landmarking results with mmPDK. The meshes are rendered by the normalized Gaussian curvatures and the landmarks are marked by red spheres. Visually, we can see that the selected vertices are clustered in bright yellow regions which stand for high curvature regions. This demonstration visually proves that mmPDK is capable of capturing significant spatial information and yielding reasonable inferences. Fig. 1(d-i) and (d-ii) demonstrate the landmarking results on pial and white surfaces, which indicate potential in surface-based cortical structure analysis.

Next, we apply the landmarks to a surface registration application for numerically evaluating the significance of the selection. We randomly select 10 pairs of models within each dataset and use the Bounded distortion Gaussian process landmark matching algorithm to do the registration [17, 10]. The continuous Procrustes distance is used to numerically measure the performance of correspondence [1]. Fig. 4 shows a textured mapping of two registrated meshes by using mmPDK as an example. The average Procrustes distances of different methods are: HK 0.14, SMK 0.127, PK 0.142, MS 0.118, W-GP 0.086 and mmPDK 0.084. The Procrustes distance of ground-truth landmarks is 0.081. We see that the result of mmPDK is closer to the ground-truth result.

Fig. 4.

An illustration of the surface registration. The source and target meshes are on the left column. The two textured meshes show the registration result.

3.2. Three-dimensional shape models

In this set of experiments, we verify the performance of mmPDK on 3-dimensional models by applying landmarking to the morphological study of Alzheimer’s disease (AD). Structurally, AD causes abnormal atrophy of the cerebral cortex, which results in a gradually thinner grey matter, i.e. thinner cortical thickness, than the normal aged people [20, 26]. We use 275 structural 3D magnetic resonance imaging (MRI) data of left cerebral hemispheres (88 AD patients and 187 Cognitively Unimpaired (CU) visitors) from the baseline collection of Alzheimer’s Disease Neuroimaging Initiative phase 2 (ADNI2) [18, 14]. We follow the pipeline in [9] to generate the grey matter tetrahedral meshes and solve the discretization eigenproblem. Each mesh contains about 160,000 vertices.

Initially, we demonstrate two visualization results. We randomly choose one mesh and visualize one of its uncertainty maps in Fig. 5. The uncertainty of the mesh is divided into ten classes in the descending order. Then, we compute 5,000 landmarks with mmPDK and visualize them in Fig. 6. For better visual effects, we render the faces that the landmarks belonging to and provide several zoom-in views. In morphological studies, these landmarks stand for positions that are most sensitive to structural changes. The visualization results show that points mainly centralize in ROIs such as the temporal pole and temporal gyrus etc.. Previous studies show that these regions may be closely related to AD judged by their functionalities, clinical diagnosis and morphometry changes [20].

Fig. 5.

Illustrations of uncertainty maps on a grey matter tetrahedral mesh. All the vertices are divided into ten classes. 1 and 10 denote the highest and lowest uncertainty classes, respectively. Two zoom-in regions are provided for more visible demonstrations.

Fig. 6.

Demonstration of landmarks in a grey matter tetrahedral mesh. The landmark is marked in green. Two regions are zoomed in for more visible demonstrations.

Next, we use anatomical landmarking as a manifold learning technique. The general idea is using the WKS features of landmarks to form a simplified feature expression of one subject. The significance of different landmarking methods is numerically reflected by the classification results. The SVM and 10-folds cross-validation are used in classifications. The performance is measured by ac-curacy (ACC), sensitivity (SEN) and specificity (SPE). 275 left cerebral hemisphere structural MRIs including 88 AD patients and 187 Cognitively Unimpaired (CU) visitors are used. We select 1000, 2000 and 3000 landmarks on each subject and concatenate their WKS together to a feature matrix. Considering the landmarks of different subsects are not registered, we compute the principal component of the feature matrix and reshape it to a vector. We take this vector as the final subject-wise expression of features. Table. 1 lists the classification results. It shows that a well-selected set of landmarks may be more discriminative than using global features regarding accuracy. One explanation is that such a massive mesh usually contains redundancies and a representative subset is able to decrease the side effects of redundant information. In general, we decrease the feature dimension from a hundred thousand to thousand and this simplified feature space is statistically representative of the original massive data.

Table 1.

Grey matter atrophy classifications with 1000, 2000 and 3000 landmarks. The global feature is taken as the reference. mmPDK shows a leading performance.

		HK	PK	SMK	mmPDK	Global
1000	ACC	0.950	0.900	0.964	0.971	0.973
	SEN	0.954	0.903	0.947	0.961	0.957
	SPE	0.946	0.897	0.984	0.984	0.993
2000	ACC	0.942	0.900	0.950	0.978	0.973
	SEN	0.955	0.864	0.929	0.970	0.957
	SPE	0.925	0.944	0.977	0.987	0.993
3000	ACC	0.941	0.913	0.950	0.980	0.973
	SEN	0.953	0.930	0.930	0.974	0.957
	SPE	0.928	0.892	0.973	0.988	0.993

Overall, the experiments verify the effectiveness of mmPDK. Our work brings insights on the potential of convolutional Bayesian techniques, or more general advanced Bayesian learning techniques, for effective medical image analysis.