USPoint: Self-Supervised Interest Point Detection and Description for Ultrasound-Probe Motion Estimation During Fine-Adjustment Standard Fetal Plane Finding

Abstract

Ultrasound (US)-probe motion estimation is a fundamental problem in automated standard plane locating during obstetric US diagnosis. Most recent existing recent works employ deep neural network (DNN) to regress the probe motion. However, these deep regressionbased methods leverage the DNN to overfit on the specific training data, which is naturally lack of generalization ability for the clinical application. In this paper, we are back to generalized US feature learning rather than deep parameter regression. We propose a self-supervised learned local detector and descriptor, named USPoint, for US-probe motion estimation during the fine-adjustment phase of fetal plane acquisition. Specifically, a hybrid neural architecture is designed to simultaneously extract a local feature, and further estimate the probe motion. By embedding a differentiable USPoint-based motion estimation inside the proposed network architecture, the USPoint learns the keypoint detector, scores and descriptors from motion error alone, which doesn’t require expensive human-annotation of local features. The two tasks, local feature learning and motion estimation, are jointly learned in a unified framework to enable collaborative learning with the aim of mutual benefit. To the best of our knowledge, it is the first learned local detector and descriptor tailored for the US image. Experimental evaluation on real clinical data demonstrates the resultant performance improvement on feature matching and motion estimation for potential clinical value. A video demo can be found online: https://youtu.be/JGzHuTQVlBs.

1. Introduction

Motivation

Ultrasound (US) scan is an indispensable diagnostic tool in obstetrics care because of its safety, real-time nature and relatively low cost. However, obstetric US scanning is highly experienced-operator dependent. Automatic probe movement prediction for acquiring standard imaging planes might assist less-experienced users to perform scanning confidently. In the case of obstetric US scanning, it is typically not difficult for an operator to find a coarse approximate position for a standard imaging plane (searching); However, locating the probe position to accurately locate a diagnostically acceptable standard plane (finetuning) is not easy. Searching probe motion involves sharply translation and rotation, while fine-tuning probe motion is predominately rotation. Most of the time, probe movements in the fine-adjustment stage are rotation with very small translation. The proposed method focuses on the latter problem.

During the training of the trainee sonographer, the high-quality standard plane is stored in advance captured by expert sonographer. When close to the target plane of interest, it translates to US image-probe motion prediction between the current and target US image-pair. The predicted motion guides the trainee to capture the saved standard/target plane more efficiently to increase their practical scan ability. For the situation without pre-captured high-quality standard plane, the learned local feature can be inserted as a feature extraction frontend with a time-series model, e.g. LSTM and Transformer, for a series of motion predictions. The time-series predicted motions could guide the non-expert to choose the optimal action to approach the standard/target plane.

Related Work

Existing methods of US-probe motion estimation can be grouped into heuristic methods and data-driven deep regression methods. The former inferences the probe motion from elaborate hand-designed heuristics such as speckle decorrelation [2,7,14] extracted from the US image. The latter [6,8,12] takes advantage of the powerful non-linearity of DNN to regress the probe motion from the large-scale US-probe data. These learning-based regression methods leverage the DNN to overfit on the training data of some specific users. As mentioned in [17], the main limitation is naturally the lack of generalization ability in real clinical applications. So our proposed method is back to feature extraction and description. Due to the peculiar data structure, it is difficult to extract local features directly from the US image. Our target is to learn a general US local feature from the large-scale US-probe data recorded by experienced sonographers. The classic local features from computer vision such as SIFT [10], ORB [15] are not tailored for US images result in inferior performance of interest points detection and description on US image due to the domain variance between natural grey-style images and US images. Recently, the deep learned local feature [3,5,16] significantly improve the local feature extraction and matching on the natural images. However, these approaches are difficult to apply to US images due to the lack of geometry information (feature correspondence, known pose and depth) for the network training.

Contribution

Instead of considering local feature extraction and probe motion estimation separately, we present a new way of jointly thinking about these two problems on US images. We present a self-supervised local feature, e.g. USPoint to predict keypoint locations, scores and descriptors for US-probe motion estimation during fine-adjustment standard plane finding. It is achieved by embedding a differentiable USPoint-based motion estimation inside the proposed network architecture, supervised only by automatically obtained ground truth (GT) probe motion information. The reason why we named it self-supervised learning is that similar to the stereo image or audio signals, the position information can be captured by the motion sensor naturally during clinical data collection. As Fig.1 shows, US-probe motion estimation between the current and the target US images is achieved by learned local feature extraction and further deep motion estimation. Compared to conventional, hand-designed heuristics, our method learns priors over geometric transformations and regularities from data directly. Compared to conventional, deep parameter regression, our method is back to generalized US feature learning. In summary, our proposed network includes the following new features: I) a tailored US image-based local detector and descriptor are learned in a self-supervised manner without expensive human-annotation of local feature; II) the probe motion estimation is achieved by USPoint-based regression from the learned local features with attentional graph matches; III) these two tasks are jointly learned in a unified framework to enable collaborative learning for mutual benefit.

Fig. 1. The pipeline of the proposed approach.

2. Methodology

Overview

As Fig.2 shows, the proposed network architecture consists of four stages: feature encoder, local detector and descriptor, attentional graph matching and motion estimation. Stage one, feature encoder is composed of a series of convolution stacks, which transforms the US image to a dense high-dimensional feature representation. Stage two, local detector and descriptor, is a three-stream architecture designed to jointly learn the pixel-wise position and feature vector of interesting points. It extracts the sparse feature representation of detected points from the dense feature representation of stage one. Stage three, attentional graph matching aggregates the feature representation of each interesting point from the other salient points, and further finds the correspondence according to the descriptors. Stage four, motion is estimated via the differentiable singular value decomposition (SVD) from the matched interesting point pairs. During training, the motion sensor (IMU) attached on the probe provides a motion signal as the supervision, while it is not required in the deployment. The complete network is trained in an end-to-end way without expensive human-annotations.

Fig. 2.

The four-stage network architecture: feature encoder, local detector and descriptor, attentional graph matching and motion estimation. The IMU is only used to provide GT for training, while it is not required during inference.

Feature Encoder

The feature encoder is a VGG-style [18] sub-network, which takes an US image as input and generates a dense, low spatial and highdimensional feature representation. Each pixel in the feature map refers to an 8×8 pixel patch in the original US image. This dense feature representation is then fed to the local detector and descriptor to extract the sparse feature representation of each interesting point. The specific architecture and parameter are provided in the Supplementary Material.

Local Detector and Descriptor

As shown in Fig.2, the local detector and descriptor jointly learn the position and feature vector of each interesting point, which is composed of three head-branches for score (middle), detection (top) and description (bottom). The score-branch (middle) includes 2 convolutional layers with 256 and 1 channels followed by a sigmoid activation which bounds score predictions S in the interval (0,1). It regresses the probability of interesting points in each 8×8 pixel patch, so that the top N interesting points can be selected from the original US image according to the scores.

The detection-branch (top) includes 2 convolutional layers with 256 and 2 channels followed by a sigmoid activation. Following [3], it regresses the relative position P_reiative(x,y) of each interesting point in the corresponding 8×8 patch, where x ∈ (0,1) and y ∈ (0,1) after the sigmoid operation. The position/index of each 8×8 patch is defined as P_patch(w,h), where (w,h) is the pixel coordinate value in the feature map. The interesting point position P_point(X,Y ) of each patch in the original US image is calculated from P _reiative(x,y) and P_patch(w,h),

$$P_{point}\left(X,Y\right)=\left(\left(w+x\right)\ast s^n,\left(h+y\right)\ast s^n\right),$$ (1)

where s = 2 is the stride of the pooling layer, and n = 3 is the number of pooling layers in the feature encoder. The descriptor-branch (bottom) includes 2 convolutional layers with 256 and 256 channels respectively, which generates a semi-dense grid of the descriptor for each 8×8 patch.

To achieve a real-time performance and reduce the computational memory, up-sampling back to the full resolution through deconvolution operation is abandoned. Instead, bi-cubic interpolation is used to interpolate descriptors according to the positions of interesting points. The top N interesting points are selected according to their detection scores S from the score branch. And their position (X,Y) and associated descriptor D ∈ R are further extracted by the detection and descriptor brunches respectively. This sub-network transforms dense feature representation to sparse feature representation for the next stage.

Attentional Graph Matching

Because the local feature description of each interesting point is extracted from a small patch in the original US image, the associated feature representation suffers from ambiguities due to lack of contextual cues. Inspired by [16], an Attentional Graph Neural Network (AGNN) [20] is employed to aggregate the contextual cues from the other interesting points to increase the distinctiveness. To integrate both position and appearance information of an interesting point i,i ∈ N into the AGNN, the interesting point position information (X_i,Y_i,S_i) is embedded into a high-dimensional feature representation via a Multilayer Perception (MLP). This feature representation is further combined with the initial visual descriptor D_i to define a new feature vector F_i as,

$$F_i=D_i+MLP\left(X_i,Y_i,S_i\right).$$ (2)

The interesting points in both the source and the target images are considered as nodes of the complete graph. This graph includes two types of undirected edges, intra-image edges and extra-image edges. Intra-image edges connect the current interesting point i with all the other interesting points j in the same image, while extra-image edges connect the current interesting point i with all the other interesting points j in the other image. A message passing formulation [1] M is used for information propagation along both the intra-image and extra-image edges. The high-dimensional feature vector of each graph node is updated as,

$$F_i=F_i+MLP\left(F_i\oplus M_{i,jintra}\right)+MLP\left(F_i\oplus M_{i,jextra}\right),$$ (3)

where ⊕ refers to the concatenation. The intra-image message $M_{i,j}{}^{intra}$ and extraimage message $M_{i,j}{}^{extra}$ are calculated via the self-attention mechanism [19],

$$M_{i,j}^{intra}={\displaystyle \sum _{j\in \left(intra\right)\text{softmax(}}\frac{F_i\cdot F_j^T}{\sqrt{d}})\hspace{0.17em}{F}_j,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{M}_{i,j}^{extra}={\displaystyle \sum _{j\in \left(extra\right)\text{softmax(}}\frac{F_i\cdot F_j^T}{\sqrt{d}})\hspace{0.17em}{F}_j}}.$$ (4)

Here d denotes the dimension of the feature vector. The AGNN can automatically build connections between a current interesting point and nearby or similar salient interesting points, because both position and appearance information are encoded into the feature representation of each graph node.

Further, the pairwise scores γ_i,j are calculated via the similarity of matching the descriptor $F_i^{source}$ and $F_j^{target}$ of the interesting point i and j in the source and target images,

$${\gamma }_{i,j}=\langle F_i^{source},F_j^{target}\rangle ,\hspace{0.17em}\forall \left(i,j\right)\in source\times target,$$ (5)

where $\langle \cdot ,\hspace{0.17em}\cdot \rangle$ refers to inner product. The pair-matching is optimized by the Sinkhorn algorithm [4] to select the matched interesting point pairs.

Motion Estimation

Given the matched correspondence pairs $P_i^{source},P_j^{target}$ with the pairwise score γ_i,j, the 3D relative motion between the source and the target US images is regressed by the differentiable SVD. Firstly, the matched pixel positions P are feed into a Transform Net [13] to obtain a matrix T ∈ R_3×3. Then the pixel coordinate values^source,p_targetj through $P_i^{source}p,\hspace{0.17em}P=j^{target}T$ × are transformed to three-P. T is a dynamic and dimensional values p_i deterministic matrix, which is directly regressed by the Transform Net from data. Their centroid values ^-p^source,p^-target are further calculated as,

$${\overline{p}}^{source}=\frac{1}{N}{\displaystyle \sum _{i=1}^N{p}_i^{source},{\overline{p}}^{t\arg et}=\frac{1}{N}{\displaystyle \sum _{j=1}^N{p}_j^{t\arg et},}}$$ (6)

where N is the number of matched pairs. Then the cross-covariance matrix Σ is given as,

$$\Sigma ={\displaystyle \sum _{i,j=1}^N\left(p_i^{source}-{\overline{p}}^{source}\right){\gamma }_{i,j}{\left(p_j^{t\arg et}-{\overline{p}}^{t\arg et}\right)}^{⊺}}.$$ (7)

Σ is further decomposed by SVD, and then the relative motion between source and target US images is obtained as,

$$U,\Delta ,V^{⊺}=SVD\left(\Sigma \right),\hspace{0.17em}R=V{U}^{⊺},\hspace{0.17em}t={\overline{p}}^{target}-R_{\overline{p}}{}^{source}.$$ (8)

U,V ∈ SO(3) and Δ is the diagonal but potentially signed. R ∈ SO(3) is the rotation matrix, and is further transformed to a quaternion Q. t ∈ R³ is the translation vector.

Crucially, the motion estimation is differentiable, which backpropagates the motion error to the local detector, descriptor and matching sub-networks. This guarantees that the whole network is trainable in an end-to-end fashion. Most importantly, it builds a relationship between the local feature learning and the supervision signal from the IMU sensor, so that the local detector and descriptor are learned in a self-supervised way without expensive human-annotations. Moreover, the learned pairwise scores allow the matched interesting point pairs to contribute differently according to their confidences in the differentiable SVD. Thus, jointly learning the feature and motion enables collaborative learning between two sub-networks during backpropagation to minimise the loss function.

Loss Function

The US image pairs, i.e., source and target US images, are input to the network, and their relative 3D motion, i.e., translation t and quaternion Q, is predicted. The corresponding GT, i.e., translation t_GT and quaternion Q_GT measured by the IMU sensor is provided as the supervision for the network training. The loss function L is defined as,

$$ℒ\left(I^{source},\hspace{0.17em}{I}^{target}\right)=\hspace{0.17em}{\Vert t-t_{GT}\Vert }_2+\lambda \left(1-\left|Q\cdot Q_{GT}\right|\right),$$ (9)

where λ is the scale factor.

3. Experiments

Data Acquisition and Processing

The data used in this paper came from a large-scale multi-modal fetal US dataset Anonymous including synchronized US video and IMU signal. We consider the standard biometry planes of the 2nd and 3rd trimester, i.e. head circumference/trans-ventricular (HC/TV) plane, abdominal circumference (AC) plane and femur length (FL) plane. 47589 US image-pairs extracted from the video clips are then used for the proposed method training and testing, divided into five-fold cross validation. Please find more details on data processing and network training in the Supplementary Material.

Performance Evaluation

Typical learned local detector, descriptor and matches on the US image pairs are shown in Fig.3. The colourized dots and lines refer to the detected interesting points and matched correspondences. The Jet colourmap is used to represent their scores i.e. red refers to a high score while blue refers to a low score. We provide qualitative results on different anatomy planes such as the abdomen, heart, head, arm, foot, lip-nose and spine. Most of the matched correspondences maintain a good degree of global consistency with a few wrong lines crossing. The proposed method consistently predicts a high number of correct matches even within a lot of repeated texture. In order to demonstrate the robustness of learned local feature, the results for US image-pairs within different settings such as scale, crop and zoom settings are provided. We use four standard evaluation metrics: Matching Precision (MP), Matching Score (MS), Positional Drift Percentage (PDP) and Angular Offset (AO) used in the most popular benchmark datasets to analyse the experimental results. According to the metrics definitions, the MP, defined as (correct matches)/(predicted matches) is used for the learned descriptor evaluation, while the MS, defined as (correct matches)/(detected interesting points) is used for the learned detector evaluation. As GT for local feature evaluation, we manually annotated the correct matched correspondences on 300 US image pairs. It is very difficult to annotate the interesting points with matches directly in the US images. Our strategy is to manually select the corrected matches from the predicted matches by the proposed and baseline methods. In this case, manually annotating the interesting points in the US images is not required. For the probe ${\ell }_2\left(\widehat{t}-t_{GT}\right)/{\ell }_2(t_{GT}$ translation evaluation, we use the positional drift percentage, defined as), instead of the absolute positional error. Because the translation is usually very small during the US-probe fine-tuning, a small absolute positional error doesn’t mean a good performance. For the probe rotation evaluation, we use the quaternion rather than Euler angle to represent 3D rotation because the Euler angle has a $arccos\text{(}\widehat{Q}\cdot Q_{GT}$ well-known Gimbal lock problem. Following PoseNet [9], the AO is defined as 2 ). The probe motion GT is directly obtained from the motion sensor (IMU) or US-Simulator. Note that the IMU and US-Simulator are only used to provide GT for network training and performance evaluation, while they are not required in practical inference.

Fig. 3. Typical learned local detector, descriptor and matches for US image pairs.

The color indicates the matching score (Jet colormap, blue=low, red=high). (Color figure online)

As the baseline methods, we chose geometry-style SIFT [10] with FLANN [11], and learning-style SuperPoint [5] with SuperGlue [16] for the local feature matching comparison. We chose deep regression [12] method for the motion estimation comparison between two US images. The only difference is that we only use US image for the motion estimation for a fair comparison due to the proposed method only uses US image during inference. The original method [12] use both US image and IMU signal to predict the motion. From Table1, we can see that the MP and MS of proposed method outperform both the geometry and learning based methods. The main reason is that our proposed local detector and descriptor are tailored for US image while the baselines are designed for natural images. When comparing with the MP and MS of the image-pairs from different anatomy standard planes, we did not find significant differences. From Table2, it can be seen that the average PDP and AO of the proposed method outperform the baseline with different frame-intervals. For deep motion regression, elaborate matched local features provide more geoemtry cues than the original US images.

Table 1. Performance comparison between the proposed method and baselines for local feature matching evaluation.

Method	MP (%)	MS (%)
SIFT [10] + FLANN [11]	45.06	6.58
SuperPoint [5] + SuperGlue [16]	66.57	33.16
Proposed Method	74.72	53.98

MP and MS refer to Matching Precision and Matching Score separately by percentage (%).

Table 2. Performance comparison between the proposed method and baseline for probe motion evaluation.

Method	50-frames		100-frames		150-frames		250-frames
	PDP(%)	AO(°)	PDP(%)	AO(°)	PDP(%)	AO(°)	PDP(%)	AO(°)
Deep regression [12]	8.14	2.54	7.92	4.23	9.21	5.46	10.79	7.98
Proposed method	4.76	1.99	4.83	2.91	5.46	3.23	6.27	5.45

N-frames refers to the interval, i.e., number of frames between the US image pair. The average PDP and AO are evaluated by percentage (%) and degree (°).

4. Conclusions

In this paper, we present a novel approach for jointly local feature extraction and motion estimation during the fine-adjustment stage of standard fetal plane finding in obstetric US scan. A novel local feature USPoint, including interest point detector, score and descriptor, tailored for US image is learned in a selfsupervised manner. The probe motion is further estimated based on the learned local feature detection, description and matching. The collaborative learning between the local feature learning and motion estimation benefits both subnetworks. We will further verify the generalization ability of the USPoint applied in other US-based tasks in the next step. The future work will combine the US landmark retrieval [21] with this USPoint-based motion estimation to achieve a course-to-fine visual-assisted intervention [22] to guide US-probe movement.