Towards Standard Plane Prediction of Fetal Head Ultrasound with Domain Adaption

Abstract

Fetal Standard Plane (SP) acquisition is a key step in ultrasound based assessment of fetal health. The task detects an ultrasound (US) image with predefined anatomy. However, it requires skill to acquire a good SP in practice, and trainees and occasional users of ultrasound devices can find this challenging. In this work, we consider the task of automatically predicting the fetal head SP from the video approaching the SP. We adopt a domain transfer learning approach that maps the encoded spatial and temporal features of video in the source domain to the spatial representations of the desired SP image in the target domain, together with adversarial training to preserve the quality of the resulting image. Experimental results show that the predicted head plane is plausible and consistent with the anatomical features expected in a real SP. The proposed approach is motivated to support non-experts to find and analyse a trans-ventricular (TV) plane but could also be generalized to other planes, trimesters, and ultrasound imaging tasks for which standard planes are defined.

1. Introduction

Standard plane (SP) acquisition is a routine clinical examination task during obstetric ultrasound (US) scanning. In this study, we consider the trans-ventricular (TV) plane which is the plane used for head circumference (HC) biometry and to assess fetal brain development. The TV standard plane is obtained by live B-mode scanning stopped at a cine-buffer frame with particular anatomical structures that are clearly presented. However, the quality of the retrieved US image is highly dependent on the experience of the sonographer. An inadequate visualisation of any key anatomical structures of the desired plane will typically require a second image capture that causes burden for both the pregnant mother and sonographer, and failure to detect any abnormal structures may lead to misdiagnosis.

To support medical education of trainee sonographers, previous related work has mainly focused on simulating random anatomical views of US from scratch. One branch of work in this field [1, 2] target at searching numerical solutions of the biological tissues in wave space or using physic ray-tracing approaches, which are computationally expensive to use. Another branch has proposed deep learning-based frameworks to simulate ultrasound images from phantoms [3] or to enhance synthetic images [4]. Whilst promising results have been achieved, such models are suitable for training simulation but challenging to deploy in real-world clinical scanning with large anatomical variations. With this observation, [5] proposed to generate anatomy based on human annotations of clinical US and control over variability of the generated fetal head. Liang et al. [6] simulated US images from manually-labeled segmentation maps. There are also a few attempts to predict or synthesize medical images from videos of other modalities. For example, [7] used a spatio-temporal prediction network to outline the lesion area in 4D CT brain perfusion imaging. However, that method is not directly applicable to US prediction due to the large potential changes in the anatomical planes.

In this paper we propose a Domain-Adaptive SP Generator (DASPG) that learns to predict the SP image with the underlying anatomical structures inferred from the spatial and temporal features of the video searching for the SP (defined as SP search video). The challenges faced in designing a solution for this task are: 1) compared to a SP image, SP search video usually contains noticeable artefacts such as motion blur and distortion; 2) many frames in the video are not related to the final SP. To address these obstacles, the key step of DASPG is to translate the raw video features to a clean, standard biometric plane with a domain adaption (DA) module. Domain adaption has been broadly exploited to transform representations across different modalities, such as image-to-video of daily activities [8] and magnetic resonance (MR) imaging to computed tomography (CT) in medical resources [9]. In this paper we pose the problem of predicting the SP image from video as a transfer learning task. Specifically, we regard the video during scanning as the source domain and its resulting SP image as the target domain, and apply domain adaption to align between these two imaging representation domains. To avoid an averaged solution, we separately model the spatial and temporal features of the video input with a U-Net [10] and a temporal convolutional network (TCN) [11], respectively, that keeps the spatial properties of the US image appearance while propagating the temporal dynamics. Furthermore, we add a Generative Adversarial Network (GAN) to our model to create realistic ultrasound textures in the predicted image.

The paper contributions are three-fold: 1) We propose the first model to predict an ultrasound SP from a search video starting at a random position. 2) We show that domain adaption can effectively convert a raw search video to the SP image with clear anatomy. 3) The feasibility of our approach is demonstrated on real-world fetal brain ultrasound of both Anomaly and Growth scans. The predicted plane is realistic with required anatomical structures that can be used as a target image to guide the SP detection.

2. Methodology

The proposed DASPG model predicts the corresponding standard anatomical plane image when given a search video of TV. Different from identifying an observed SP [12, 13], our aim is to generate a standard view to assist inexperienced sonographers. The overall architecture is given in Fig. 1. Let $X={\left\{x_t\right\}}_{t=t_i}^{t_n}$ denotes the input US video sequence (the search video), and y its SP image frame at time T. The proposed generator G learns a mapping X → y that consists of a stepwise spatial encoder E_S and a temporal extractor E_T to separately model the spatial and temporal feature of the search video, a domain adaption module DA to transfer the observed scanning knowledge to the targeted SP representation, and a spatial decoder D_S to synthesize the SP image.

Fig. 1.

The architecture of DASPG training and inference. The inference process is shaded in gray. X^′ and y^′ represent the reconstructions from the input video and SP image, respectively.

2.1. Domain-Adaptive Standard Plane Generator

Since most of the interpretable structures are within the fetal skull, the US image is initially transformed by a pre-trained Spatial-Temporal Network (STN) [14] to discard the surrounding structures before feature encoding. This preprocessing step ensures the area of interest in fetal head structure is located in the center of the image.

The obtained US sequence X is convoluted separately in the space and time dimensions, which is more efficient in preserving image properties than a joint 3D convolution [15]. Specifically, we leverage the contracting path of U-Net to form the 2D spatial encoder E_S. The output E_S(X) aggregated for input length |t_n − t_i| is then fed into a TCN E_T to capture the dynamics of the anatomical context from adjacent time slices. E_T consists of four residual convolution blocks operated along the time channel. Within each block, there are two layers of dilated causal convolution, weight normalization, and nonlinear activation (ReLU) followed by a residual connection at the input and output feature representations. Compared to a recurrent convolutional network (RCN), TCN reduces the computational complexity in sequential modeling with lower memory cost, which is an important design consideration for prediction in real-world clinical scanning.

Domain adaption from US video to SP image

By definition an SP contains defined structures with anatomical meanings which may not exist or be clearly visible in other frames of a search video which are “off plane”. To investigate the implicit correlation between the search video and the searched SP, we exploit transfer learning to adapt the knowledge between them. We first define the US video-based high-level representation E_T (E_S(X)) is from the source domain, and the image-based representation E_S(y) embedded from the shared spatial encoder E_S is from the target domain. As shown in Fig. 1, the domain adaption (DA) module has a residual block [16] with two fully-connected (FC) layers on the flattened feature map to extract the domain expertise from video representation.

The output representation of DA is fed into a spatial decoder D_S to translate the SP image. D_S is formed by the expansive path (decoder) of U-Net without skip connections. Table 1 demonstrates the detailed feature sizes of the above-mentioned modules in generating a 224 × 224 SP image.

Table 1. The output feature shape of each module component in DASPG. B denotes the mini-batch size.

Module	Output Size
ES	B × |tn − ti| × 256 × 14 × 14
ET	B × 256 × 14 × 14
DA	B × 256 × 14 × 14
DS	B × 224 × 224

2.2. Objective Function and Adversarial Training

The DA module is optimized through a domain transfer loss ${\mathcal{L}}_{DA}$ to maximize the cosine similarity between the flattened high-level representations of the US video DA(E_T(E_S(X))) and image E_S(y):

$${\mathcal{L}}_{DA}=1-<\text{DA}\left(E_T\left(E_S\left(X\right)\right)\right),E_S\left(y\right)>$$ (1)

To constrain the spatial encoder-decoder (i.e., E_S and D_S), we reconstruct the search video X and SP image y with ${\mathcal{L}}_1$ loss, respectively:

$${\mathcal{L}}_{rec}=\frac{1}{\left|t_n-t_i\right|}{\displaystyle \sum _{t=t_i}^{t_n}\Vert D_S\left(E_S\left(x_t\right)\right)-x_t\Vert +\Vert D_S\left(E_S\left(y\right)\right)-y\Vert }$$ (2)

The autoencoder structure for reconstruction is shared between video frames and the SP image to recognize the ultrasound-specific spatial semantics. We also regularize the predicted image G(X) with the intensity loss ${\mathcal{L}}_{SP}=\Vert G\left(X\right)-y\Vert$ supervised by the content of the real image y.

As an auxiliary loss, adversarial learning is also employed in the proposed DASPG to increase the realism of the synthesized SP image. We use MobileNetV2 [17] with lightweight depthwise-separable convolutions as the discriminator D to classify between a real SP image y and the predicted SP image G(X), and the adversarial loss ${\mathcal{L}}_{adv}$ is given by:

$${\mathcal{L}}_{adv}={𝔼}_X\text{log}\left(1-D\left(G\left(X\right)\right)\right)+{𝔼}_y\text{log}D\left(y\right)$$ (3)

The overall objective combines all four losses $\mathcal{L}={\mathcal{L}}_{DA}+{\mathcal{L}}_{SP}+{\mathcal{L}}_{adv}+{\mathcal{L}}_{rec}$ with equal weights.

3. Experiments

3.1. Dataset and Implementation Details

The experimental dataset contains 103 routine obstetric videos of the fetal head from the Anomaly scan (within the second trimester) and the Growth scan (within the third trimester). An US video clip is selected within 10s before the cine-buffer-corrected SP and downsampled to 6Hz. The training/test scan split is 74/29. The training scans are augmented by 1) random flipping horizontally or vertically, and 2) randomly selecting 12 consecutive frames (i.e., |t_n − t_i| = 12) as training input. Each test scan is split into four non-overlap clips of 12 frames length to form the test clips for evaluation. For model implementation the kernel size of E_T is set to 2, and the output channels of its 4 layers are 8, 6, 3, and 1. In DA, there are 1,024 hidden units and a ReLU activation between the two FC layers. The whole network is trained for 300 epochs with an AdamW optimizer. The initial learning rate is 1e-3 decayed by 1e-2 every 100 epochs.

3.2. Quantitative Evaluation

Numerically we compare the quality of a generated image with a real SP using Kullback Leibler (KL) divergence and Fréchet SonoNet Distance (FSD) [5]. KL divergence characterizes tissue-specific speckle differences based on the histogram statistics of two US images, and FSD tests the overall quality of US image appearance. Different from Fréchet Inception Distance (FID) [18] designed for natural images, FSD is more effective in measuring the ultrasound-specific image quality by using SonoNet-64 [19] as the image feature extractor [5].

We first evaluate each component of DASPG in Table 2. The baseline model given in the first row is when only considering the GAN-based generator [20] along with the reconstruction loss to constrain the autoencoder. When comparing the KL divergence in the first and second rows, we observe that regularization of image intensity substantially benefits the generator. Furthermore, by adapting the predicted distribution to the target SP, the involvement of the DA module consistently increases the performance (see the results in the bottom row). The reduction in FSD also indicates that DA narrows the image quality gap between the SP searching and capture stages. In terms of architecture, integrally modeling the spatio-temporal characteristics in US video using 3D residual convolutions (R3D) [21] is less stable with a large standard deviation, where spatial features might mix up with temporal features in the SP regression. When comparing the bottom two rows, the dilated convolution in TCN is superior to RCN in modeling the temporal patterns in ultrasound SP searching.

Table 2. Quantitative results of different temporal architectures and losses in terms of KL divergence (↓) and FSD (↓).

Architecture			Loss				KL Divergence	FSD
R3D	2D+RCN	2D+TCN	${\mathcal{L}}_{DA}$	${\mathcal{L}}_{SP}$	${\mathcal{L}}_{adv}$	${\mathcal{L}}_{rec}$
		✓			✓	✓	1.217±0.070	182.24
		✓		✓	✓	✓	0.513±0.086	97.46
		✓	✓	✓		✓	0.265±0.100	83.41
✓			✓	✓	✓	✓	0.368±0.228	93.44
	✓		✓	✓	✓	✓	0.476±0.107	129.96
		✓	✓	✓	✓	✓	0.251±0.101	80.41

3.3. Qualitative Evaluation

Example SP predictions for different models and scans are presented in Fig. 2. The visual appearance of the main anatomical structures, such as the skull, the choroid plexus (CP), midline, and cavum septum pellucidum (CSP) are recognized in predicted TV planes with the individual visibility varying between examples, as shown in Fig. 2(c). Comparing Fig. 2(a) and (b) shows that the intensity loss improves the baseline by preserving the grayscale map and speckle texture of an US image. However, compared to Fig. 2(c), the predicted plane without DA in Fig. 2(b) is blurred and closer in appearance to the last plane in the input scanned video (the rightmost column in Fig. 2). A shift to the target plane using DA (in Fig. 2(c)) helps create a realistic SP with clearer boundaries of the anatomical structures.

Fig. 2. Qualitative performance of the predicted standard plane images.

Here, (a) baseline with only ${\mathcal{L}}_{adv}$ and ${\mathcal{L}}_{rec}$ (b) the proposed generator without DA module (c) the proposed generator with DA module. Note that A1-3 are three examples of Anomaly scans, and G1 is an example of a Growth scan.

In terms of data diversity, while image appearance in the search video is far from the SP (shown in the challenging case of A3), the prediction still correctly estimates the outline of head and the direction of internal anatomical structures. Further, the prediction of G1 shows that the model can generate a realistic result for the third-trimester scan which has higher variability in fetal head appearance. This shows the model has generalizability toward different phases of obstetric scan.

To test how the choice of input video affects prediction quality, in Fig. 3, we compare predicted planes generated by input video clips at different temporal distances from the real SP. Predictably, the quality of the generated plane increases when the input clip is closer to the real SP. The anatomical structures predicted from the scan approaching SP are more clear and more recognizable since the video dynamic becomes stable with the visual content more similar to the standard view.

Fig. 3. The predictions at different observation levels.

4. Conclusion

We have proposed a fetal US SP image predictor (DASPG) with anatomical structures inferred from the input SP search video. Apart from a video-based encoder and an image-based decoder, the main architecture of DASPG is a domain adaption module that translates the encoded features of an SP search video to the SP image representation in target domain. The predicted SP image on TV scan is shown to be anatomically consistent with a real SP image, which can be used as the target image with the expected structures to guide the clinical SP acquisition. As future work, we will apply DASPG on more standard views with clinical evaluations such as anatomical landmark detection and usability study.