Automatic alignment of standard views for transesophageal echocardiographic images

Abstract.

Purpose

3D transesophageal echocardiography (TEE) has become an important modality for pre- and peri-operative imaging of valvular heart disease. TEE can give excellent visualization of valve morphology in 3D rendering. As a convention, 3D TEE images are reformatted in three standard views. We describe a method for automatic calculation of parameters needed to define the standard views from 3D TEE images using no manual input.

Approach

An algorithm was designed to find the center of the mitral valve and the left ventricular outflow tract (OT). These parameters defined the three-chamber view. The problem was modeled as a state estimation problem in which a 3D model was deformed based on shape priors and edge detection using a Kalman filter. This algorithm is capable of running in real time after initialization.

Results

The algorithm was validated by comparing the automatic alignments of 106 TEE images against manually placed landmarks. The median error for determining the mitral valve center was 7.1 mm, and the median error for determining the left ventricular OT orientation was 13.5 deg.

Conclusion

The algorithm is an accurate tool for automating the process of finding standard views for TEE images of the mitral valve.

1. Introduction

Echocardiography is an important modality for cardiac imaging due to its ease of use, low cost, real-time visualization, and lack of ionizing radiation and nephrotoxic contrast compared with CT or x-ray.¹ 3D transesophageal echocardiography (TEE) is a form of ultrasound cardiovascular imaging² in which a probe is passed into the patient’s esophagus and ultrasound images are recorded from there. 3D TEE provides additional benefits for imaging complex valvular pathologies and guide interventions as unobstructed imaging can be done from the esophagus.^3,4

Images taken from esophagus can capture most of the heart and the great vessels and can give additional and more accurate information on the valvular anatomy compared with transthoracic echocardiography (TTE). TEE is, among other things, used to identify mitral valve regurgitation⁵ and has become a widely available and indispensable technique in the operating room,³ where it is recommended for use in adults patients undergoing several forms of interventions, including valve repair and complex endocarditis. 3D TEE is recommended as a reliable and possibly preferred method for measurements related to the aortic annulus by the American Society of Echocardiography and the European Association of Cardiovascular Imaging.⁶

The standard view of cardiac images is the standard positioning of 2D slices through a 3D cardiac image so that the slices are showing the structures of interest to the reader. The heart is automatically aligned and positioned in the slices as a cardiologist has been trained to look at it. Traditionally, moving the 2D planes into the correct position has to be done manually, but automatic reformation of the standard view saves time during examination. There have been several papers on automatic placement of the standard view in both TTE and TEE images. For 2D images, automatic determination of standard views images has been done by Gao et al.⁷ and østvik et al.,⁸ and for 3D images, automatic placement of the standard view has been studied by Orderud et al.⁹ and Lu et al.¹⁰ on TTE images and by Curiale et al.¹¹ on TEE images. The latter work used Hough-transforms to find circular structures and compared that to known estimates of the size and thickness of the mitral and aortic valves.

In this paper, a method for automatically determining the center of the mitral valve and the position of the aorta is described. From these landmarks, the standard view of ultrasound images taken with a TEE probe can be reformatted. This method uses deformable Doo–Sabin surfaces¹² and fits them to the image using a Kalman filter.¹³ It is capable of running in real time.

Doo–Sabin surfaces have also previously been used for cardiac modeling, for instance, by Orderud,¹⁴ Dikici,¹⁵ and Bølviken et al.¹⁶ The Kalman filter is an algorithm for finding unknown variables based both on model-based predictions and measurements and has previously been used for segmentation of cardiac ultrasound images by Smistad and Lindseth¹⁷ and Bersvendsen,¹⁸ among others.

The method in this paper is based on the method used by Orderud et al.⁹ for TTE images, but it is augmented by having it determine good initial values of the Kalman filter to adjust for the large variability in location of the outflow tract (OT) due to the rotation of the TEE probe. The algorithm was applied to 106 ultrasound images for validation. This was compared with manually placed landmarks that marked the mitral valve and aorta.

2. Materials and Methods

2.1. Deformable Model

The method works by fitting a deformable model to an ultrasound image. The model used was a Doo–Sabin model, a generalization of quadratic B-splines originally described by Doo and Sabin.¹² The model was based on the model used by Orderud et al.,⁹ but some changes were made. The model consists of two distinct submodels: one modeling the left ventricle (LV) and the other modeling the left ventricular OT. Figure 1 shows the models.

Fig. 1.

The LV model was composed of a model of (a) the LV cavity and (b) the LV OT. The blue points and lines shows the control nodes.

The aortic and LV models are simultaneously deformed to perform the segmentation of respective structures. The LV model was meant to find the general shape of the LV, including the position of the mitral valve. The OT cylinder determined the position of the aortic valve.

2.2. Kalman Filter

The model was fitted to the image using a Kalman filter process.¹³ The Kalman filter fits a model by optimizing the state vector $x$, which determines the properties of the model. The filter also provides a covariance matrix $P$ as output, giving a measure of the uncertainty of each entry of $x$. $P$’s diagonal entries give the standard deviation of the state vector.

The measurements of the ultrasound image used as input into the Kalman process consisted of edge detection of the current frame. The edge detection was done relative to the surface points in the direction of the surface normal vectors on the model from the prediction step. The edge detector used a step edge algorithm, searching for the biggest change in intensity along the normal vector for 3 cm. The Kalman filter updated the state vector based on the measurements. This was described in greater detail in an article by Orderud et al.⁹ An example of the models described in Sec. 2.1 fitted to a TEE ultrasound image is shown in Fig. 2.

Fig. 2.

An example of the models fitted to a TEE ultrasound image using the algorithm described in Sec. 2.2.

The process was manually tuned on 18 ultrasound images to determine the initialization values for the Kalman filter, with the exception of the global value of rotation around the $y$ axis, which was handled in its own step, as detailed in Sec. 2.3. The tuning images were kept separate from the images used for validation. The input of the algorithm was only the ultrasound image. No user input was used.

Once models were fitted to the image, the placement of the models was used to determine the alignment of the standard view. An example of the finished standard views based on the landmarks from the algorithm is shown in Fig. 3.

Fig. 3.

An example of the result of the fitting algorithm and the resulting standard views. (a) The apical long axis with aorta in view, (b) the two-chamber view, (c) the four-chamber view, and (d) the short-axis view.

2.3. Initialization of Kalman Filter

The Kalman filter’s initial value of rotation in the $x\text{–}z$ plane was determined in its own stage due to the large variation in the true value between files. A preliminary model, identical to the model used in the rest of the algorithm, was rotated to eight different angles. These angles spanned the angles 0 deg to 360 deg, meaning there was 45 deg between them. The rotated models were fitted to the image using a Kalman filter, and the output covariance matrices from the filter were analyzed to find the best fit. The best fitted model was determined using the values of the covariance matrix. The $k$’th element of the diagonal of $P$ is the variance of the $k$’th element of $x$. Adding together the elements on the diagonal corresponds to adding together the uncertainty of each element in the state vector.

The diagonal entries corresponding to the nodes close to the apex of the LV model were excluded, and the weights of the remaining nodes were set to 0.5. Because of the importance of the OT cylinder, the variances of the three states directly related to it were given increased importance and were multiplied by 3. For the global variables, most significance was placed on its rotations as those are most related to proper OT cylinder placement. Thus only those values were used. The formula is

$$s=\sum _{i=1}^3\mathrm{Var}(x_{R_i})+0.5\sum _{i=1,i\in x_A}^{18}\mathrm{Var}(x_{{\mathrm{LV}}_i})+3·\sum _{i=1}^3\mathrm{Var}(x_{{\mathrm{OT}}_i}),$$

where $x_R$ are the indices of entries relating to global rotation, $x_A$ are the indices of entries relating to the apex of the LV, $x_{\mathrm{LV}}$ are the entries relating to the LV, and $x_{\mathrm{OT}}$ are the entries related to the OT. The formula was determined through testing on ultrasound images kept separate from the validation files.

The starting depth was set as 75% of the depth of the center of the image in the case of a full volume acquisition and in the center of the image in the case of zoomed images.

Once initialization was done, the Kalman filter can be run, setting the starting rotation according to the above algorithm.

2.4. Determining Standard Views

Once the model was fitted to the image, landmarks on the model were used to determine the standard view. The long-axis view was determined by landmarks on the LV model. 60 points were placed around the base and used to estimate the center of the mitral valve. Six points were placed evenly on the midwall of the LV and were used to estimate the center of the model. The long axis of the model was set to be the line going through the center of the mitral valve and the center of the model.

Two landmarks were placed on each side of the OT cylinder so that the average of them would be the center of the cylinder. The apical long-axis view was defined as the plane that contains both the long axis and the center of the OT cylinder. The two-chamber and four-chamber planes were set to be 60 deg and 120 deg off from the apical long-axis view, respectively. Finally, the short-axis view should have its center at the estimated center of the mitral valve and should go through the OT cylinder.

An example of the estimated standard views is shown in Fig. 3.

2.5. Evaluation of Images

106 TEE 4D anonymized ultrasound images were used to evaluate the algorithm. All of the images were of the mitral valve, with some showing the entire LV. The images were collected from several institutions under data agreements ensuring compliance with applicable privacy legislations. All ultrasound images used as figures in this paper came from the same data and can be used as such under the same data agreements. All data were provided by GE and were previously acquired for research purposes using GE Vivid Echocardiographic Systems E95 and S70 (GE Vingmed Ultrasound, Horten, Norway) with a 6VT-D TEE probe.

For each of these images, landmarks were placed to mark the mitral valve and aortic valve, and these were used as the ground truth. The landmarks were originally placed as part of a different project done by Andreassen et al.¹⁹ using GE’s 4D AutoMVQ software (EchoPAC SoftwareOnly v204, GE Vingmed Ultrasound, Horten, Norway). Figure 4 shows an example of landmark placement on an image.

Fig. 4.

A long-axis view of one ultrasound image, with landmarks placed to mark the centers of the mitral valve and the aorta. The red dot is the manually placed mitral center, and the blue is the mitral valve calculated by the algorithm. The green is the manually placed aorta, and the yellow is the automatically placed aortic valve center.

Each image was evaluated on two criteria. The first measure was the distance between the estimated center of the mitral valve from the algorithm and the center from the ground truth in 3D space. The second measure used was the angle between three points: the algorithm’s estimate of the aortic valve, the center of the mitral valve from the ground truth, and the aorta placement from the ground truth. In an ideal case in which the two aorta placements are the same, the result should be 0 deg. For angle calculation, the points were projected unto a plane, and depth of the points was not considered. An image showing how the mitral valve/aorta angle error was determined is shown in Fig. 5.

Fig. 5.

A short-axis view of one ultrasound image, with landmarks placed to mark the centers of the mitral valve and the aorta. The red dot is the manually placed mitral center used as ground truth, the green is the manually placed aortic valve center, and the yellow is the automatic aortic valve center. The two lines marks the angle used for evaluation of the aorta error.

3. Results

Evaluating the two measures discussed in Sec. 2.5 across all 106 files, the median error on mitral valve placement was 7.1 mm. The 25th percentile was 4.56 mm, and the 75th percentile was 10.7 mm. The median error on mitral/aorta angle was 13.5 deg, and the 25th and 75th percentiles were 3.85 deg and 28.6 deg, respectively. No manual input was used in the algorithm.

Figures 6 and 7 show histograms of the error distributions of the mitral valve placement and mitral/aorta angle, respectively. Figure 8 shows a scatter plot of the error of each file for both of the error measurements.

Fig. 6.

A histogram of the error in estimating the center of the mitral valve for each of the 106 ultrasound images. Errors are in millimeters and are placed in groups of between 0 and 3 mm, between 3 and 6 mm, and so on.

Fig. 7.

A histogram of the error in estimating the angle of the aorta compared with the mitral valve center for each of the 106 ultrasound images. Errors are in degrees and are placed in groups of between 0 and 15 deg, between 15 and 30 deg, and so on.

Fig. 8.

A scatter plot of the errors for each of the 106 ultrasound images. The $x$ coordinate is given by the mitral valve center error, and the $y$ coordinate is given by the mitral valve/aorta angle error.

Initialization of the Kalman filter as detailed in Sec. 2.3 on average took 2.3 s with a standard deviation of 1.1 s. This only needed to be done once at the beginning of the evaluation. After that, evaluating a single frame took an average of 3.5 ms, with a standard deviation of 1.1 ms.

4. Discussion

The purpose of this work was to develop a method for marking the salient features of a TEE image, primarily for the use of automating the process of finding the standard view of the image. This would shorten the time during examination compared with doing it manually.

This work focused on finding the mitral valve and the aortic valve. Should other parts of the image be of interest, new landmarks could be placed to capture those features. In the future, it could be interesting to expand the uses of the algorithm in this way, for instance, by capturing the position of aortic valve or the entire circumference of the mitral valve.

This work is based on the work done by Orderud et al.,⁹ which was done on TTE images using a Kalman filter without an initialization stage and with a different model. It had a more accurate mean placement of mitral valve and a slightly higher mean error on aorta angle compared with this paper (where the mean values were 8.4 mm and 22.7 deg for the mean mitral valve and aortic errors, respectively). The use of different image types makes a direct comparison of results difficult.

Lu et al.¹⁰ used a supervised learning algorithm on TTE images and had a mitral valve error similar to Orderud, but a higher aortic angle error. Leung et al.²⁰ used sparse registration on TTE images and got better results on both metrics. Neither of these were capable of running in real time, and comparison is again difficult due to the different image types.

Median values were calculated to give an easier comparison with a similar article by Curiale et al.,¹¹ in which Hough transform was used to find the mitral and aortic valves. In that paper, the same median errors were 6.3 mm and 24.8 deg, respectively. Although our method produced a slightly higher error in terms of defining the mitral valve center, there was significant improvement in terms of the mitral valve/aorta angle.

The method in this paper allows for running in real time, with estimates of mitral valve and aorta being updated continuously based on the previous frame. This is true even for continuous changes in probe rotation and position as the Kalman filter can adapt to the changes.

A challenge with using images from the TEE probe is that the probe can be flexed in two directions as well as rotated by the user,² meaning that the angle of the aorta in the image can vary greatly been acquisitions with respect to the axis of the acquisition volume. Segmentation of the aortic root is important for detecting the orientation of the 3D data; only using the LV segmentation would not be sufficient due to LV shape symmetries.

Given the variability in depth of view, the advantages in having a full LV model are uncertain. It is possible that it would be better to cut off the region close to the apex. This might make the model less accurate in the cases in which the entire LV is seen, but it would further shorten computation time. For future use, it might be best to have separate models depending on the depth and field of view. Another direction for further research is to consider other elements of the heart anatomy that could be modeled to increase accuracy, such as the left atrium (LA).

4.1. Limitations

The placement of the ground truth is somewhat subjective, especially in images with a limited field of view or bad quality. Some of the images had high error, and the aorta was not always found by the initialization process. Overall, both error measurements show good results, with improvements in mitral/aorta error over previous research.

The images with the worst mitral valve placement did not necessarily have bad aortic valve placement. An example of poor mitral valve placement is shown in Fig. 9. For this image, the model was fitted well to the aortic outflow valve, but the model had trouble finding parts of the ventricle wall because of the limited field of view, leading to an inaccurate mitral valve center placement.

Fig. 9.

A picture showing the fitting of the model to an ultrasound image in a case in which the mitral valve estimate has a high error. The model still finds the aortic valve, but the model does not find the ventricle wall on the opposite side, leading to a badly placed mitral valve center. MV stands for mitral valve.

An example of poor aortic angle is shown in Fig. 10. In that image, the mitral valve was found, but the aorta is about 90 deg wrong, due to confusion caused by an indent in the LV and LA walls. Regions where the LV is less smooth, as can be seen in that image, can be the cause of confusion for the algorithm.

Fig. 10.

A picture showing the fitting of the model to an ultrasound image in a case in which the aorta angle estimate has a high error. The model is able to place the mitral valve, but it confuses an indent in the left chamber and the LA wall for the aorta.

4.2. Initialization

The Kalman algorithm has difficulty converging to the right solution if the initial state vector is too different from the correct orientation. Determining a good initial state vector is of great importance for good results, and there have been several articles on the issue of initialization of a Kalman filter, for instance, by Linderoth et al.²¹ and Snare et al.²² The latter also used deformable shapes on cardiac images and had several trial models that were applied to the image and then scored based on edge detection to determine which fit the best. It is still considered an open problem.

Due to the above issue, the algorithm had a stage of initialization determining the Kalman’s filter starting angle in terms of rotation around an axis going from the apex to the base of the LV model. This was done based on the values of the covariance matrix that the Kalman filter outputs; the details are described in Sec. 2.3.

When evaluating the rotation, the nodes close to the apex of the LV model were not considered, as those nodes are often not in the image when the TEE image has limited depth and were not considered very relevant to how well the model fits close to the mitral valve or OT.

This method has the disadvantage of extra computation time at startup, but it has the potential to be used for a variety of situations in which the initial state of the Kalman filter is uncertain. Other uses of this could be an avenue of further research. It would be possible to extend the initialization step in this work to test for, for instance, the optimal starting position in addition to starting angle. This would increase the computation time at start-up, and there is a chance of the wrong option being chosen, which happened in some cases in this work. In general, it can be used to search for good starting values for the Kalman filter in which there is a lot of variance, and these uses could be a direction for new research. It is possibly a new contribution to the literature on the initialization problem for the Kalman filter.

5. Conclusions

We introduced an algorithm for determining the mitral valve and left ventricular OT placement in 3D ultrasound TEE images and detailed how this information can be used to construct the standard views. We used a Kalman filter algorithm for segmentation, with a custom stage for determining starting values of the filter. The algorithm got results comparable to or better than previous algorithms on the topic.

Biography

Biographies of the authors are not available.

Disclosures

Several of the authors work at GE Vingmed Ultrasound.