Ultrasonic image analysis and image-guided interventions

Abstract

The fields of medical image analysis and computer-aided interventions deal with reducing the large volume of digital images (X-ray, computed tomography, magnetic resonance imaging (MRI), positron emission tomography and ultrasound (US)) to more meaningful clinical information using software algorithms. US is a core imaging modality employed in these areas, both in its own right and used in conjunction with the other imaging modalities. It is receiving increased interest owing to the recent introduction of three-dimensional US, significant improvements in US image quality, and better understanding of how to design algorithms which exploit the unique strengths and properties of this real-time imaging modality. This article reviews the current state of art in US image analysis and its application in image-guided interventions. The article concludes by giving a perspective from clinical cardiology which is one of the most advanced areas of clinical application of US image analysis and describing some probable future trends in this important area of ultrasonic imaging research.

1. Introduction

Advances in ultrasonic hardware have transformed the quality of information provided by a medical ultrasound (US) scan today relative to 15–20 years ago. Most hospitals in the Western World now have fully digital two-dimensional US systems in radiology, cardiology and obstetrics departments; many have US systems—particularly for clinical diagnosis in cardiovascular medicine and obstetrics; and portable systems are becoming more widely used, with this trend towards smaller systems likely to continue, particularly driven by healthcare needs and the requirement for low-cost imaging solutions in the developing world. As a result, Zetabytes (10²¹ bytes) of ultrasonic data are acquired every minute around the world and are used to make clinical decisions. US is the first modality of choice in, for instance, obstetrics, breast mass assessment and biopsy guidance, and used extensively in assessment of cardiovascular disease. However, and perhaps not always appreciated by researchers from outside the imaging field, clinicians use medical US to provide information for diagnosis, guidance of treatments or to assess the success of therapy, and not to generate pretty pictures of anatomy. US cannot compete with magnetic resonance imaging (MRI) and X-ray computed tomography (CT) in this respect. However, US is certainly not inferior to MRI and CT—it is just different. For instance, it is the only imaging modality capable of imaging soft tissue deformations quickly enough for interventional procedure guidance—information that fluoroscopy, CT or MRI cannot provide. Further, its portability and low cost relative to its expensive alternatives are significant positives that mean US has a very exciting future ahead.

This review focuses on the impact of US on medical image analysis and image-guided interventions. These fields deal with reducing the large volume of digital images (X-ray, CT, MRI, positron emission tomography and US) to more meaningful clinical information using software algorithms. Their role in clinical workflow and decision making is increasing. These research fields are also quite young—around 20 years old—with their origins in computer vision, video image processing and robotics. The fields recently accelerated their clinical relevance and acceptance due to increased computational power of desktop computers, and the maturity of certain classes of method that delineate anatomical boundaries (image segmentation), align images of the same or different imaging modalities (image registration), perform automated quantification and provide real-time processing solutions suitable for interventional use. This review article outlines the state of the art in US image analysis and US-based image interventions illustrated by some success stories and on-going work by the authors and others. It concludes by taking a forward look at possible future directions in this field.

2. Image analysis

Working with US imaging as an information source has positives and negatives.

On the positive side, US is a relative cheap imaging modality compared with MRI and CT, it is portable and it offers a real-time acquisition capability. Principal negatives are that acquisition is operator-dependent and (often significant) training is required to acquire good data. In addition, the presence of speckle—the coherent interference pattern that gives US images their characteristic granular-looking pattern—complicates analysis (but can be a strength, see below). Therefore, the challenge is to work with its strengths, and to try to overcome its limitations by embedding prior knowledge of image formation and the object of interest (geometry, expected image appearance, etc.), and to use carefully designed acquisition protocols wherever possible to minimize operator-dependency.

US imaging does not share the high anatomical definition of CT and MRI. Image appearance is dependent on factors including orientation of the probe with respect to the object, signal attenuation and missing information due to signal dropouts and acoustic shadows. These offer unique challenges to US image processing and analysis relative to CT and MRI analysis. The large majority of the US image analysis literature today is based on B-mode images (also sometimes called display images, or log-compressed US images) in part because this is the most easily available clinical US data offered by commercial systems. Unlike the case of MRI research, which is heavily driven by advances in MR pulse sequences developed by the academic community on MRI research systems, there are few commercial US research machines which provide access to the radio-frequency (RF) or post-processed RF (in-phase quadrature) data. The number of articles on methods that analyse the RF signal is increasing, in part through the maturity of elastography (see separate article) but also recognition of the limitation of B-mode processing alone. Working with B-mode images is more challenging, because the image formation process is nonlinear and difficult to model mathematically. This means that you cannot invoke simple imaging physics models to aid B-mode image processing. Sometimes empirical models of B-model images have been assumed but only for application-specific cases (e.g. [1]).

2.1. Speckle-reduction techniques

Given the variability of image quality a large amount of research effort has gone into attempting to produce images of standard appearance and to reduce speckle. This is both to improve the appearance of images but importantly for image analysis, both intensity inhomogeniety and speckle can challenge intensity-based segmentation methods. Attenuation of US with depth means that a deeper region of the same object or tissue (e.g. a liver) can appear darker than a less deep region. In our own work, for instance, we have used an Expectation–Maximization approach to reduce the effects of attenuation on breast US images [2] and an image surface interpolation method to standardize two-dimensional echocardiographic images while not distorting speckle [3]. Speckle-reduction is a well-studied problem. Here speckle is treated as noise and something that needs to be removed or reduced. At the RF signal level, speckle is often modelled by a Rayleigh distribution but this is not suitable for log-compressed US images. Recent work has shown that a Gamma distribution [1] or a Fisher–Tippett distribution [4] are good approximations for US images. Other work, such as [5] has assumed a general multiplicative model. Adaptive filters [6–8], partial differential equation-based approaches [9,10] and wavelet-based approaches [11,12] to US speckle reduction all appear in the literature for speckle-reduction. Results from one recent approach that adapted the non-local-means filter to speckle noise [13] is shown in figure 1 to illustrate the general effect of a speckle-reduction method. Note, however, that it is not always desirable to remove speckle as it acts as a natural tissue ‘tag’ and moves as tissue is deformed (cf. in many respects to tagged MRI [14]). The presence of speckle is critical to the success of techniques such as strain imaging and speckle tracking [15], and also for many methods of US tissue characterization [16].

Figure 1.

Is speckle noise or a feature? (a) Original B-mode image of a breast mass. (b) Speckle reduction applied to the original image. Note that the texture has been removed from the image but major structures remain well-defined. (c) Elastography image of the same mass derived from RF data. Speckle tracking is an implicit or explicit part of elastography. (Results courtesy of Aymeric Larrue and Andy di Battista, U Oxford UK).

2.2. Image segmentation

Image segmentation is the technical term used to mean manually, semi-automatically or fully automatically delineating the boundaries of an object or tissue regions. Using computer algorithms to assist in US image segmentation is a well-studied problem and there are some good software solutions available, including within commercial packages, although primary use is still within the research community. Readers are referred to the articles in [17,18] for recent reviews of the general field of US segmentation and the case of carotid plaque image segmentation, respectively.

Given that US images do not necessarily define complete object boundaries, the most successful automated image segmentation methods have been those that use prior knowledge of object shape, motion and image appearance. A range of methodologies have thus been developed to embed these constraints, a non-exhaustive list of alternatives including Bayesian approaches [19,20], active contours and active appearance models [21–24], and level-sets [25,26]. An excellent example of this is the work on applying active appearance models applied to echocardiography by Leung et al. [27]. In this approach, image training datasets are used to define the expected shape variation of a typical left ventricle. The shape model is then used to constrain the segmentation solution as the model is fit to a new image example. Figure 2 illustrates this approach.

Figure 2.

Three-dimensional active appearance model segmentation. (a) Average left ventricular object shape and image appearance over a large training set of patients (centre) and the first three most important modes of variation of shape and appearance. (b) Initialization of the model in a three-dimensional dataset; (c) final segmentation result. (Results courtesy of Hans Bosch and Esther Leung, Erasmus MC, Rotterdam NL).

Another way to embed prior knowledge is to take a data-driven or learning-based approach in which image intensity patterns are learned using a classifier to distinguish an object of interest from its surroundings where the classifier has been trained on a large database of example images. This can be viewed as object detection rather than the classic image segmentation problem of object boundary delineation, and used, for instance, to quickly find an image region in which to subsequently manually or automatically take an image measurement. Advanced machine-learning techniques such as AdaBoost, Probabilistic Boosting Trees and Random Forests, and their application to assist US image segmentation are being actively researched [28–30], and we are likely to see more applications of this class of method in the future.

In the case of four-dimensional US imaging (three spatial dimensions and the fourth being time), the problem is that of detection, segmentation and tracking. For instance, to perform fast and accurate tracking of the left ventricle from four-dimensional transthoracic echocardiography data, the algorithm in [31] learns a cardiac motion model using ISOMAP and K-means clustering. This is extended in [32] to automatically quantify the three-dimensional myocardial motion from high frame rate instantaneous US images using learning-based information fusion.

In the case of US segmentation problems that are very difficult to automate, or when a fast segmentation is required, an alternative strategy is to use a semi-automated approach where user-initialization or interaction is used to guide the process of segmentation. Indeed, many of the methods described above in practice are initialized manually (mathematically meaning that the optimization method underpinning the algorithm is sensitive to initialization conditions). Particularly for intensity-based methods, it is a common practice to ‘seed’ an algorithm with manually selected regions-of-interest defining ‘object’ and ‘background’ intensity distributions. This is illustrated in figure 3, which shows semi-automated segmentation of a placenta from a three-dimensional US image using a variant of the graph-based method known as the random walker algorithm [33]. Figure 3a shows a slice from the original three-dimensional US scan. Figure 3b shows the seeding step, where the green areas indicate background voxels, red areas indicate foreground voxels. Figure 3c shows both the boundaries found by the random walker algorithm, and the manually defined boundary independently defined by a clinical expert. Note that in the case shown here, only one object was seeded, but the random walker can be seeded by multiple seeds in general. It can also be automated but was chosen here to illustrate its power as an interactive technique. Related work includes [34] which used the NCut technique combined with anisotropic diffusion for unsupervised US segmentation.

Figure 3.

(a) Image slice through a three-dimensional ultrasound scan of a placenta; (b) seeding for random walker algorithm. (c) Boundary found by automated algorithm (blue) and manual segmentation (yellow). (Result courtesy of Gordon Stevenson, University of Oxford, UK).

2.3. Ultrasound image registration

The field of image registration deals with the alignment of two or more images of the same imaging modality or different imaging modalities. The degree of difficulty in solving a given image registration problem is principally determined by the complexity of the geometric transformation between the two (or more) images (rigid, affine, non-rigid, etc.) and defining the matching criteria between the images, although other factors such as algorithm speed, capture range (sensitivity to initialization) and required accuracy of alignment are also key factors in selecting a method, particularly for real-time diagnosis and image-guided intervention applications. By far, the most popular method to US-based image registration uses the similarity-measure-based registration approach. For US–US registration, as used for instance in motion-tracking, free-hand US volume reconstruction and US mosaicing [35–39], classic similarity-measures such as normalized cross-correlation and normalized mutual information work well, although US-specific similarity measures based on a maximum likelihood framework have also been proposed [40,41].

US-X registration, where X is another imaging modality is more challenging. In this case, the similarity measure needs to capture some functional relationship between the intensities of the different imaging modalities. Unless pre-processing is done, this needs to be as insensitive as possible to US artefacts and insonification angle effects. Some groups have used mutual information [42], the generalized correlation-ratio [43], transformed an US image into a ‘vesselness’ image prior to registration [44] or proposed to define the similarity measure in terms of local-phase representation of each image rather than intensity [45,46]. Figure 4, for instance, shows an example of the method of Zhang et al. [46] applied to echocardiography-MR image registration. Here, the echocardiography image is shown as a green overlay on the corresponding MR image slice. Applications of US-X registration will be considered further in §3.

Figure 4.

Two-dimensional non-rigid alignment of echocardiography and cardiac MRI. (a) Cardiac MRI image, (b) ultrasound image with red, blue and yellow markers indicating points about which local alignment corrections are made at successive iterations of the registration algorithm. (c) The green overlay is the echocardiography image superimposed on the cardiac MR image slice. (Result courtesy of Weiwei Zhang, U. Oxford, UK).

Finally, US–US registration is the first step of fusion three-dimensional echocardiography a technique initially proposed in [47] for improving three-dimensional echocardiographic image segmentation by using multiple three-dimensional US scans, but which has been further improved (e.g. [48–51]), and demonstrated in clinical pilot studies to improve the field of view, anatomical definition and contrast-to-noise ratio relative to single scans for diagnostic purposes [52]. Fusion echocardiography (FE) is a software post-processing solution related to US spatial compounding [53] but it does not involve electronic beam steering of a transducer array (which has a restricted field of view). Rather than compounding (averaging) the intensity values the ‘best estimate of information content’ is taken by considering intensities from all scans. For instance, in figure 5 the wavelet-based fusion method described in [52] has been used.

Figure 5.

(a) Two-dimensional cut planes from a single three-dimensional echocardiography dataset: top left 4-chamber view, top right 2-chamber view, bottom left short axis view, bottom right 4-chamber view. Note the noisy recording with insufficient delineation of the endocardial borders. (b) Two-dimensional cut planes from a three-dimensional dataset where the dataset in figure 5(a) and three others of similar quality were combined together using the image fusion algorithm: top left 4-chamber view, top right 2-chamber view, bottom left short axis view, bottom right 4-chamber view. All cut planes show a clear delineation of the myocardium from the cavity in all segments.

3. Image-guided interventions

Image-guided interventions were traditionally mostly performed under X-ray fluoroscopic imaging. The first such procedures were in fact performed shortly after the invention of X-ray at the end of nineteenth century. With the introduction of US imaging into medicine in the second half of twentieth century, gradually it was also considered for its use in interventional procedures. Because of its non-ionizing characteristic and its cost effectiveness, its popularity has increased considerably in the last decades. US is the modality of choice for many clinical applications. US has also become a common imaging tool to support pain management during various interventional procedures. The large number of applications in which US is regularly used as the main modality for diagnosis and treatment is too large to be reviewed within this article. Here, we rather focus more on the recent medical imaging and computer-assisted intervention techniques, which enable multimodal imaging and propose novel techniques for integrating US into further clinical applications. In spite of its many advantages over other imaging modalities such as X-ray, CT and MRI in term of its real-time nature, its small size and possibility of further miniaturization, its suitability for integration into interventional procedures and its price, rapid increase of its interventional use has been hampered by several characteristics of this imaging modality. One of these is its high user dependency. This property could be considered by experts as an advantage and by trainees as a disadvantage. This results in two issues. First, the training becomes an important subject for interventional use of US imaging. Second, the success of interventional US imaging procedures is highly dependent on the experience level of its user and therefore difficult to assess. In this regard standard US procedures sometimes have difficulties in establishing themselves as an alternative or a replacement for traditional X-ray-, CT- or MRI-based image-guided procedures. This has pushed the scientific community towards many related research directions. On one side, US simulation has become an active subject of research with the clear objective of improving and accelerating the training of practitioners. On the other side, the registration of interventional US and other standard imaging modalities such as CT and MRI has been another focus of research, in this case with the objective of allowing the physician to transfer their diagnostic information and intervention plans into an US-based procedure, which in turn reduces the user dependency on interpretation of interventional US images and helps improve and standardize the procedures. In addition to these points which are the focus of our discussion, there are two other directions of active research in interventional US imaging. The first one is US mosaicing as well as compounding into three-dimensional and four-dimensional volumes. This allows for imaging and visualization of larger areas and volume, once again reducing the need for the physician to compound such dynamic information mentally [54–58]. In the following, we also emphasize the suitability of US as an enabling imaging technology in medical robotics.

Many technical development ideas in image-guided interventions have first been developed and tested for brain surgery applications. It is no surprise that researchers are also looking at using US imaging to improve the performance of image-guided intervention techniques for brain surgical applications. A neuro-navigation system based on a custom-built three-dimensional US was proposed in [59,60]. They claimed that the use of three-dimensional US improves the results by providing real-time imaging feedback. Rasmussen et al. [61] used a commercial three-dimensional US system provided by the company SonoWand, for intra-operative recording of multiple three-dimensional US volumes and navigation through a co-registered pre-operative surgical plan. The registration and data visualization was based on external optical tracking and therefore did not include automatic estimation of brain shift. They, however, presented a post-operative study of a vessel-based co-registration of three-dimensional Power Doppler and segmented vessel trees from magnetic resonance angiography to evaluate its potential for brain shift estimation. Recently, Rasmussen et al. [62] presented an image-based method for registration of MRI to three-dimensional US acquired before the opening of the dura and provided promising results for four cases of brain tumour surgery. The real-time estimation of brain shift using intra-operative two-dimensional and three-dimensional US and its fusion with pre-operative MRI and electrophysiological data for surgical navigation is still an ongoing subject of research and development. We expect to see many exciting development in this domain in the following years.

US is probably one of the most suitable imaging modalities for supporting medical robotics applications. A visual servoing scheme, based on real-time two-dimensional US imaging, was proposed in [63] to control the insertion of a surgical forceps with four degrees of freedom into a beating heart within a minimally invasive procedure. Real-time three-dimensional US was employed in [64] to detect the instrument and track the tissue in order to synchronize the motion of instrument with the heart. Such work could enable advance robotics solution to be deployed in minimally invasive cardiac surgery simplifying the human control of the robot through US-based compensation of fast cardiac motion.

US-assisted cardiac interventional guidance is considered in [65]. The proposed system co-registered patient-specific models obtained from pre-operative data with intra-operative real-time US images within an advanced mixed reality visualization environment also integrating the three-dimensional virtual models of the tracked surgical instruments. The authors demonstrate the high accuracy of the system and its appropriateness for cardiac interventions.

Miniaturization of US sensors [66] has allowed their integration into catheters, endoscopes and laparoscopes. For example, US is systematically used for needle guidance in many procedures. Many systems also offer the visualization of a needle within the US B-mode plane by mechanical construction and pre-calibration of the US detector array and the needle insertion support. The use of such systems in transrectal US (TRUS) as well as endoscopic US has entered into routine practice in many clinical applications.

Since its introduction, intravascular imaging (IVUS) has been considered and used by many interventional cardiologists not only for analysis and characterization of plaques composition [67] but also for navigation during stent placement [68]. This has incited many academic and industrial researchers to focus on many different challenges required for the integration and use of this technology within vascular interventions. In general, IVUS catheters are navigated under X-ray fluoroscopy guidance and the existence of pre-operative CT angiography data in many cases encourages the registration of these modalities. In addition, the integration of IVUS with other intra-vascular modalities such as intravascular optical coherence tomography or the recently introduced intra-vascular near-infrared fluorescence imaging [69] could complement this imaging and increase its successful use for image-guided interventions.

Transesophageal echocardiography (TEE) is routinely used in operating rooms for pre-operative analysis, intra-operative guidance and post-operative assessment of various minimal invasive valve repair and replacement procedures. A robust learning-based framework was introduced in [70] to estimate patient-specific models of the aortic and mitral valves morphology and function from four-dimensional TEE data. See figure 6. This work is extended in [71] by incorporating models of the left ventricle and atrium, to enable comprehensive hemodynamic analysis within the left side of the heart.

Figure 6.

(Top) Patient-specific model of the mitral valve with subvalvular apparatus out of four-dimensional transesophageal echocardiography images. (Bottom) The morphological and functional quantification of the mitral valve, based on the model. Courtesy of Razvan Ionasec, Siemens Corporate Research, Princeton, USA.

The fusion of intra-cardiac three-dimensional echocardiography with C-arm CT for treating atrial fibrillation using catheter ablation is investigated in [72]. In this work, an intracardiac real-time three-dimensional US catheter is presented and used. The authors develop methods for registration and visualization of CT reconstruction obtained by an angiographic C-arm and real-time three-dimensional US data acquired with the proposed US catheter.

Tracked B-mode US has also been frequently used for treatment of different cancers. For instance, different properties of US imaging were used in [73] to define a specific algorithm to co-register CT and US for its use in radiation therapy of head and neck cancer. A novel CT-US registration method was proposed in [74] satisfying both requirements of full automation and real-time computation. The method simultaneously optimized the linear combination of different ultrasonic effects simulated from CT, and the CT-US affine registration parameters. The method is aimed at assisting treatment of cancer in both the liver and kidney. Instead of a global rigid or affine registration of US with CT, Wein et al. [75] presented first results for a full deformable registration using the simulation-based approach on the graphics processing unit to better represent the real situation. However, this still does not achieve the real-time performance required for image-guided applications and could be considered as work in progress.

A recent approach is based on creation of a statistical model from CT or MR and its registration to two-dimensional or three-dimensional US. A statistical deformation model of the femur and pelvis was built in [76] and registered to a set of surface points obtained from intra-operative three-dimensional US data. A statistical shape model of a lumbar spine and interventional three-dimensional US were registered in [77] for image guidance in spinal needle injection. In [78], a patient-specific statistical motion model of the prostate was built from biomechanical simulations of a finite element model of the prostate obtained from pre-operative MR data. This was then registered to intra-operative three-dimensional TRUS images for its use in guiding prostate biopsy and interventions.

Another new trend is the use of tracked nuclear probes to get functional intra-operative information. Surgeons, however, wish to position such functional information within the context of anatomical imaging. The registration of intra-operative and dynamic functional imaging to pre-operative CT or MRI data requires the algorithms to compute complex deformation in real-time. Another alternative offered by two-dimensional and three-dimensional US imaging is the fusion and visualization of co-registered US and functional imaging data (see figure 7). A system for systematic co-registration of tracked nuclear gamma and B-mode US probes was proposed in [79]. The ease of integration of real-time US imaging and different functional imaging systems allows the development of novel techniques [80] and even hardware [81] and their integration into procedures in the near future. The real-time co-registered acquisition of US and functional data will also open the path for deformable registration of interventional functional data to pre-operative CT or MRI data, by taking advantages of US-CT and US-MR registration methods discussed earlier in this paper.

Figure 7.

(a) Augmented reality visualization of tracked US and gamma probe acquisition on a liver phantom with implanted malignant and benign tumours. (b) US imaging display augmented by the gamma information presented as a red circle where the probe is sensing high activity. The count rate of detected activity is provided by the gamma probe and is displayed within the upper right part of the display.

4. Clinical perspective

The driver for image analysis in clinical echocardiography for many years has been the desire to have automated quantification tools to compute measures such as the left ventricular (LV) ejection fraction, and local wall motion and myocardial thickening as indicators of LV dysfunction and ischaemia (e.g. [82]). Many of the advances described in §1 are motivated by this. Although three-dimensional echocardiography-based LV volumes are underestimated with respect to cardiac MR [83] three-dimensional echocardiography still offers a very convenient and cheaper way to assess LV function. Sophisticated commercial software packages for visualizing and manipulating three-dimensional echocardiography now exist, but few semi or fully automated quantification methods have been well-validated or commercialized [84,85] and none are used in routine clinical practice in hospitals. The limiting factor remains the wide variability of quality of patient data with the literature citing between 40 and 60 per cent of all routine clinical data having a ‘good or intermediate acoustic window’ meaning having a good enough quality for an image analysis method to work reliably on it. From a clinical perspective this means that it is not cost-effective to introduce automated image analysis into clinical workflow at the current time.

The technique of fusion three-dimensional echocardiography mentioned in §1 (see also figures 5 and 8) provides one way to improve image quality and the field of view, both limiting factors of current US technology. Here we briefly discuss some of the challenges in clinical three-dimensional echocardiography today from the clinical perspective and suggest how FE may help to overcome some of them. In particular, several aspects of fusion three-dimensional echocardiography make it particularly attractive for clinical echocardiography: besides better image quality and comprehensive datasets, three-dimensional FE has the potential for a major improvement in workflow in the clinical echo laboratory.

Figure 8.

(a) Two-dimensional orthogonal cut planes from a single three-dimensional transoesophageal echocardiography dataset with 9 short-axis cross-sections shown in the lower right quadrant. (b) Equivalent for a fused three-dimensional echocardiography reconstruction clearly indicating superior image quality. (Result courtesy of Daniel Augustine, University of Oxford, UK).

4.1. Improving three-dimensional echocardiography image quality

Over the last 10 years there has been some improvement of three-dimensional echocardiography image quality but still a large proportion of three-dimensional studies are not good enough to be used instead of standard two-dimensional echocardiography. This means that there are parts of the left ventricle and other heart chambers in which the border between the heart muscle and the blood pool cannot be tracked visually or by an automated contour finding system. In the first clinical trial using three-dimensional FE only approximately 70 per cent of three-dimensional non-fused (i.e. native) datasets were graded as having intermediate or good image quality [51]. Note that this study included a high number of healthy younger adult individuals—in a typical clinical echo laboratory one would expect a higher number of patients with insufficient image quality for quantitative analysis. The study went on to show that three-dimensional FE increases the proportion of participants with general image quality graded as good or intermediate quality from 70 per cent with standard three-dimensional echocardiography to 97 per cent with three-dimensional FE. This difference should translate into improved diagnostic potential for even small areas of wall motion abnormality. The improvements in image quality seen with three-dimensional FE are much more impressive than the progress obtained from newer transducers.

Even for the fetal heart getting consistent, shadow free, data remain problematic due to the uncontrollable nature of foetal orientation. The utility of image fusion has been demonstrated in a recent clinical study [86] where it was shown that it reduced the variability of volume estimates by about 50 per cent relative to measurement on a single scan.

The few clinical studies validating fusion three-dimensional echocardiography to-date have focused on the left ventricle. However, there is also a need to improve the border definition of the other heart chambers. In particular, there is a need to improve the display of the right ventricle. Because of the geometry of the right ventricle two-dimensional echocardiography for assessment of volumes and function is challenging and three-dimensional echocardiography has been shown to provide good agreement with cardiac MRI. We hypothesize that three-dimensional FE of the right ventricle will result in a similar improvement as observed in three-dimensional fusion of the left ventricle and are currently performing a study with MRI as the gold standard.

4.2. Three-dimensional datasets including the entire organ

At present abnormally large hearts cannot be displayed in a single three-dimensional echocardiography dataset. Three-dimensional FE provides a comprehensive dataset, in which the sonographer can navigate and follow structures without reviewing different loops which are not aligned. With three-dimensional FE it is possible to maintain the highest volume rate and line density. In particular, three-dimensional TEE suffers from limited field of view due to the limited size of the transducer. The image quality is less an issue because of the proximity of the transducer to the heart. Although three-dimensional TEE has become a valuable tool for guiding cardiac surgery and interventions such as transcutaneous closure of shunts and paravalvular leaks, it is often difficult to assess the exact topography of pathology in separate three-dimensional datasets. The availability of a comprehensive dataset displaying the entire heart would facilitate planning and guiding interventional procedures. Early results on three-dimensional FE applied to TEE are reported in [87], see figure 8.

4.3. Simplified image acquisition: a vision

Echocardiographic imaging requires the skills of an experienced sonographer to find the optimal acoustic window and adjustment of the probe. Unfortunately, often the optimal acoustic window is missed. With three-dimensional FE there is the potential to become less dependent on the sonographer's experience: a first step is using a simple protocol of consecutive acquisitions in the vicinity of where a single dataset is normally acquired and then the fusion process is started. The optimal cut-planes can then be found in the high quality three-dimensional FE reconstruction and analysed by using already available programs such as Qlab (Philips) or four-dimensional LV analysis (TomTec) [88]. With faster implementations of three-dimensional FE, it might be possible to scan and fuse consecutive heart beats in near real-time. Recently, transducers have become available that can acquire datasets within a single heart beat and do not require breath holding. Eventually the image acquisition may only need the simple placement of a new type of probe comparable to putting electrocardiogram patches and then further acquisition is automatic. That would have major financial implications as the cost of a sonographer and the scan time per patient would be greatly diminished.

5. Conclusion

In this article, we have outlined how US has evolved in the fields of medical image analysis and image-guided interventions over the past 15–20 years. In spite of strong competition from MRI and CT in particular, there is no question that US will continue to play a key role in multi-modality clinical decision making in the future and if anything its role will increase as three-dimensional US imaging becomes more widely available as a cost-effective three-dimensional imaging modality. A key barrier to more widespread use of automated analysis methods remains the high level of expertise required to acquire good quality data but as we have seen, providing solutions to this is an active area of research at the current time. Today, US acquisition and image analysis are generally seen as two separate tasks but in the future we can expect to see the boundaries between the two blurred with the ‘intelligence’ of image analysis software driving the advanced capabilities of US machines.