Intelligent navigation to improve obstetrical sonography

Abstract

‘Manual navigation’ by the operator is the standard method used to obtain information from two-dimensional and volumetric sonography. Two-dimensional sonography is highly operator dependent and requires extensive training and expertise to assess fetal anatomy properly. Most of the sonographic examination time is devoted to acquisition of images, while ‘retrieval’ and display of diagnostic planes occurs rapidly (essentially instantaneously). In contrast, volumetric sonography has a rapid acquisition phase, but the retrieval and display of relevant diagnostic planes is often time-consuming, tedious and challenging. We propose the term ‘intelligent navigation’ to refer to a new method of interrogation of a volume dataset whereby identification and selection of key anatomical landmarks allow the system to: 1) generate a geometrical reconstruction of the organ of interest; and 2) automatically navigate, find, extract and display specific diagnostic planes. This is accomplished using operator-independent algorithms that are both predictable and adaptive. Virtual Intelligent Sonographer Assistance (VIS-Assistance®) is a tool that allows operator-independent sonographic navigation and exploration of the surrounding structures in previously identified diagnostic planes. The advantage of intelligent (over manual) navigation in volumetric sonography is the short time required for both acquisition and retrieval and display of diagnostic planes. Intelligent navigation technology automatically realigns the volume, and reorients and standardizes the anatomical position, so that the fetus and the diagnostic planes are consistently displayed in the same manner each time, regardless of the fetal position or the initial orientation. Automatic labeling of anatomical structures, subject orientation and each of the diagnostic planes is also possible. Intelligent navigation technology can operate on conventional computers, and is not dependent on specific ultrasound platforms or on the use of software to perform manual navigation of volume datasets. Diagnostic planes and VIS-Assistance videoclips can be transmitted by telemedicine so that expert consultants can evaluate the images to provide an opinion. The end result is a user-friendly, simple, fast and consistent method of obtaining sonographic images with decreased operator dependency. Intelligent navigation is one approach to improve obstetrical sonography.

INTRODUCTION

Obtaining information from two-dimensional (2D) and volumetric sonography depends on ‘manual navigation’ performed by the operator – this requires knowledge and expertise. For centuries, humans have used a map and a compass to guide travel – a typical example of manual navigation. However, technological developments (i.e. satellites) have allowed the use of a global positioning system (GPS) navigation to guide effortlessly toward a specific destination with few steps and minimal operator dependency. We propose that the same conceptual framework which underpins GPS navigation can be applied to ultrasound, and we term its application ‘intelligent navigation’. Intelligent navigation refers to the systematic examination of volumetric data and the efficient extraction of diagnostic information. This article describes the principles involved with intelligent navigation in ultrasound. It is intended to serve as an introduction to the application of this technology for examination of the fetal heart, which has been described previously¹.

THE UNSTATED COMPLEXITY OF TWO-DIMENSIONAL SONOGRAPHY

The challenge of displaying a three-dimensional (3D) object on a 2D surface is one of the oldest problems in art, and one for which there is no unique solution². This task also applies to sonography³, and we have traditionally relied on 2D ultrasound to image the fetus. Real-time 2D sonography requires the operator to be continuously and actively engaged. The following tasks need to occur simultaneously: manipulation of the transducer; visualization of the screen; adequate hand-eye coordination; manipulation of the console of the ultrasound machine to adjust parameters that affect the image; knowledge and recognition of fetal anatomy; 3D mental reconstruction of 2D images; and constant adaptation to dynamic changes (e.g. fetal movement or change in position). As a result, during the entire examination, the operator must be engaged in simultaneous acquisition, display and interpretation of images. One of the benefits of real-time sonography is that once an image has been acquired (e.g. four-chamber view), no additional manipulation is needed. However, the flip side is that the responsibility of acquiring the image in the correct plane rests entirely on the sonologist. If the planes required to image the great vessels (i.e. pulmonary artery and aorta) adequately are not obtained during the examination, such planes cannot be reconstructed afterwards. Nevertheless, the instantaneous nature of real-time 2D sonography is one of the reasons why many sonologists prefer this over volumetric sonography.

2D sonography is highly operator-dependent and requires extensive training, skill and expertise to assess the fetus and other structures properly. As a result, reproducibility between examiners is often only fair to moderate^4–6. Sonographic images of the same fetus obtained by two different operators may vary in orientation, size, and appearance, all of which can influence diagnostic interpretation. Indeed, Tegnander and Eik-Nes reported that experience has a significant impact on examination of the fetal heart and on the prenatal detection rate of major congenital heart defects⁷. Experienced sonographers/midwives successfully obtained both the four-chamber view and great vessels in 75%, while the success rate was only 36% (P < 0.001) for less-experienced sonographers/midwives. As real-time sonography is a tomographic technique, the display of images is also limited to the specific 2D plane acquired. Therefore, interpretation and diagnosis are strictly dependent on whether or not a diagnostic plane has been successfully obtained. Images can only be obtained sequentially, and examinations may be lengthy when optimal imaging conditions are not present (e.g. suboptimal fetal position).

Fetal lie, presentation and position can all affect the configuration of standard diagnostic planes. For example, if a fetus is in a vertex presentation, the normal cardiac apex will point to the left side of the screen (Figure 1a), while if a fetus is in a breech presentation, the same cardiac apex will point to the right side of the screen (Figure 1b). As interpretation is dependent on pattern recognition, the more permutations of the same diagnostic plane that the sonologist needs to remember, the more challenging is the interpretation. An example of this concept is shown in Figure 2, which shows the diagnostic plane of the left ventricular outflow tract when the fetus is in a breech presentation and the spine is at 10 o’clock (Figure 2a), and the same plane when the fetus is in a vertex presentation and the spine is at 6 o’clock (Figure 2b). Although the information contained in both images is exactly the same (anatomical structures and their relationships with each other), it is more difficult to interpret the image of the fetus that is in a breech presentation with the spine at 10 o’clock. As will be discussed later, one of our goals in developing ‘intelligent navigation’ was to reorient the diagnostic planes automatically so that there is consistent display of a given plane and anatomical structures, allowing their relationships to be more easily recognized as normal or abnormal.

Figure 1. (a) Fetus in vertex presentation.

The normal cardiac apex points to the left side of the image. (b) The same fetus as in (a); however, the image has been flipped (for illustrative purposes) to show that, if the fetus is in a breech presentation, the cardiac apex points to the right side of the image.

Figure 2. (a) Left ventricular outflow tract when the fetus is in a breech presentation and the spine is at 10 o’clock.

Anatomical structures are more difficult to recognize in this image compared with that in (b). (b) Same image as in (a); however, it has been reoriented for illustrative purposes to show the fetus in a vertex presentation with the spine at 6 o’clock. Anatomical structures are easier to recognize in this more optimal fetal position. LV, left ventricle; RV, right ventricle.

Another challenge is that fetal position changes frequently throughout the sonographic examination. Informative images can be obtained successfully when the fetus offers an optimal window (e.g. spine down); however, when the fetus moves to a different position (e.g. spine up), the same informative images may not be obtained. This universal experience could be solved if a rapid image acquisition is achieved when the fetus is in the optimal position. Today, this can only be accomplished with volumetric sonography.

DEFINITION OF INTELLIGENT NAVIGATION

We propose the term ‘intelligent navigation’ to refer to a new method of interrogation of a volume dataset whereby identification and selection of key anatomical landmarks allow the system to: 1) generate a geometrical reconstruction of the organ of interest; and 2) automatically navigate, find, extract, and display specific diagnostic planes. This is accomplished using operator-independent algorithms, which are both predictable and adaptive. ‘Predictable’ refers to the property that allows the generation of diagnostic planes in a consistent way. We have employed the term ‘adaptive’ to describe a function that permits the method to fit the anatomy of each particular case under examination. Other methods are available to examine the fetal heart^8–15. Yet, they require the operator to manipulate the volume dataset and employ methodology to generate standard cardiac views.

‘Intelligence’ is conventionally defined as ‘the ability to apply knowledge to manipulate one’s environment or to think abstractly as measured by objective criteria (as tests)’¹⁶. Similarly, use of the term ‘intelligent’ in the context of this report describes the system’s ability to automatically analyze a set of variables contained within the volume dataset, thus making the extraction of anatomical information operator-independent. Intelligent navigation is not restricted to ultrasound and can be applied to any volume dataset, such as those obtained with computed tomography (CT), magnetic resonance imaging (MRI) or nuclear medicine technology. For the purposes of this communication, however, we will describe ‘intelligent navigation’ as it pertains to obstetrical sonography and the fetal heart.

COMPARISON OF NAVIGATIONAL METHODS (MANUAL VS INTELLIGENT) IN SONOGRAPHY

3D sonography allows examiners to move from a mental 3D reconstruction mode of 2D images to an actual 3D visualization of anatomic structures¹⁷. The introduction of volumetric sonography was expected to improve the quality of imaging and also the detection of congenital anomalies. A distinct characteristic of 3D sonography is that volume datasets theoretically contain all the information acquired from the transducer sweep. Consequently, sonologists can dissect the dataset to examine anatomical areas of interest in planes of section other than the original acquisition plane^18–21. This requires interaction of the operator with the volume dataset, which is generally carried out through ‘manual navigation’ (e.g. operating the x, y, z controls; scaling; and parallel shifting). The same applies to spatiotemporal image correlation (STIC) volume datasets of the fetal heart^22–27, which are displayed as a cine loop of a complete single cardiac cycle in motion.

Sonographic examinations (whether 2D, 3D or STIC) can be considered to have two phases: 1) an acquisition phase during which the transducer acquires tomographic images or volume datasets; and 2) retrieval and display of relevant diagnostic planes. Two-dimensional sonography of the fetal heart (also a form of ‘manual navigation’) devotes most of the time to the acquisition phase, while ‘retrieval’ and display of diagnostic planes occurs rapidly (essentially instantaneous). Therefore, the time required to complete a fetal echocardiographic examination with display of all recommended diagnostic planes is variable, but generally is a lengthy examination. In contrast, volumetric sonography (3D or STIC) has a rapid acquisition phase (in seconds), but the retrieval and display of all the relevant diagnostic planes (‘manual navigation’) is often time-consuming, tedious and challenging. The interrogation of volume datasets requires substantial knowledge of anatomy, the tools used for interrogation (i.e. software) and considerable expertise. Intelligent navigation, however, interrogates the captured volume dataset and then automatically extracts and displays the diagnostic planes. The advantage of intelligent over manual navigation in volumetric sonography is the short time required for both acquisition and retrieval and display of diagnostic planes, along with the added benefit of identifying orientation and anatomical structures (e.g. automatic labeling of each cardiac chamber), which will be discussed later. Table 1 shows a comparison of these methods of performing obstetrical sonography: 1) live 2D (manual navigation); 2) volumetric (manual navigation); and 3) volumetric (intelligent navigation).

Table 1.

Comparison of three methods used to perform obstetrical sonography: live two-dimensional (manual navigation); volumetric (manual navigation); and volumetric (intelligent navigation)

Sonographic volume dataset (3D or STIC)
Characteristic	Live 2D sonography (manual navigation)	Manual navigation	Intelligent navigation
1. Ease and speed of image/volume acquisition	Variable	Easy and fast	Easy and fast
2. Retrieval and display of diagnostic planes	Rapid (essentially instantaneous)	May be prolonged	Rapid (essentially instantaneous)
3. Duration of time to obtain images	Unpredictable, may be long	Unpredictable, may be long	Predictable, short
4. Process of obtaining images	Ongoing and sequential	Ongoing and sequential; however, images may be displayed simultaneously in multiplanar view	Diagnostic planes displayed simultaneously
5. Ease of obtaining images	May be difficult	May be difficult	Easy
6. Postprocessing of images (e.g.rotation, alignment)	No	Yes, manually	Yes, automatically
7. Other images obtained besides original acquired image	No	Yes	Yes
8. Consistency of images (e.g. position, orientation) between cases	No	Yes, manually	Yes, automatically
9. Variation in images between examiners	Increased	Increased	Decreased
10. Level of operator dependency	High	High	Average
11. Labeling of subject orientation, diagnostic planes, anatomical structures	Yes, manual	Yes, manual	Yes, automatic
12. Sonographic navigation of surrounding structures in a diagnostic plane of interest	Manual (sonologist)	Manual (postprocessing software)	Automatic (VIS-Assistance)
13. Navigational instrument (analogy)	Map/compass	Map/compass	GPS navigation

2D, two-dimensional; 3D, three-dimensional; GPS, global positioning system; STIC, spatiotemporal image correlation; VIS-Assistance, virtual intelligent sonographer assistance.

DEVELOPMENT OF INTELLIGENT NAVIGATION

We developed intelligent navigation using a commercially available software system (SONOCUBIC®; Medge Platforms, Inc., New York, NY, USA). Intelligent navigation includes the following1: 1) marking of anatomical structures in the volume dataset using Anatomic Box® (a feature of SONOCUBIC); 2) display of diagnostic planes; and 3) operator-independent sonographic navigation and exploration of the surrounding structures in previously identified diagnostic planes, using Virtual Intelligent Sonographer Assistance (VIS-Assistance®).

Marking of anatomical structures in the volume and display of diagnostic planes

After a volume has been acquired (either 3D or STIC), the software scrolls through the volume and displays the individual slices (1 pixel in thickness) as a 2D cine loop. For STIC volumes, this is known as STICLoop™¹. The next step is to identify the anatomical structures that will allow geometrical modeling of the organ of interest (i.e. the fetal heart). The underlying concept is that several marked structures (or points) within the volume dataset are sufficient to generate spatial coordinates that would provide a means for reliable and accurate reconstruction of the organ. The Anatomic Box feature is employed to accomplish this. The anatomical points have been chosen so that they are parsimonious. In other words, we have determined the minimum number of points that would permit reliable modeling; otherwise, the procedure would become cumbersome for the user. There is a tradeoff between increasing the number of points that would increase accuracy of modeling and the ease of completing the marking process so that it can be practical during clinical examinations. A menu is displayed that identifies the anatomical structures to be marked and in what specific order, all of which are crucial for geometrical modeling performed by the software system. The process is simplified because the system automatically scrolls through the volume to the level of the most likely location of the anatomical structure to be marked. The user may need to perform minor scrolling adjustments to mark anatomical structures appropriately.

Once marking of structures by the user is complete, the system automatically and immediately rotates, aligns, dissects, and scales the volume dataset to display the diagnostic planes of interest simultaneously. For STIC volumes, fetal cardiac diagnostic planes are shown as a cine loop of a complete single cardiac cycle in motion.

Automatic labeling through intelligent navigation of anatomical structures within the diagnostic planes is possible because the system ‘infers’ the actual location of structures in space. This occurs regardless of the way in which the volume dataset was acquired. This was intended to assist readers of images in recognizing anatomical structures, and also to allow the images generated by the system for a particular case to be compared with what is considered normal. Similarly, labeling of fetal orientation (e.g. left, right, cranial, caudal) and naming of each of the diagnostic planes is also possible. The labeling is distinctive because it stays with the corresponding anatomical structure(s), even as the image is increased in size (zoom). Automatic labeling is an optional feature, and all areas of labeling are activated by pressing a single button.

A significant advantage of intelligent navigation technology is that each diagnostic plane is independently and automatically displayed to always show the same and consistent view, regardless of the initial orientation of the structure of interest. Therefore, the technology automatically realigns the volume, and reorients and standardizes the anatomical position, so that the fetus and diagnostic planes are consistently displayed in the same manner each time, regardless of the fetal position or initial orientation (e.g. converts a true breech to a ‘vertex’ presentation, places the spine at 6 o’clock). This is done so that marking of anatomical structures is easier, as each would be expected to be in the same location on the screen. Moreover, structures and fetal anatomy are more easily recognizable by the user. An important feature of the system is that it also generates a geometrical reconstruction of the organ of interest that fits the anatomy of each particular case under examination. This occurs despite the gestational age and anatomical variability (e.g. cardiac axis and geometry), making the algorithm adaptive.

VIS-Assistance

VIS-Assistance is a tool developed for the purpose of operator-independent sonographic navigation and exploration of surrounding structures in a diagnostic plane of interest (e.g. four-chamber view). VIS-Assistance can be activated for each diagnostic plane, and the net effect is further ‘scanning’ (as a videoclip) that has been programmed to yield additional information within a specified territory of the diagnostic plane. This was developed because there are anatomical variations in organs, and the sonologist may want additional information not displayed in the original generated diagnostic plane. One can think of this feature as a ‘virtual’ sonographer. VISAssistance ‘scans’ the volume in a purposeful and targeted manner with the goal of visualizing specific structures. Navigational movements in VIS-Assistance are intelligent due to the design of pivot points that change, and around which sequential movements are centered. Therefore, the objectives of VIS-Assistance are: 1) to improve the success of obtaining the view of interest (e.g. fetal echocardiography view); and 2) to provide more information about the diagnostic plane and its surrounding structures.

The following examples illustrate the application of VIS-Assistance to the fetal heart: 1) although the four-chamber view may be depicted successfully in the diagnostic plane, if the user wishes to obtain more information about the pulmonary veins or the atrial septum, VIS-Assistance can be activated¹; 2) the left ventricle may appear foreshortened in the five-chamber view diagnostic plane. To determine whether this is due to a tilted plane (azimuth) or because the left ventricle is truly hypoplastic, one can activate VIS-Assistance; and 3) the left ventricular outflow tract diagnostic plane may not show excursion of the mitral or aortic valve leaflets fully. In this situation, VIS-Assistance can be activated, as navigational movements will occur around the area of these valves.

What are the advantages of VIS-Assistance?

The underlying concept of VIS-Assistance may be comparable with performing manual navigation in live 2D sonography or within a volume dataset. However, VIS-Assistance has several unique characteristics and advantages when compared to these methods:

• There is automatic navigation through the volume dataset. Operator dependence is therefore decreased, because there is no manual navigation performed of the structure of interest (e.g. the fetal heart).

• There are consistent navigational movements through the volume each time that VIS-Assistance is activated. In contrast, when performing manual navigation of a volume dataset or live 2D sonography, each examination and series of steps (e.g. step order; rotation on x, y, and z axes; parallel shifting; transducer movements) may not be identical for the same examiner or between examiners.

• The types of navigational movements through the volume are unique and fluid and would be difficult or impossible to perform otherwise. For example, VIS-Assistance can be designed such that swinging in oblique planes can occur from a single pivot point in a sequential series of movements equidistant from each other. This precision and consistency may not be achievable with live 2D sonography or manual navigation of a volume dataset.

• The time duration of VIS-Assistance is typically shorter (< 4 minutes)¹.

Other benefits of VIS-Assistance include the possibility of showing the appropriate azimuth, in which tilted planes may cause suboptimal visualization of anatomy or raise suspicion of an abnormality.

LIMITATIONS OF INTELLIGENT NAVIGATION

There are several limitations to the use of intelligent navigation. First, because this technology is applied to volume datasets, the resulting images depend on the quality of the initial volume and may be suboptimal in cases of motion artifact, shadowing, etc. (for example, if the fetal spine is not between the 5- and 7-o’clock positions at the time of volume acquisition, acoustic shadowing from the ribs or spine may be increased and prohibit adequate visualization of images). If this occurs, further volume datasets should be obtained. Second, extremes of gestational age (i.e. first or third trimesters) may make adequate visualization of images suboptimal or difficult, as a result of small anatomical size, rib or extremity shadowing, etc. Similarly, these same reasons may lead to difficulty in marking anatomical structures because they are not well visualized. Third, the Anatomic Box feature is based upon the assumption that there is normal anatomy and that structures for marking are present and in their usual location. Moreover, the user must possess the ability to mark the anatomical structures appropriately. If such conditions are not met, diagnostic planes may be either unobtainable or incorrect. Under these circumstances, one alternative may be to activate VIS-Assistance, which may be informative. Fourth, intelligent navigation is not a replacement for live 2D sonography because the latter allows measurements of fetal biometry, evaluation of organ function, pulsed Doppler interrogation of vessels, etc.

PRACTICAL ASPECTS OF INTELLIGENT NAVIGATION

Currently, Internet consultation using volumetric sonography requires dedicated software. In contrast, an important feature and benefit of intelligent navigation technology is that it can operate on conventional computers and is not dependent on specific ultrasound platforms or on the use of software to perform manual navigation of volume datasets. A valid concern is that a user may be able to acquire a volume dataset and mark the anatomical structures using Anatomic Box, but may not have the expertise to interpret and read the resulting images. As volume datasets, diagnostic planes, and VIS-Assistance videoclips can be transmitted using telemedicine, expert consultants can evaluate the images to provide an opinion. Moreover, smartphones, tablets and other devices may also be used to receive the transmitted information (diagnostic planes or VIS-Assistance videoclips). As the latter are obtained automatically once anatomical structures are marked by the user, operator dependency and the time spent by the expert consultant (i.e. time devoted to manual navigation of the volume dataset) should be decreased.

CONCLUSION

Intelligent navigation is a new method of interrogation of a sonographic volume dataset, whereby identification and selection of anatomical landmarks allow the system to automatically navigate, find, extract, and display specific diagnostic planes of interest. This is accomplished using operator-independent algorithms that are both predictable and adaptive. Virtual Intelligent Sonographer Assistance (VIS-Assistance) may also be activated, so that operator-independent sonographic navigation and exploration of the surrounding structures in a diagnostic plane of interest occurs. The end result is a user-friendly, simple, fast, and consistent method of obtaining sonographic images with decreased operator dependency. Therefore, this has the potential to improve efficiency and workflow. Intelligent navigation is one approach to improve obstetrical sonography.