Implementing Cost-effective 3-Dimensional Ultrasound Using Visual Probe Localization

Abstract

This article proposes a set-up for a 3-dimensional ultrasound system using visual probe localization on the conventional 2-dimensional ultrasound machines readily available in most hospitals. A calibrated digital camera is used for probe-tracking (localization) purposes, whereas ultrasound probe calibration is implemented using a purpose-built phantom. The calibration steps and results are detailed here. The overall system is proven effective in clinical trials through scanning of human organs. Results obtained show successful, accurate 3-dimensional representations using this simple cost-effective set-up.

Introduction

Ultrasound imaging is now advancing rapidly in terms of its resolution and quality. In most cases, despite the inherent low quality, 2-dimensional ultrasound is the most widespread and convenient medical imaging method because it is cheap, nonionizing, noninvasive, portable, interactive, and allows real-time acquisitions. However, one of its major limitations is its inherent 2-dimensionality, which limits the amount of information that could be obtained from the scans.

Three-dimensional ultrasound imaging solves this limitation by providing the ability to generate extended images. It provides better qualitative and quantitative information to enable the physicians to diagnose more effectively.¹ Three-dimensional ultrasound offers the medical practitioners a view of slices through the human body, which is not possible using the conventional 2-dimensional ultrasound system. In addition, the reconstruction of the 3-dimensional view reduces scanning and study time by the physicians, as the reconstructed 3-dimensional image allows the physician to manipulate the images and observe various parts of it without requiring further scanning, thus reducing patient waiting time and discomfort. Volume calculation may also be realized through 3-dimensional ultrasound. All of these advantages contribute to a more precise diagnosis by the physicians. This article proposes a cost-effective solution of using the conventional 2-dimensional ultrasound probe with a conventional digital camera to generate 3-dimensional ultrasound images.

Three-Dimensional Ultrasound

Generally, there are three options in producing 3-dimensional ultrasound images by acquiring a series of 2-dimensional ultrasound images.² In freehand acquisition, the sonologists are free to move the probe in any orientation, but probe localization techniques must be applied to extract the position information of the probe to be matched with the 2-dimensional images.³ In the mechanical localization technique, the probe is positioned by a robotic mechanism, which is not so favorable to the doctors as it limits the freedom of the probe movement. However, because robotic control is applied, accurate information on locating the position of the probe while scanning can be extracted in a more efficient way. Three-dimensional probes are another option but is considerably more costly as compared with the previous two techniques. In the 3-dimensional probe solution, a special probe is used to acquire the scans and their respective relative positions so that a 3-dimensional volume may be reconstructed almost in real time. It offers a better frame rate compared with other options and allows the physician to orientate the probe in the best position so the optimal view of the organ scanned could be obtained. However, the field of view of the scan area is limited when using 3-dimensional probes. There is also the option of modifying a 1-dimensional transducer array to produce 3-dimensional ultrasound.⁴ Among the listed options, the freehand acquisition is the preferred technique as it is cheap, easily adapted to existing systems, and allows the medical practitioner unhindered usage of the probes they are used to, and in the orientations and sequences preferred by the individual operator. However, real-time creation of 3-dimensional images as the acquisition is performed is not possible using this option.

Among the recent developments in 3-dimensional ultrasound is STRADX, a 3-dimensional ultrasound software developed at the University of Cambridge.¹ The software is meant for freehand 3-dimensional ultrasound acquisition using the conventional 2-dimensional machine affixed with an optical or magnetic tracking system for the purpose of probe localization. Although STRADX works perfectly well with the existing 2-dimensional ultrasound machine, the magnetic or optical tracking system it uses is expensive. Furthermore, the use of a magnetic tracker could cause serious interferences to the unshielded equipment as it is very difficult to avoid metal objects in a clinical situation. The use of an optical tracker requires line of sight at all times during the scanning process, which sometimes limits the physician's movements and examination methodology.

By supplying a set of 2-dimensional ultrasound B-scans, together with the precise position of the probe while scanning, software such as STRADX could be used to enable 3-dimensional ultrasound imaging. In this article, an approach of acquiring the information on probe position is proposed, by applying machine vision using a conventional digital camera. In this approach, the additional cost incurred by the overall system to enable existing 2-dimensional equipment to produce 3-dimensional imaging results is effectively reduced, in comparison with the use of magnetic or optical tracking system to track the probe. The cost of the proposed system is approximately U.S. 4,500. The problem of magnetic interference and occlusion may also be suppressed. Furthermore, the required equipment are easily available and can easily be made use of for more general purposes in other situations.

Methodology

The development of the proposed technique is divided into several steps (Fig. 1). Machine vision is proposed for the purpose of 3-dimensional tracking of the probe, where a calibrated camera is used to identify the real-world position of the probe with respect to the camera. The probe has then to be calibrated to deduce its position with respect to the ultrasound scan. From these two calibration steps, the position information of the ultrasound scan can be extracted and tagged to the respective scan. These are fed into STRADX to render the 3-dimensional image and perform additional operations on the scanned volume, such as segmentation, surface fitting, etc.

Fig 1.

Flowchart for 3-dimensional ultrasound using the proposed scheme.

Camera Calibration

A conventional low-cost digital camera is used to capture the probe's image to enable the calculation of the real-world position of the probe.⁵ The prerequisite of this effective measurement is calibration to determine the internal camera geometric and optical characteristics, referred to as intrinsic camera parameters, which will be used in calculating the orientation of the probe with respect to the camera. A camera calibration toolbox,⁶ developed based on previous work by Zhang⁷ and Heikkila and Silven,⁸ simplifies this task.

Formulation

In the camera calibration toolbox, the camera model used is a combination of the pinhole camera model and the distortion model.⁷ The basic information needed for calibration is real-world, 3-dimensional coordinates of known points and the corresponding 2-dimensional coordinates on the calibration images. The idea of calibration is to estimate the parameters that link the two coordinate systems. The real-world coordinates are transformed into camera frame coordinates, which is related by Eq. 1.⁷

1

Or in matrix form

2

where X_c is the coordinate in-camera frame (x,y,z) to be estimated, X is the known real-world coordinates of the same point (X,Y,Z), R_c is the rotation matrix, and T_c is the translation vector.

Next, the intrinsic parameters, which include focal length f, principal point (u,v), and scale factor s_u, need to be estimated. The projected point P (x,y,z) in the camera frame onto the image plane is related by Eq. 3.

3

The coefficients D_u and D_v are needed to change the metric unit to pixel. Thus, the corresponding point in pixels is given by

4

Two types of distortion in the camera model used are considered. The radial distortion is expressed using the following equation:

5

where k₁, k₂, and k₃ are coefficients for radial distortion and r = (u² + v²)^1/2. The tangential distortion is expressed using the following equation:

6

where p₁ and p₂ are coefficients for tangential distortion. Combining the pinhole model with the distortion, the final pixel coordinate of the point is given by

7

Calibration Object

The calibration object used in this study is a checkerboard planar pattern affixed on a rigid planar surface (see Fig. 2). Sets of images of the calibration object are required to calculate the intrinsic parameters. The real-world and image coordinates of the crossing points are input into the toolbox to calculate the coordinates in the camera frame. Empirical tests were performed to check for the optimal calibration set-up.

Fig 2.

Sample calibration object.

Determining Optimal Calibration Setup

The calibration performance is evaluated for different set-ups used by varying the different parameters, including the resolution of the calibration images, distance between camera and calibration object, number of control points on calibration images, and the number of images used. The focal length errors, both vertical and horizontal, are used as the measure of tracking performance for the different set-ups. It is defined as the estimates of the uncertainties of the calculated focal length after the calibration procedure.⁶

Resolution The first parameter studied was the image resolution. Images were captured for three different conventional camera resolutions, using combinations of images 2, 4, and 6 taken from the corresponding positions of the checkerboard plane in Figure 3. The camera was fixed at a distance of 30 cm from the planar pattern. The number of control points used for each image was 49 points (7 × 7) or the edges of 6 × 6 squares on the checkerboard calibration object. The chosen settings were based on practical usage and set-up for the camera and the proposed system. The results of the test are given in Table 1. It is observed that the image with the lowest resolution gave the smallest focal length error, which relates to the accuracy of the actual displacement. Based on this result, the rest of the calibration images were taken at the 640 × 480 pixel resolution. It may be noted that the horizontal and vertical focal length errors do not deviate much from each other and is included for completeness. This result is encouraging as it shows that a low-cost, low-resolution digital camera may be sufficient for this scheme, as toolboxes may be customized accordingly.

Fig 3.

Camera calibration object set-up showing positions of different orientations.

Table 1.

Calibration Performance for Different Image Resolutions, in Terms of Horizontal (fc₁) and Vertical (fc₂) Focal Length Error, Measured in Pixels

Resolution	fc1 Error	fc2 Error
2,048 × 1,536	152.57658	148.04606
1,024 × 768	51.81176	51.25215
640 × 480	37.49968	36.96163

Distance Next, the effect of distance from the camera to the planar pattern was investigated. The test distance between the camera and the planar checkerboard was varied, with the checkerboard being perpendicular to the camera. Distances below 20 cm could not be used because complete image of the planar pattern could not be obtained when the object was too close, resulting in calibration difficulty. The size of the image is inversely proportional to distance. As such, when the distance is large, the corners of the squares in the image become very difficult to select accurately, affecting calibration. It is observed from the graph in Figure 4 that, as expected, the error increased with an increase in the distance between camera and the planar pattern. This is practically convenient too, as it indicates that for the purpose of calibration, a smaller space and set-up is sufficient.

Fig 4.

Calibration performance of different camera–object distances, measured in terms of focal length error.

Number of Control Points Knowing the best resolution and distance, the number of control points used for calibration can then be analyzed. For a marker of m × m squares, the number of control points is the number of intersections on the selected squares on every calibration image, given by Eq. 8.

8

The number of control points is varied from 4 to 121 points (ie, 2 × 2 to 10 × 10 squares). The result of the test is plotted in Figure 5. The number of control points tested is dependent on the selection of the square region of interest on the planar pattern (ie, 9 points for 2 × 2 squares, 16 for 3 × 3, etc.), with a minimum number of 16 points for calibration. As expected, it was found that the greater the number of control points, the smaller the focal length error, using a larger number of sample coordinates and minimizing the error. The error decreased significantly from 16 to 36 points, and then gradually thereafter.

Fig 5.

Calibration performance of different numbers of control points, measured in terms of focal length error.

Size of Squares With the camera fixed at a 20-cm distance, sets of images with different sizes of squares were captured and used for calibration. From the results in Figure 6, it is observed that larger squares produce lower errors.

Fig 6.

Calibration performance of different size of squares, measured in terms of focal length error.

Number of Calibration Images For the previous experiments, the images used were the combination of images 2, 4, and 6. The number of images (or planes) used for calibration can be varied. Different combinations of images were tested to observe the effects of the different orientation of the planar pattern on calibration. For combination of planes with different orientation as shown in Figure 3, the result obtained is as given in Figure 7. For some combinations, the toolbox failed to calibrate the camera as a result of insufficient information derived from the orientations of the planar pattern, giving rise to large inaccuracies. From the plot, it can be seen that the combination of all seven planes produces the smallest error, whereas combination 1–6 gives the largest error due to the steep angle of orientation from the camera, which causes difficulty in determining size of the boxes and world coordinates. Generally, combinations of just two images produced poor result. The smallest error for two images is given by images 2 and 3, which is at the orientation of 70° and 80° from the focal axis in Figure 3. The reason for this is that the angle between the two planes were rather small and both of the planes were at some angle (ie., not perpendicular) from the camera.For combinations of three images, the largest error is given by the set of images 5–6–7. The results for combination sets of images 1–4–7, images 2–4–6, and images 3–4–5 are presented in Table 2 for more detailed analysis. From Table 2, it can be concluded that with an increase in the angle of separation between the planes, the focal length error is decreased. The error for the combination of all images (seven planes) and combination of images 1–4–7 is almost similar. This is also reflected in the best four-image combination of 1–3–5–7.

Fig 7.

Calibration performance of different numbers of planes and orientations.

Table 2.

Focal length errors for selected three-image combinations

Three-Image Combination	Angle between Images (°)	fc1 Error	fc2 Error
3–4–5	10	39.8718	39.4408
2–4–6	20	13.0883	12.5515
1–4–7	30	8.0261	7.5671

Optimal Calibration Setup From the results, an optimal calibration set-up that considers all the parameters above is proposed. Using the best set-up of 1.5 × 1.5 cm squares, 121 control points at a distance of 20 cm, with planar orientation of images 1–4–7, it is found that calibration of good accuracy may be achieved. The calibrated camera can be used to capture the image of the ultrasound probe affixed with a planar checkerboard marker patch to determine the position of the probe with respect to the camera. Once the camera is calibrated, in view of practicality and through further experimentation, it was found that a marker patch of size 3 × 3 cm containing 16 control points is sufficient for use on the probe.

Probe Calibration

The position information required for 3-dimensional reconstruction is the relative position of the B-scans with respect to the calibrated camera. With the position of the probe known from the calibrated camera, a probe calibration step is performed to determine the actual position and orientation of the ultrasound scans with respect to the probe.⁹ As there are a number of different interacting components in the set-up, each having its own coordinate references, the calibration result determines the transformation from the coordinate system of the ultrasound images to the camera coordinate system used to capture the image of the probe during each ultrasound scan. Different phantoms are used in different calibration methods,² and a suitable calibration set-up ensures that the transformation accuracy is maintained.

Coordinate System

The different components of the proposed probe calibration set-up and their corresponding coordinate systems are shown in Figure 8. P is the coordinate system of the B-scan ultrasound image plane, with the origin at the top left corner of the image. The y axis is in the direction of the beam from the ultrasound probe, x axis is in the lateral direction, and z axis is the elevation, out of the plane of the B-scan. M is the coordinate system of the marker placed on the probe. C is the coordinate system of the camera used to capture the image of the marker, to identify the location of the probe. Finally, V is the coordinate system of the reconstructed volume from the acquired 2-dimensional B-scans.³

Fig 8.

The coordinate systems involved in the reconstruction step for a 3-dimensional ultrasound system.

Three-Dimensional Reconstruction Requirements

Three-dimensional reconstruction is the mapping of each B-scan ultrasound image into the reconstructed volume, V. Pixels in the B-scan image (X_p in the P coordinate system) is transformed first into the marker's coordinate system (M) through a rigid transformation T_m, followed by the camera coordinate system (C) through transformation T_c, and finally to the coordinate system of the reconstructed volume (V) through transformation T_v. Expressed as a multiplication of homogeneous transformation matrices, the entire process is represented by Eqs. 9 and 10, where u and v are the image pixel coordinates on the B-scan, with s_x and s_y as the scaling factors.

9

10

Transformations between two coordinate systems involve six degrees of freedom, namely the three translations in the x, y, and z axes, and the three rotations, one about each of these axes (α, β, and γ, respectively) in sequence. Eq. 11 summarizes the transformation undertaken between the coordinate systems.

11

where

12

13

14

15

16

17

18

19

20

The transformation T_c in Eq. 9 is determined during the camera calibration step. The transformation T_v is included for convenience, to eliminate the need to construct a large voxel array that is mostly empty when the reconstructed volume is aligned with the camera coordinate system.⁸ Only T_m, ie., the transformation from the B-scan coordinate system to the marker coordinate system, and the estimates of s_x and s_y in Eq. 10 need to be solved.

Probe Calibration Set-up

Generally, probe calibration for 3-dimensional reconstruction is implemented by scanning a phantom whose 3-dimensional geometrical properties are known, and then using the sequence of 2-dimensional images taken of the phantom, the necessary parameters are estimated. Broadly, probe calibration methods can be classified into three types, depending on the phantom used: multimodality registration,² (the measure of the phantom acquired in the calibration process is compared with the measured value in other modalities, such as magnetic resonance or computed tomography volume), wire phantom² (the cross section of wires is detected in the ultrasound scans and used as calibration input), and single-wall phantom³ (ultrasound scans of a wall, eg., the wall of a water bath, are used for calibration). Although the single-wall phantom is preferred, it suffers from a problem with ultrasound beamwidth thickness, resulting in a blurred reflection of the wall in the ultrasound scans when the scans are taken at an oblique angle, allowing weak reflections from unwanted points in the ultrasound beam. The Cambridge phantom² was developed to overcome this problem by ensuring that only the center beam is reflected back to the transducer, thus resulting in a strong and precise line appearing on each ultrasound scan.

A probe calibration set-up based on the Cambridge phantom, satisfying the single-wall equation in Eqs. 9 and 21, was developed using a steel ruler on two nonelastic wheels with a probe holder, as shown in Figure 9.

21

The steel ruler is positioned such that it is always in line with the center of the two wheels. The probe holder ensures that the position of the probe, in relation to the ruler, is fixed such that the ultrasound scans acquired for all the calibration orientations are consistent. The phantom and experimental set-up allows accurate calibration, without introducing any additional degrees of freedom. The marker is attached to the top of the probe in such a way that it is always in clear view of the camera. The cube shape ensures that sufficient number of control points on the checkerboard pattern can be identified from the different orientations to determine the probe position. The phantom is immersed in a warm water bath to simulate human tissue. The scanning tip of the waterproof ultrasound probe is immersed completely in the water as well.

Fig 9.

Probe calibration phantom.

The phantom is set in a number of orientations, covering all the six degrees of freedom, as shown in Figure 10. For each orientation, the image of the probe and the corresponding ultrasound B-scan are captured.

Fig 10.

Orientations of the probe for calibration.

The top of the steel ruler is detected in each scan (as in Fig. 11). Together with the calculated position of the marker that was determined using the calibrated digital camera, probe calibration calculates the actual position of the scans relative to the marker attached to the probe. The values of the coefficients are determined by satisfying Eqs. 11–20.

Fig 11.

Arrows indicate detected top of the steel ruler.

To test that the calibration was successful, the top of steel ruler may be segmented as a line in each 2-dimensional slice. These lines are shown in Figure 12, each depicting the different orientations tested during calibration. Because the position of the ruler is fixed in the phantom such that it is always in line with the probe beam, the rendered 3-dimensional volume image would be a straight line. This is observed by introducing a virtual plane through the cross sections of the segments, as in Figure 12.

Fig 12.

Segmented top of ruler for the different orientations.

Results

Confirming the validity of the set-up through visual testing of the entire system is completed by satisfying Eqs. 9, 10, and 21 with the coefficients determined from the calibration steps. Freehand scans (see Fig. 13) were taken using the calibrated 2-dimensional probe and reconstructed into 3-dimensional volume. Figure 14 shows the results achieved for two of the tested objects. The segmented scans, where the surface of the scanned object in each slice is outlined, is given on the left. The 3-dimensional volume rendered, represented through surface fitting of the segments, is given on the right. It is observed that the set-up has been successful in producing accurate 3-dimensional visualization from a series of 2-dimensional ultrasound scans, with the aid of positioning information using a digital camera.

Fig 13.

Freehand scan using the calibrated probe to scan a cylinder and rectangular box.

Fig 14.

Reconstruction of 3-dimensional surface from 2-dimensional slices.

The test objects were affixed with nails serving as distance markers, which are used to further test the accuracy of the calibration process. Three nails with actual interval of 20 mm were used, as illustrated in Figure 15.

Fig 15.

Nails placed 20 mm apart on the scanned object.

The distances between the nails on the reconstructed objects were measured and compared with the actual distances, as given in Table 3. The scaling factor for the reconstructed object is described by Eq. 22

22

where D_A is the actual distance on the object and D_M is the measured distance on the 3-dimensional reconstruction. A scaling of m = 1 is chosen through selection of appropriate monitor resolution for viewing the reconstruction.

Table 3.

Actual Distances between Nails Compared with the Measured Distances on the Reconstructed 3-Dimensional Objects for the Box and Cylinder

Test Object	Distance (mm)	Measured Distance, y	Actual Distance, x
Box	Nail 1–2	21	20
	Nail 2–3	20	20
	Nail 1–3	41	40
Cylinder	Nail 1–2	21	20
	Nail 2–3	20	20
	Nail 1–3	41	40

Comparison of measured and estimated values against actual values in Table 3 and 4 proves that the calibration enables generation of 3-dimensional objects, which are accurate in its visual representation.

Table 4.

Estimated Dimensions of Test Objects Based on Reconstructed Measurements

Distance (mm)	Calculated Distance	Actual Distance
Width of cube	15	15
Height of cube	24	24

The set-up was then subjected to clinical test by performing a live ultrasound acquisition on human body parts. Figure 16 shows a sample probe position image during a gallbladder scan.

Fig 16.

Sample probe position image for gallbladder scan.

Based on the position information extracted from the series of probe position images, the 2-dimensional ultrasound slices were rendered in 3-dimensional volume using STRADX. Manual segmentation on the organ of interest, the gallbladder, was performed and 3-dimensional surface fitting of it was successfully rendered, as shown in Figure 17.

Fig 17.

Segmented gallbladder 2-dimensional slices and its respective 3-dimensional surface fitting. (a) Segmented ultrasound slices. (b) Reconstructed 3-dimensional surface.

Two other human body parts, aorta and spleen, were also scanned and the results of 3-dimensional surface fittings are shown in Figure 18. The accuracy of the live organs acquisition and reconstructions were verified by a medical specialist.

Fig 18.

Segmented aorta and spleen 2-dimensional slices and their respective 3-dimensional surface fitting. (a) Segmented ultrasound slices. (b) Reconstructed 3-dimensional surface.

Conclusion

This article proposes a cost-effective set-up for reconstruction of 3-dimensional ultrasound volume using conventional 2-dimensional ultrasound equipment and a digital camera. Based on the results obtained, consisting of the 3-dimensional surface fittings of test objects and human organs, it can be concluded that the proposed set-up is successful in its desired objective. Although the reconstructed models shown may not be refined because of human error, inaccuracies caused by the manual segmentation process, they have good resemblance to the organs scanned.

All the ultrasound acquisitions in these experiments were performed in a well-lit environment to ensure clear visibility of the probe marker in each probe position image. Because clinical ultrasound is usually performed in a low-light environment, a special marker (eg., luminous or attached with a small light) could be used to enhance the brightness of the marker in each probe position image captured.

By replacing the use of optical or magnetic tracking system with a calibrated camera, the overall cost of 3-dimensional ultrasound system based on STRADX has been effectively reduced. The overall cost of the proposed system (using Ricoh RDC i500 digital camera) amounts to U.S. 700, which is almost a seventh of the cost of the option of using a magnetic tracking system (eg, Ascension miniBIRD 800, which costs approximately U.S. 4,500). This solution allows for better result from existing 2-dimensional ultrasound equipment, and is feasible for use of establishments that are unable to opt for newer more costly methods of acquiring better quality 3-dimensional ultrasound imaging. The 3-dimensional tracking technique by applying camera calibration can also be used for other nonmedical applications.