Ultrasound imaging and segmentation of bone surfaces: A review

Abstract

Due to its real-time, non-radiation based three-dimensional (3D) imaging capabilities, ultrasound (US) has been incorporated into various orthopedic procedures. However, imaging artifacts, low signal-to-noise ratio (SNR) and bone boundaries appearing several mm in thickness make the analysis of US data difficult. This paper provides a review about the state-of-the-art bone segmentation and enhancement methods developed for two-dimensional (2D) and 3D US data. First, an overview for the appearance of bone surface response in B-mode data is presented. Then, classification of the proposed techniques in terms of the image information being used is provided. Specifically, the focus is given on segmentation and enhancement of B-mode US data. The review is concluded by discussing future directions of research and additional challenges which need to be overcome in order to make this imaging modality more successful in orthopedics.

INTRODUCTION

The demand for an accurate, efficient and less invasive surgery is increasing the importance of surgical procedures performed with computer assistance. High precision during interventions, the ability to perform operations with minimal incision, decreased operation time, declined uncertainty for intra- and post-operative complications and improved safety (reduced radiation exposure) to both patient and surgeon are some of the important advantages of such systems. The first application of computer assistance in surgery, with the use of stereotactic frames, was for neurosurgery in order to locate brain tumors¹.

Following this initial procedure, computer-assisted surgery (CAS) systems started to grow in different application domains with computer-assisted orthopedic surgery (CAOS) being one of the main ones. Due to the rigid structure of the target anatomy (bone) orthopedic procedures are especially suitable for CAS systems. The first clinical study, where a CAOS system was investigated, described an in-vitro validation experiment for spinal fusion surgery². Following this initial study, similar technologies have been adapted to many different procedures in the field of orthopedics such as total hip replacement (THR)³, total knee replacement (TKR)⁴, intramedullary nail locking⁵ and tibial, pelvic and distal radius osteotomies^6–8.

Imaging is one of the most important components of any CAOS system. Currently the dominant imaging modalities used in CAOS are pre-operative X-ray, computed tomography (CT) and magnetic resonance imaging (MRI) and intra-operative two- (2D) or three- (3D) dimensional fluoroscopy (fluoro-CT)^5,9–12. Due to the superior bone imaging quality, CT is the preferred pre-operative modality used for planning the intervention^11,12. However, MRI has also emerged as pre-operative imaging alternative specifically for procedures where CT is not necessarily required such as femoroacetabular impingement surgeries^13,14 or total knee arthoplasty using patient specific instrumentation^15,16. The main drawbacks of MRI and CT in CAOS are the cost of imaging, radiation exposure from CT and mostly being performed pre-operatively for planning the intervention. The principal imaging modality for intra-operative imaging in CAOS is 2D/3D fluoroscopy devices^10,17–20. However, 2D representation of a 3D anatomy necessitates multiple fluoroscopy scans taken from different directions. Therefore, accuracy of the surgical procedure and reproducibility of the surgical actions therefore largely depend on information obtained from 2D fluoroscopy images and the experience of the surgeon. Three-dimensional fluoroscopy units provide 3D information; however, they are twice as expensive and not widely available as standard 2D units. Finally, both of these modalities operate with ionizing radiation causing important concerns for the safety of the operating staff and patient.

In order to provide a radiation-free, real-time and 3D imaging alternative to intra-operative fluoroscopy special attention has been given to incorporate ultrasound (US) into CAOS procedures^21–25. US-based guidance systems for non-surgical procedures such as epidural anesthesia and/or spinal blocks have also been developed^26–30. Nonetheless, due to the high levels of noise, imaging artifacts, limited field of view and bone surfaces appearing blurred and several mm in thickness, US has not become a standard imaging modality in orthopedics (Fig. 1). These difficulties prohibited the use of US as a standalone intra-operative imaging modality and the collected scans were registered to pre-operative plan developed using CT/MRI data. Accurate, automatic and real-time registration is critical for efficient image guidance and anatomical restoration in many orthopedic procedures. In order to achieve this goal, focus has been given to develop bone segmentation methods from US data which is the main focus of this review paper.

Figure 1.

B-mode US images of different bone surfaces (left to right: femur, pelvis, distal radius). Red arrows point to the soft tissue interfaces with similar intensity values as the bone surfaces, which are pointed with yellow arrows.

In their review paper Noble and Boukerroui³¹ provide an overview of many segmentation methods developed for US imaging in the context of soft tissue segmentation. Specific focus was given to cardiology, breast cancer and prostate cancer³¹ applications. Moradi et al.³², in their review paper, focus on the application of US-based diagnosis systems for prostate tissue. A review of segmentation and registration methods of MRI and US data for prostate cancer detection has been provided by Zhu et al.³³ Katouzian et al.³⁴ provide an overview of the segmentation algorithms developed for intravascular US (IVUS) images and discuss methodological challenges associated with the developed algorithms. Molinari et al.³⁵ present a review paper about measurement of intima-media thickness (IMT) and wall segmentation from carotid US data. In a recent book chapter publication, Schuman provides a review of US registration approaches for computer assisted orthopedic interventions³⁶. This paper provides a review of methodologies developed for the segmentation of bone surfaces from US data. In this review paper, methods developed for the enhancement of bone surfaces from radiofrequency (RF) and B-mode US data are also included since these enhanced images could potentially be used as an input to a subsequent segmentation or registration method.

The review paper is organized as follows: It starts with an analysis of US image response characteristics for bone surfaces. It then examines major works on bone US image segmentation and enhancement classified by image information being used with a specific focus on B-mode data. The review is finalized with the author’s conclusions and discussions about possible future trends in the field of bone imaging with US.

MATERIALS AND METHODS

Bone response characteristics in ultrasound data

US images are formed by measuring the amplitude and the travel time of the reflected sound echoes to reach the US transducer. If there is a high acoustic impedance mismatch between two tissue interfaces the reflected sound echo signal will have a high amplitude resulting in a high intensity pixel appearance in the formed US image. A high-intensity pixel in an US image indicates a strong likelihood of the presence of a boundary, such as soft tissue interface or bone. Since bone tissue has one of highest acoustic impedance values compared to other tissues such as muscle, fat, liver or water, most of the signal is reflected back from the bone interface resulting in a high-intensity feature in the reconstructed US image. This high-intensity bone feature is followed by a region with very low-intensity values appearing right after bone boundary. This low-intensity region, “shadow region”, is one of the typical US imaging artifacts denoting that interior bone surfaces cannot be imaged with US imaging. The high-intensity feature depicting bone boundary response looks like a line with a shape closely resembling the surface. However, compared to bone surface appearance in CT or fluoroscopy imaging, the surface response in US is not a sharp transition region but rather has a thickness which can reach a value of 4 mm in certain cases. The response thickness is affected by the inclination of the image surface with respect to the US transducer. The greater the inclination of the imaged surface, the greater the response thickness. Furthermore, the finite beam width in the elevational direction produces a thick response when it is projected to a 2D plane. The actual bone boundary may merge within this response³⁷ (Fig. 2).

Figure 2.

(a) Propagation of an ultrasound (US) pulses (red shapes) along one particular scanline (yellow line). Example echo (blue shape) was generated by reflections of the pulse from structures in the tissue medium all along this path, and the echoes (red shapes) travel back to the transducer. The image is showing a B-mode US image of a distal radius obtained in vivo. (b) Thicker bone surface response is observed due to the elevational beam thickness. The actual bone response ends up being merged inside this thick response. (c) Effect of the imaged surface geometry on the bone surface response. The received echoes from different directions other than the main direct line of sight result in an increased bone response.

Figure 3 shows B-mode US images obtained in vivo by scanning a femur bone. Data are collected by changing the orientation of the US transducer with respect to the imaged 3D anatomy. Despite the fact that the US images are representing the same bone, the features corresponding to the bone surface appear different with varying scanline profiles (Fig. 3; bottom row). The bone surface feature shown on the right US image has the typical characteristics namely a high-intensity feature followed by an intensity drop representing the shadow region. This is also confirmed with the corresponding scanline profile (Fig. 3; bottom right). However, the response features on the middle and left US images appear very different. The bone surface response on the left image is not a high-intensity region any longer and soft tissue interfaces have similar intensity values as the bone feature. The bone surface response on the middle US image has a thick response and the shadow region does not have a distinct intensity drop compared to the other two images shown.

Figure 3.

Bone response characteristics. Top row: In vivo US images of femur bone. Bottom row: Bone scanline profiles belonging to the bone surfaces shown in the top row. The selected scanline is indicated as white vertical line in the B-mode US image data.

Due to the previously mentioned imaging artifacts manual identification of bone surfaces from B-mode US data has been reported in the earlier proposed US-guided CAOS applications^22,23,38. Despite the accurate results obtained in these studies, a major drawback was the time needed for manual segmentation. The accuracy of the segmentation also depends on the operator performing the manual segmentation which may introduce significant inter- and intra-user variability^23,38. Several groups have explored ways to automate US bone segmentation for various applications. The proposed methods were based on the use of image intensity and gradient information or local phase image features. In the next sections, we outline the details of the developed methods in terms of the features being used.

Segmentation and enhancement of bone surfaces using image intensity and gradient information

Initial work on segmenting bone surfaces from US data has focused on the use of image intensity and gradient information. Since the shadow region is an important indication of bone surface presence in the acquired US data, this information was later incorporated, as an additional feature, into the proposed segmentation methods to improve the accuracy and robustness of the segmentation. In their work, Kowal et al.³⁹ used a combination of depth-weighted thresholding, image morphology and connected component labeling. The proposed method was validated on scans obtained from bovine and porcine phantoms. A mean accuracy of 0.42 mm and a 0.8-second average processing time for each US image frame was reported. Daanen et al.⁴⁰ proposed a multistage framework. During the first stage, the B-mode US intensity image was masked with an Otsu-thresholded image to create a new image called fuzzy intensity image (FII). FII results in the enhanced representation of the high-intensity features belonging to the bone surfaces in the acquired US images. The obtained FII was used during the second stage of the algorithm which used the fact that all the US signal is reflected back from the bone surface creating the shadow region. Bone structures were segmented, from the FII, by scanning each column in the image and extracting the final pixel before the intensity drop. This extracted surface was further optimized by using the information that the bone surfaces appear as continues features in the image. The method was validated on three different cadavers and in-vivo patient scans. Results were compared with manual expert segmentation. For patients, the maximum mean error was 8.8 pixels, with a pixel size of 0.112 mm × 0.109 mm, while the minimum mean error was 4.545 pixels. For cadavers, the maximum mean error was 4.056 pixels and the minimum mean error was 2.679 pixels. The time to delineate one image was less than 4 seconds. Jain et al.³⁷ proposed a Bayesian probabilistic framework converting the B-mode US images to a probability image where the bone surfaces were represented with higher values. Qualitative results on in-vitro scans, obtained by scanning a female pelvic bone immersed inside a water tank, were reported. Quantitative validation was not reported due to the fact that the authors were interested in using the enhanced images as an input to a registration method. Foroughi et al.⁴¹ introduced a method based on dynamic programing. The two-stage segmentation method involved enhancement of US images using intensity and shadow region information followed by a cost function which was iteratively optimized to segment the bone surfaces. The proposed method was evaluated on cadaver scans and achieved an accuracy of less than 0.3 mm with a computation time of 0.55 seconds. Lopez-Perez et al.⁴² segmented elongated bones such as ulna, humerus or clavicle using an active snake models. Phantom data validation achieved a surface reconstruction error of 1.16 mm. Masson-Sibut et al.⁴³ proposed a three-stage bone segmentation algorithm. The first stage involved vertical gradient image filtering followed by intensity thresholding. The second stage was used to eliminate false positive points by using the continuity information of bone surface. The final stage consisted closing the extracted bone contour using least square polynomial approximation. In-vivo volunteer scans were collected for quantitative validation. The reported maximum the RMSE was 0.8 mm with a processing time of 10–15 images per second.

Some groups have combined segmentation techniques with intra-operative registration of CT to US^21,44–46. The purpose of these studies was to overcome the limitations of US image segmentation by fusing CT datasets where bone is more easily identified. Amin et al.²¹ combined three different image features: high-intensity feature indicating bone surface presence, edge feature obtained from the bone shadow region by using a directional edge detector and a spatial prior obtained from the CT scan. Instead of explicit segmentation of the bone surfaces from the US data, the three features were incorporated into the registration method and optimized at each iteration. No accuracy results for the bone surface segmentation were provided; rather, registration error results were reported which were 1.94 mm for average translation and 0.90° for average rotation²¹. Brendel et al.⁴⁴ used only an adaptive depth gain compensation method where high-intensity features appearing in the middle of the US image were enhanced. Bone surfaces from CT data were segmented by using the physics US bone imaging. The mean registration accuracy, obtained from in-vivo human lumbar spine scans, was 0.5° and 0.5 mm. In the study by Ionescu et al.⁴⁵, the bone surface was extracted using linear filtering followed by watershed-based image segmentation. This segmentation is then updated during the US-CT registration process. Reported mean accuracy for in-vivo vertebrae surface registration was lower than 2° and 0.87 mm. Beitzel et al.⁴⁶ proposed the use of image thresholding, morphology, connected component labeling and cubic B-spline smoothing operations, respectively for segmenting the bone features from US data. This segmentation was updated iteratively during the registration process. Bone detection validation on cadaver scans achieved a mean root mean square distance (RMSD) of 0.38 mm between the automatic and manual segmented surfaces. The C++ implementation using Open CV achieved a processing time of 20–30 milliseconds. Yan et al.⁴⁷ used a backward tracing method making use of the US shadow regions for surface extraction. The backward tracing method traversed shadow regions until it reached the high-intensity bone pixels which were selected for segmentation. The extracted bone surfaces were used during a subsequent CT-US registration method. Phantom and cadaver validation experiments yielded target registration errors (TRE) of under 1 mm for the phantom and under 1.6 mm for the cadaver scans. Registration time was 2 minutes but the computation time for US bone surface extraction was not reported. Penney et al.⁴⁸ enhanced bone surfaces using a measure based on the high pixel intensity, representing to the bone surfaces, and the ratio of the number of zero pixels between the pixel of interest and the last pixel of the corresponding column to the total number of pixels between them. The main idea was based on the fact that both of these measurements will result in high values for bone features because of the large difference in acoustic impedances between soft tissue and bone. Registration was performed on scans obtained from cadaver experiments. The reported mean TRE was 2.3 mm. Bone surface localization accuracy was not reported. Rasoulian et al.⁴⁹ used the same method as that by Forough et al.⁴¹ for segmenting the spine bone surfaces from US data. However, the Gaussian filter, part of the original algorithm, was replaced by sticks filter followed by anisotropic diffusion filtering. Phantom and sheep cadaver experiments resulted in a mean TRE of 1.99 mm and 2.2 mm, respectively. Bone localization error was not reported. In a recent work, Baka et al.⁵⁰ proposed a machine learning-based approach where the intensity-based features were used as an input to a two different classifiers to create probability maps showing the expected bone interface in the collected US images. During the second step, bone surfaces were segmented using a dynamic programing approach similar to that by Forough et al.⁴¹ The reported segmentation recall and precision rates were 0.87 and 0.81, respectively.

Segmentation and enhancement of bone surfaces using image local phase information

Image phase information is a key component in the interpretation of a scene as it conveys more information regarding the image structure than magnitude information⁵¹. Figure 4 demonstrates the importance of image phase for bone US data. Investigating the reconstructed images (Fig.4c,d), we can see that the dominant structure in the image corresponds to the one where the phase information was combined (Fig. 4). The use of local phase image information in the context of segmentation and/or enhancement of bone structures from US data was first proposed by Hacihaliloglu et al.⁵² and extensively investigated in their consecutive publications^53–56. In their initial work, a 2D Log-Gabor filter was used to extract the image phase information which was combined into a feature descriptor named phase symmetry (PS) in order to extract bone features⁵². The method was validated on phantom and in-vitro experiments and achieved a mean localization error of 0.4 mm with a processing time of 0.5 seconds. The same method was also used for the localization of bone fractures⁵². This work was later extended for segmenting 3D bone surfaces from volumetric US data and was validated on clinical scans as well^53,54. In further works of Hacihaliloglu and colleagues^55,56, local phase information was used to construct an image descriptor called local phase tensor (LPT) which enhanced spine bone surfaces from US data. Validation was performed by registering a statistical shape model (SSM) of the spine to the enhanced bone surfaces. The overall reported target registration error (TRE) was 0.2 mm ± 0.4 mm where the processing time for a 2D US image was 0.05 seconds.

Figure 4.

Importance of image phase information for image reconstruction. (a) In-vivo 2D B-mode US image of human femur bone. (b) Different in-vivo 2D B-mode US image of the human femur bone obtained by changing the US transducer position and imaging setup. (c) New reconstructed image obtained using the phase information from image (b) and magnitude from image (a). (d) New reconstructed image obtained using the phase information from image (a) and magnitude from image (b). Investigating (c) and (d) we can clearly see that the dominant feature corresponds to the image from where the phase information was taken.

One of the important aspects of local phase-based image enhancement is the optimization of the filter parameters. Hacihaliloglu et al.^53,57 achieved this using the information obtained from the image domain. Optimum filter parameters improved the localization error results and the robustness of the algorithm to typical US imaging artifacts^53,57. Since the filtering is performed in the frequency domain, Anas et al.⁵⁸ proposed a framework for the optimization of the filter parameters using the information extracted from the frequency domain. Phantom validation experiments resulted with mean localization error of 1.08 mm. The same method was recently extended to 3D for processing volumetric US data⁵⁹. Cadaver validation achieved an average mean surface and Hausdorff distance errors of 0.7 mm and 1.8 mm, respectively⁵⁹.

Hybrid approaches

Hybrid approaches, where local image phase information was combined with image intensity information, were also proposed for extraction of bone surfaces from US data^60,61. Jia et al.⁶⁰ combined local image phase information with local image intensity information and proposed a dynamic programming-based segmentation method. Validation was performed on in-vivo volunteer scans obtained from the greater trochanter area of the femur bone. The average Euclidean distance error, of the proposed method, was 0.12 mm with a processing time of 4.1 seconds. Berton et al.⁶¹ combined image intensity, gradient and shadow region features with local phase image features and local binary patterns. The obtained image features were used as an input to a classification method segmenting the image into three regions: bone, soft tissue and acoustic shadow. Validation results obtained from in-vivo volunteer scans achieved a dice similarity coefficient values of 85%, 91% and 92% for bone, acoustic shadow and soft tissue, respectively. Quader et al.⁶² combined local phase information with local signal attenuation and bone shadowing in order to create a bone descriptor called confidence in PS (CPS). The proposed method was validated on ex-vivo phantom and in vivo human data. The localization error for phantom and in-vivo data were 0.54 mm and 1 mm, respectively. The reported computation time was 0.26 seconds. Ozdemir et al.⁶³ combined local phase information with local patch image intensity statistics, shadowing, attenuation, local binary patterns and speckle statistics. The extracted features were used during a Markov Random Fields (MRF)-based graph modeling approach for segmentation of bone surfaces. In-vivo validation results revealed a root-mean-square error of 0.59 mm with a computation time of 2 minutes.

DISCUSSION

Specific imaging advantages such as, real-time, non-radiation-based 3D imaging make US a perfect candidate for CAOS procedures. However, a wide range of imaging artifacts, low SNR and bone surfaces appearing several mm in thickness has limited the applicability of this imaging modality in orthopedic procedures. Furthermore, a small field of view complicates the interpretation of the acquired data. In order to overcome these problems, a number of studies have attempted to segment the bone surfaces from US data. The segmented bone surfaces were then registered to a pre-procedure plan developed using CT or MRI imaging modalities. Since accurate and robust segmentation proved to be a difficult process some researchers have focused on the enhancement of the bone surfaces which were subsequently registered to the pre-procedure plan. The registration process also provided a solution to the small field of view problem. Despite the successful results, of the developed segmentation algorithms, integration of US into different CAOS procedures continues to be a challenging process for the clinicians and technicians.

In order to make US an essential imaging modality in orthopedics, several important criteria have to be satisfied. The most important criterion is that a clinically acceptable accuracy and robustness of the segmentation method needs to be ensured. Since the extracted surfaces are used during the subsequent registration process, the segmentation accuracy needs to be below the required clinical accuracy of the computer-guided procedure. Another important component during assessment studies is the correct dissemination and reproducibility of the studies. One of the major missing components in US bone segmentation research is that there are still no publicly available datasets on which different algorithms could be compared. A direct comparison of the developed methods is difficult since there are different metrics used for validation of the segmented surfaces to manual segmentation results. One solution to this could be a joint effort in order to construct a publicly available database. In order to make US part of a standard of care imaging modality in orthopedics, a greater level of standardization in the validation process and the construction of a globally acceptable dataset consisting of clinically collected scans are required. Of secondary, but still critical, importance is the design and associated effectiveness of the US-guided system. Potential manual interactions during US data processing need to be reduced to minimum or better completely removed. The spotlight should be the surgery and not the image acquisition or segmentation process. In addition to the automation process, the computation time of the segmentation should not delay the surgical procedure. Promising results for general medical image processing employing GPU computation power^64,65 give hope that identical applications applied to US bone segmentation could achieve dramatic improvement in the processing time. The final criterion is the assessment of the proposed methods. The developed algorithms need to be validated in surgical settings using data collected from in-vivo patient studies.

Some of the papers reviewed in this work have tried to address the aforementioned criteria with their algorithms. However, only few methods fulfill these criteria completely. Finally, most of the developed methods have focused on a single anatomical bone region and have incorporated the distinct appearance of the imaged surface into their framework in order to improve robustness and accuracy. A general framework applicable to segment any bone surfaces is still an open research question.

In recent years, researchers have been looking into designing segmentation or enhancement methods based on the information extracted from the raw RF US data⁶⁶. Although access to the RF data is currently only available in US machines with dedicated research interfaces, more companies are providing access to the signal domain, and it appears that this new and rich information will be extensively sought in the coming years. Elastography or strain imaging has also been investigated by various groups^67,68. Despite the fact that deeper tissue displacements are harder to detect, the results are promising for bone enhancement or segmentation of superficial bone surfaces such as radius, ulna or tibia. In a recent work, Zhuang et al.⁶⁹ have proposed a new beamforming approach where they have showed that the proposed phase factor-based beamforming results in the suppression of noise and enhancement of phantom bone features. Although the results were only validated on phantom data, investigating new beamforming techniques for enhancement of specular surfaces is an encouraging direction. Finally, we also would like to mention that methods based on deep learning for medical image segmentation have achieved very successful results^70,71. Investigation of similar methods for the segmentation or enhancement of bone features from US data is also a new promising research direction.

CONCLUSIONS

Accurate and robust segmentation of bone surfaces or the enhancement of bone surfaces from acquired US data plays an important role in US-based orthopedic procedures. This paper presented an extensive review of the current US bone segmentation and enhancement methods developed for various orthopedic applications. In order to avoid direct comparison, algorithms were categorized into three groups: algorithms using image intensity and gradient information, algorithms using local image phase information and hybrid approaches. Techniques belonging to the hybrid group have gained popularity as they link the strengths of both intensity and phase-based methods. It should be noted that the segmented surfaces are used to register a pre-procedure plan, developed from CT or MRI, to the acquired US scans and are not used as standalone images for guidance during the orthopedic procedures. Therefore, some of the developed methods have focused more on the success of the registration method rather than the bone localization accuracy. Accordingly, we also reviewed studies where segmented or enhanced bone surfaces were used for registration to a pre-operative modality. However, in order to fully characterize the overall guidance error of the developed image guidance system, bone surface localization error should be assessed. We finalize our review by discussing some of the recent works, investigated by various groups, which provide encouraging results and new directions for the incorporation of US imaging into various orthopedic procedures.