Topomorphologic Separation of Fused Isointensity Objects via Multiscale Opening: Separating Arteries and Veins in 3-D Pulmonary CT

Abstract

A novel multiscale topomorphologic approach for opening of two isointensity objects fused at different locations and scales is presented and applied to separating arterial and venous trees in 3-D pulmonary multidetector X-ray computed tomography (CT) images. Initialized with seeds, the two isointensity objects (arteries and veins) grow iteratively while maintaining their spatial exclusiveness and eventually form two mutually disjoint objects at convergence. The method is intended to solve the following two fundamental challenges: how to find local size of morphological operators and how to trace continuity of locally separated regions. These challenges are met by combining fuzzy distance transform (FDT), a morphologic feature with a topologic fuzzy connectivity, and a new morphological reconstruction step to iteratively open finer and finer details starting at large scales and progressing toward smaller scales. The method employs efficient user intervention at locations where local morphological separability assumption does not hold due to imaging ambiguities or any other reason. The approach has been validated on mathematically generated tubular objects and applied to clinical pulmonary noncontrast CT data for separating arteries and veins. The tradeoff between accuracy and the required user intervention for the method has been quantitatively examined by comparing with manual outlining. The experimental study, based on a blind seed selection strategy, has demonstrated that above 95% accuracy may be achieved using 25–40 seeds for each of arteries and veins. Our method is very promising for semiautomated separation of arteries and veins in pulmonary CT images even when there is no object-specific intensity variation at conjoining locations.

I. Introduction

Over the last few decades, multilayered extraction of knowledge embedded in 2-D and higher dimensional images has remained a frontline research topic [1]–[7]. We report a new multiscale topomorphologic opening method for separating two isointensity objects fused at different locations and scales due to limited resolution. One possible application may be the separation of vertebrae or other bone structures via low resolution computed tomography (CT) or magnetic resonance (MR) imaging of musculoskeletal systems [8], [9]. Other examples include tracking living cells in optical microscopic video imaging [10] or tracking moving objects/subjects in video imaging where multiple entities get mutually partially occluded from time to time. In this paper, we concentrate on the application of the proposed method to separating arteries and veins (A/V) in in vivo pulmonary CT images acquired with no blood-pool enhancing contrast. The method, while very general, will be presented in the context of this application.

The challenges in separating arteries and veins are multifolded including: 1) A/V are indistinguishable by their intensity values in noncontrast pulmonary CT images; 2) often, there is no trace of intensity variation at locations of fused arteries and veins; 3) there is complex and tight coupling and fusion between A/V trees with arbitrary and multiscale geometry, especially at branching locations; and 4) limited SNR and relatively low resolution are typical for in vivo CT imaging. Patient-specific structural abnormalities of vascular trees further complicate the task. Several works have been reported in literature addressing the task of separating arteries and veins using improvised image acquisition techniques; a thorough discussion on difficulties of such approaches, especially for smaller vessels, have been presented by van Bemmel et al. [11]. To the best of our knowledge, only a few postprocessing methods have been published for separating arteries and veins [11], [12]. The previous methods have only been applied to MR data and did not use morphological scale information. These methods primarily rely on intensity variations or the presence of edge information at the adherence locations between A/V trees and may not work for in vivo CT images where no such intensity variations are present at locations of adherence. Recently, two approaches for artery/vein classification from pulmonary CT images have appeared [13], [14]. The attempted task of A/V separation is of substantial importance since separated A/V trees may significantly contribute to our understanding of pulmonary structure and function, and immediate clinical applications exist, e.g., for assessment of pulmonary emboli. Separated A/V trees may also significantly boost performance of airway segmentation methods for higher tree generations.

Here, we amalgamate fuzzy distance transform (FDT) [15]—a morphologic feature—with topologic fuzzy connectivity [16]–[20] and a new morphologic reconstruction algorithm to develop a multiscale topomorphologic opening algorithm for separating two isointensity objects fused at different locations and scales. Distance transform (DT) [21]–[26], a shortest distance measure from the background, has been popularly adopted in many digital geometric applications [27] of binary objects. Borgefors [24]–[26] extensively studied DTs for binary 3-D objects and presented the integer-valued approximation of the optimal local step lengths that are widely used by others. Local step optimization and accuracy of DT have further been studied by other research groups [28]–[30]. Saha et al. [15] introduced a generalized fuzzy DT (FDT) and its applications have been reported in [31]–[35]. Fuzzy topology and connectivity [16], [17] have become a popular segmentation approach [17], [36]–[39]. Udupa and Samarasekera presented an elegant computation solution [17] for fuzzy connectivity and made several other fundamental contributions [17], [18], [20], [37], [40], [41] to this research area.

The proposed method for multiscale topomorphologic opening starts with a fuzzy segmentation of the assembly of two isointensity objects (the arterial and venous trees in our case), and two sets of seed points (one for each object). The method outputs spatially separated arterial and venous trees. It is designed under the assumption that fusions of arteries and veins are locally separable using a suitable morphological opening operator. The method uses a novel approach to solve the following two fundamental challenges: how to find local size of morphological operators and how to trace continuity of locally separated regions. These challenges are met by combining FDT, a morphologic feature with a topologic fuzzy connectivity, and a new morphological reconstruction to iteratively open finer and finer details starting at large scales and progressing toward smaller scales. In short, artery/vein separation is modeled as a new multiscale opening task and the method provides a unique solution that offers a new research avenue of multiscale topomorphologic operations. The method effectively employs efficient user intervention at locations where the assumption of local morphological separability does not hold due to imaging ambiguity or other reasons.

The artery/vein separation work reported here is dealing with the topological aspects of the vascular tree analysis—with the arterial and venous object “detection.” The reported work offers a solution to the problem that arises after the vascular tree is detected as a whole, e.g., using our earlier work [42] and before the venous/arterial wall surfaces can be detected with high positioning accuracy, e.g., using our recent work [43].

II. Theory and Methods

First, let us define the segmentation task addressed here and let us indicate the fundamental challenges that our method is designed to solve. Consider two isointensity cylindrical objects with significant mutual overlap, as illustrated in Fig. 1(a). Often, the intensity variations at fusions may not be a reliable feature to separate the two structures. On the other hand, the two structures may frequently be locally separable using a suitable morphological opening operator. The questions are how to determine local size of suitable morphological operators and how to combine the locally separated regions. The process is initiated by picking two seeds, one for each object, as shown in Fig. 1(b), and continues by exploring connectivity on the FDT image. Using the knowledge that the two seeds represent two different objects, a threshold may be selected on the FDT image that barely disconnects the two objects in the FDT image. Essentially, the FDT threshold indicates the radius of the optimal erosion operator separating the two cylinders [see Fig. 1(b)]. The immediate next question is how to proceed with the separation to next lower scales. The difficulty here is that the scale of the annular remainder [see Fig. 1(b)] of the morphological erosion is at least equal to that of the regions not yet separated. This difficulty is overcome using a proposed morphological reconstruction operation that fills the annular remainder while maintaining the separate identities of the two objects [see Fig. 1(c)]. This step allows proceeding with separations at a lower scale and the method progresses iteratively separating fused structures at finer and finer scales starting with the larger one. In short, the method may be summarized by the following three steps:

Fig. 1.

Schematic description of the segmentation problem addressed by the proposed multiscale topomorphologic opening method and the challenges met at different steps.

• Step 1) Find the optimum morphological opening structure scale separating the two objects specified by two sets of seeds [see Fig. 1(b) and Fig. 3(d)].

• Step 2) Block the annular remainder around currently separated regions enabling morphological separations at lower scales [see Fig. 1(c) and Fig. 3(e)].

• Step 3) Repeat steps 1 and 2 until convergence.

Fig. 3.

Illustration of intermediate results at different steps of the multiscale topomorphologic opening. (a) 3-D rendition of a computer-generated phantom representing two isointensity objects fused at various scales and locations. (b) Few cross-sectional images of the phantom. (c) FDT image on the central xy plane. Intuitively, it is a union of two FDT maps, one for each cylinder. Although the cylinders are locally separable in the FDT image, no global thresholding serves the purpose. (d) Results of separation of two cylinders after first iteration using FDT-based connectivity. (e) Morphologic reconstruction on the result shown in (d). Regions marked in cyan (or yellow) represent the expansion of the red (respectively, blue) object after morphological reconstruction. (f) and (g) Same as (d) after second (f) and terminal (g) iterations. (h) and (i) 3-D rendition and cross-sectional images of the final result.

The overall workflow diagram of the proposed method is presented in Fig. 2. As input, it receives fuzzy segmentation of the assembly of two isointensity objects and two sets of seeds—one for each object—and it outputs the separated objects. Although the method immediately extends to multiple objects, here, we formulate a solution for two objects only and will refer to one object as the “rival” of the other. Let us consider an image consisting of two isointensity fuzzy objects with significant noise that overlap at different scales, as shown in Fig. 3(a); a few cross-sectional images are shown in Fig. 3(b). The two cylinders with gradually reducing radii are running in parallel. The diameter of one cylinder is significantly larger than that of the other; a sinusoidal swing is added to both cylinders in the xy-coordinate direction so that centerlines of both cylinders lie on one xy plane, say the central xy plane. Usefulness of the method may be better understood in three and higher dimensions. On the other hand, it is always easier to illustrate a method in two dimensions. Therefore, we show results of different steps of the method on the central xy plane first. One seed is identified at the approximate center of each cylinder at the topmost cross section. As shown in Fig. 2, the process is iterative and begins with marking the seeds followed by FDT computation [15] of the assembly of two cylinders prior to iterations; see Fig. 3(c) for the results of FDT computation. It is visually notable from the FDT image that we cannot find a single FDT threshold to separate the two cylinders from their head to tail. Therefore, no straightforward morphological opening operator can solve the problem. On the other hand, over a small region, the two cylinders are separable using their FDT values. In other words, the problem demands regional selection of optimal opening structures to separate the two cylinders.

Fig. 2.

Modular representation of the multiscale topomorphologic opening algorithm separating two isointensity objects fused at different scales and locations.

The aforementioned observation motivated us to use local scale to normalize FDT values. This approach reduces the effect of spatial scale variations. Local scale at a point p is defined as the FDT value of the locally deepest point (a point with locally maximum FDT value) that is nearest to p. With the normalized FDT map, the method adopts an iterative strategy that first separates the cylinders over large-scale regions using FDT-based relative connectivity [18], [20] where a point becomes associated with an object if its connectivity to the point is strictly greater than that of its rival. A separator is built between the two objects using a morphological reconstruction method that simultaneously and radially dilates each currently segmented region until blocked by its rival (maximum radius of the dilating structure is determined by FDT values). Fig. 3(d) and (e) shows the results of initial separation and morphologic reconstruction of two cylinders after the first iteration. In the next iteration, the FDT-connectivity paths of one object are not allowed to enter into the region already assigned to its rival. This strategy facilitates object separation at smaller scale regions [see Fig. 3(f)]. The iterative process is continued as long as any change is detected step to step. For this phantom image, the method stopped after 12 iterations (see Fig. 3(g)–(i) for final separation). In the remainder of this section, we formally describe the theory and algorithms.

A. Basic Definitions and Notation

A 3-D cubic grid, or simply a cubic grid, is represented by the points in , where is the set of all integers. A grid point, often referred to as a point or a voxel, is represented by a triplet of integer coordinates. Two points p, q ∊ are called adjacent if and only if α(p, q) = 1, where α is a reflexive and symmetric relation. Standard 26-adjacency [44] is used here for α, i.e., two points (x₁, x₂, x₃) and (y₁, y₂, y₃) in are adjacent if and only if max_1≤i≤3, |x_i − y_i| ≤ 1, where |·| denotes the absolute value. Two adjacent points are often referred to as neighbors of each other; the set of 26-neighbors of a point p excluding itself is denoted by (p). An object is a fuzzy subset {(p, (p))|p ∈ } of , where : → [0, 1] is its membership function. An object may be derived from an acquired image [45] using a suitable image segmentation algorithm [7] that has been extensively studied for decades. Following the main focus of the theory and methods addressed in this paper, we start with the definition of objects instead of images. The support Θ( ) of an object is the set of all points with nonzero membership, i.e., Θ( ) = {p p ∈ and (p) ≠ 0}. Images are always acquired with a finite field of view. Thus, we will assume that an object always has a bounded support. Let S denote a set of points; a path π in S from p ∈ S to q ∈ S is a sequence 〈p = p₁, p₂, …, p_l = q〉 of points S in such that every two successive points on the path are adjacent. A link is a path 〈p, q〉 consisting of exactly two mutually adjacent points p, q ∈ .

B. FDT

DT is a local depth measure inside an object and we call it “FDT” when the object representation is fuzzy. FDT [15] accounts for both partial voxel occupancy and spatial heterogeneity of an object distribution. The length of a link 〈p, q〉 is calculated as 1/2( (p) + (q)||p − q||, where ||·||, denotes any Euclidean L2 norm. The length of a path π = 〈p₁, p₂, …, p_l〉 in a fuzzy object , denoted by , is defined as the sum of lengths of all links along the path, i.e.,

$${\mathrm{\Pi }}_{\mathcal{O}}(\pi )=\sum _{i=1}^{l-1}\frac{1}{2}({\mu }_{\mathcal{O}}(p_i)+{\mu }_{\mathcal{O}}(p_{i+1}))\Vert p_i-p_{i+1}\Vert .$$ (1)

Between any two points p, q ∈ , infinitely many paths may exist; let (p, q) denote the set of all paths from to p to q. A path π_p,q ∈ (p, q) is one of the shortest paths from p to q in , if and only if, (π_p,q) ≤ (π) for all π ∈ (p, q). It may be noted that the shortest path between two points p, q ∈ may not be unique. The fuzzy distance from p ∈ to q ∈ in an object , denoted by (p, q), is the length of one of the shortest paths from p to q, i.e.,

$${\omega }_{\mathcal{O}}(p,q)=\underset{\pi \in \mathcal{P}(p,q)}{min}{\mathrm{\Pi }}_{\mathcal{O}}(\pi ).$$ (2)

It has been shown in [15] that fuzzy distance satisfies the metric properties in both continuous as well as discrete spaces in any dimension. The FDT of an object is represented as an image {(p, (p))|p ∈ }, where : → ℜ⁺ denotes the FDT value at a given point, ℜ⁺ is the set of positive real numbers including zero, and (p) is defined as the fuzzy distance between p and its nearest points in Θ̄( ) = − Θ( ). In other words

$${\mathrm{\Omega }}_{\mathcal{O}}(p)=\underset{q\in \overline{\mathrm{\Theta }}(\mathcal{O})}{min}{\omega }_{\mathcal{O}}(p,q).$$ (3)

C. Local Scale and FDT Normalization

Intuitively, “local scale” at a point p is defined as the FDT value (or, the ordinary DT [26] value for binary images) at the locally deepest point that is nearest to p. Toward this aim, we first define “nearest point” as follows. Let S ⊂ Θ( ) denote a set of points and let p ∈ Θ( ) be a point inside the support of a fuzzy object . The nearest point of p in S over an object , denoted by N (p, S), is the point q ∈ S such that ∀r ∈ S, (p, q) ≤ (p, r); in case there are multiple such q’s, the point with the largest FDT value is chosen. Let S_max ⊂ Θ( ) be the set of points with locally maximal FDT values in , i.e., S_max = {p p ∈ Θ( ) and ∀q ∈ (p), (q) ≤ (p)}, where (p) is the (2l+1) × (2l+1) × (2l+1) neighborhood of p; in this paper, we have used (p) to avoid noisy local maxima. Finally, local scale at a point p in an object , denoted by (p), is defined as the FDT value of the local maximal point nearest to p, i.e., (p) = (N (p, S_max)).

In order to reduce the effects of spatial variation in object scales, we normalize the FDT value at each point p by dividing the original FDT value by the local scale, i.e., dividing (p) by (p). Note that the normalized FDT values always lie in the interval [0, 1]. This step of normalizing the FDT value is simple yet effective in reducing the artifacts of scale variation in an object. Although this step causes no change in the FDT map for phantom images where local radii change monotonically along the axis of each structure [see Fig. 3(c)], it is useful when local radii vary nonmonotonically along an object [e.g., arteries and veins in Fig. 5(d) and Fig. 6(d)]. Essentially, FDT normalization is equivalent to equating the height of all local maxima of the original FDT map. It can be shown that by this process, all local minima and valley lines of the original FDT map are preserved and may thus be exploited as separating features by the multi-scale topomorphologic opening algorithm. Therefore, all separations of the original FDT map will be preserved in the normalized FDT map. The major advantage of the FDT normalization is that it reduces the number of iterations in most real applications. In the rest of this paper, by “FDT,” we will refer to the “scale-normalized FDT” and (p) will denote the scale-normalized FDT value at p unless stated otherwise.

Fig. 5.

Results of application of the method to a human in vivo pulmonary multidetector CT image. (a) Coronal image slice from a thoracic CT image of a 22-year-old female. (b) Fuzzy segmentation of vasculature. (c) and (d) Original and local-scale normalized FDT maps of the vasculature. (e) 3-D surface rendition of left and right pulmonary vascular trees. (f) Color-coded 3-D rendition of separated arterial and venous trees computed by the proposed method.

Fig. 6.

Same as Fig. 5 but for another human subject.

D. Fuzzy Morphoconnectivity

This step is essentially the same as used when determining the fuzzy connectivity [16], [17] except that the FDT values are used instead of the intensity values. We follow the formulation of fuzzy connectivity method introduced by Rosenfeld [16]. Fuzzy morphoconnectivity strength of a path π = 〈p₁, p₂, …, p_l〉 in a fuzzy object , denoted by (π), is defined as the minimum FDT value along the path

$${\mathrm{\Gamma }}_{\mathcal{O}}(\pi )=\underset{1\le i\le l}{min}{\mathrm{\Omega }}_{\mathcal{O}}(p_i).$$ (4)

A path π_{p, q} ∈ (p, q) is one of the strongest morphological paths from to p to q, if and only if, (π_{p, q}) ≥ (π) ∀π ∈ (p, q). The fuzzy morphoconnectivity between two points p ∈ and q ∈ , denoted by (p, q), is the strength of one of the strongest morphological paths between p and q

$${\gamma }_{\mathcal{O}}(p,q)=\underset{\pi \in \mathcal{P}(p,q)}{max}{\mathrm{\Gamma }}_{\mathcal{O}}(\pi ).$$ (5)

Intuitively, fuzzy morphoconnectivity between two points p, q ∈ indicates the size of the morphological erosion operator that barely disconnects the two points.

E. Object Region Detection

This step is accomplished by comparing the fuzzy morphoconnectivity values for the two target objects and similar to that presented in [18], [20], and [41]. Specifically, the region acquired by , denoted as , is defined as

$$R_{\mathcal{A}}=\left\{p\mid \underset{a\in S_{\mathcal{A}}}{max}{\gamma }_{\mathcal{A}}(a,p)>\underset{b\in S_{\mathcal{B}}}{max}{\gamma }_{\mathcal{B}}(b,p)\right\}$$ (6)

where and are the sets of seeds for objects and , respectively. It may be noted that we need to specify the FDT maps and for the two objects and in order to complete the definition of and similarly the definition of . In this paper, at the beginning of the algorithm, the two FDT maps and are both assigned to the FDT map computed from the assembly of the two fuzzy objects. However, the theory and algorithms work equally well even if we initiate the algorithm with two different FDT maps computed from two different representations of the assembled objects, each representation emphasizing one over the other. It allows taking advantage of features that partially distinguish the two objects. After every iteration, the values of and are updated following the strategy described in Section II-G.

F. Morphological Reconstruction

Two regions and , defined in the previous section, provide a separation of the two objects based on fuzzy morphoconnectivity analysis; an output of this step is illustrated in Fig. 1(b) and Fig. 3(d). Here, the idea is to morphologically extend the two separated regions and to fill the annular space around them without compromising their separation; results of this step are illustrated in Fig. 1(c) and Fig. 3(e). In order to define the morphological reconstruction process, we first introduce the notion of “morphological neighborhood.” The morphological neighborhood of a set of points X in an object , denoted by (X), is a set of all points p ∈ Θ( ) such that for some q ∈ X, (p, q) < (q) and p is connected to q by a path π = 〈p = p₁, p₂, …, p_l = q〉 of monotonically increasing FDT values. It may be noted that the original FDT map is used here without scale normalization as morphological neighborhood is defined on the basis of original geometry and size of a local structure. Finally, (or ) is extended by the set of points p ∈ ( ) [respectively, points p ∈ ( )] that are strictly closer to (respectively, ) than (respectively, ). Results of this step are illustrated in Fig. 1(c) and Fig. 3(e). It may be noted that gaps between the separated regions visible in Fig. 3(d) are filled in Fig. 3(e) after the morphological reconstruction. This step is important as it stops paths running through those gaps that would be creating problems during separation of smaller scale regions in subsequent iterations. Let ( ) and (R_B) represent morphological reconstructions for and , respectively.

G. Modification of FDT Maps and Seeds

This step is similar to the iterative strategy described in [18] and [41] for intensity-based fuzzy connectivity. For each of the objects, we set the FDT values to zeros over the region currently acquired by its rival object. This step stops paths from one object to run through the acquired territory of its rival object. Thus, a hypothetical wall is created between the two regions currently separated. Specifically, the FDT images of the object are updated after each iteration as follows:

$${\mathrm{\Omega }}_{\mathcal{A}}(p)=\{\begin{array}{ll}0,\hfill & \text{if}p\in N_{\mathcal{O}}(R_{\mathcal{B}})-M_{\mathcal{O}}(R_{\mathcal{A}})\hfill \\ {\mathrm{\Omega }}_{\mathcal{O}}(p),\hfill & \text{otherwise}.\hfill \end{array}$$ (7)

The FDT map of the other object is updated similarly. The seeds and for the two objects are replaced by and , respectively. With this setup, the algorithm enters into the subsequent iteration, i.e., steps described in Sections II-D–II-G are repeated as illustrated in Fig. 2.

Algorithms

Algorithms for computing FDT and fuzzy connectivity have been presented in several papers including [15], [17], [18], and [37]. The algorithmic solutions for local-scale computation, FDT normalization, object region detection, and modification of FDT maps and seeds are trivial following their descriptions in Section II-C, II-E, and II-F. However, the algorithmic realization of morphological neighborhood computation in the context of morphological reconstruction is not straightforward from the descriptions in Section II-F. Therefore, a pseudocode for the algorithm of computing morphological neighborhood is presented in the algorithm.

begin compute-morphological-neighborhood
input
1) a set of points X
2) a fuzzy object
     = {(p, )| p ∈  and  :  → [0, 1]}
3) original FDT map  :  → ℜ+ without scale normalization.
output
1) morphological neighborhood (X)
auxiliary data structure
1) monotonic distance transform map (p) :  → ℜ|ℜ is the set of all real numbers
2) a sequential queue Q
for each point p ∈ 
     assign  (p) = 0
for each point p ∈ X
     assign (p) = –(p)
     push p in Q
while Q is not empty
     pop a point p from Q
     for each point q ∈ (p) and (q) < (p)
           assign$\mathit{link}(p,q)=\frac{1}{2}({\mu }_{\mathcal{O}}(p)+{\mu }_{\mathcal{O}}(q))\Vert p-q\Vert$
           if (q) > (p) + link(p, q)
              assign (q) = (p) + link(p, q)
              push q in Q
(X) = {p|p ∈  and (p) < 0}
end compute-morphological-neighborhood

III. Experimental Methods and Results

A. Phantoms

Performance of the method has first been examined both qualitatively and quantitatively on mathematically generated phantoms (see Figs. 3 and 4). Six mathematical phantoms were computer-generated, each being an assembly of a pair of tubular objects running parallel across the slice direction with different levels of fuzziness, overlap, scale, and noise. Initially, the phantom images were generated at high resolution, and then, downsampled using 3 × 3 × 3, 4 × 4 × 4, and 5 × 5 × 5 windows to simulate partial volume effects. Each downsampled image was further degraded with additive random noise at SNR of 12. Using a graphical user interface, exactly two seed points were manually selected for each phantom. One seed was placed for each tubular object near its center on the slice at the largest scale level. Although there is no theoretical restriction on positioning the seed points, the seeds were positioned visually close to the centerline of each cylinder. Example phantoms and their object separation results are depicted in Fig. 4 at 4 × 4 × 4 and 5 × 5 × 5 downsampling only, as the method has always successfully separated the two cylinders at 3 × 3 × 3 downsampling. The phantom image of Fig. 3 was generated at 4 × 4 × 4 downsampling. At extremely low resolution and high noise, the morphological information may be entirely lost at a few locations leading to a failure of separations, as shown in Fig. 4(n), (t), and (v). Specifying another pair of seeds at each of these locations would have helped achieving correct separations. Cross-sectional images of the first phantom are given at both resolutions to demonstrate the 3-D phantom image quality used and the complexity of separating the two objects. The smallest radius of the phantom shown in Fig. 4(a) is 1.87 pixels after downsampling by 4 × 4 × 4 pixels; the largest radius in the same example at the same downsampling resolution is 18.75 pixels. Except for the examples in Fig. 4(a) and (f), the radii of the two cylinders differed significantly. It may be noted that at the upper part of the object colored in red in Fig. 4(c), the object visually appears disconnected. We confirm that each of the red and blue objects in Fig. 4 has been explicitly checked by a computerized algorithm and found to be a single connected component as expected according to the presented theory of the method. Results of separation of two objects in each of these phantoms were exactly the same for seed selection by two independent users, each selecting seeds visually close to the center of corresponding object at the highest scale level, demonstrating the theoretical robustness of the method with respect to seed selection.

Fig. 4.

(a)–(k) Results of applying our method to several computer-generated 3-D phantoms after 4 × 4 × 4 downsampling. (a) and (b) 3-D rendition and cross-sectional images of one phantom. (c) Separated cylinders. (d)–(k) Results for other four phantoms. (l)–(v) Results of application of the method after downsampling. Note that this level of downsampling makes separation impossible for the smallest scale features.

A qualitative analysis was performed on the six phantoms images in Figs. 3 and 4. For this purpose, the ground truth of separations of two objects in these phantoms was generated from the knowledge of their generation at the original resolution prior to downsampling. For each tubular object in a phantom, its center-line along with local radii was recorded at the time of its computerized generation. Let {(c_{i, j}, r_{i, j})|i = 1, 2; j = 1, 2, …, L_i} be the set of points and local radii along the centerline of the ith object in a phantom prior to downsampling. Let p = (x, y, z) be a point with nonzero intensity value in the corresponding phantom image obtained by downsampling at l × l × l, where l = 4 or 5. Thus, p_l = (x × l, y × l, z × l) denotes the point corresponding to p prior to downsampling. Local-radius-normalized distance of p from the centerline of the ith object, denoted by D_i(p, l), is defined as follows:

$$D_i(p,l)=\underset{j=1,2,\dots ,L_i}{min}\frac{\Vert p_l-c_{i,j}\Vert }{r_{i,j}}.$$ (8)

Here, local-radius-normalized distance is used to avoid a bias in separation due to local radius difference between two objects. Finally, in the true separation of two objects, the point p is assigned in the first object (or the second object) if D₁(p, l) < D₂(p, l) [respectively, D₂(p, l) < D₁(p, l)]. Note that in a rare situation of D₁(p, l) = D₂(p, l) (since, c_{i, j}s are points in Euclidean three space), the point is not assigned to any of the two objects.

Let T₁ and T₂ denote the true segmentations of the two objects in a phantom image computed as aforementioned, and let S₁ and S₂ denote the segmentations of corresponding objects in the same phantom image computed by the current method. True positive and false negative of the computerized separation of two objects are defined as follows:

$$\text{True}\text{positive}=\frac{(T_1\cap S_1)\cup (T_2\cap S_2)}{T_1\cup T_2}$$ (9)

and

$$\text{False}\text{negative}=\frac{(T_1\cap S_2)\cup (T_2\cap S_1)}{T_1\cup T_2}.$$ (10)

Results of quantitative analysis of true positive and false negative of separation of two tubular objects in each phantom image at different downsampling rates using our algorithm are presented in Table I. These results demonstrate that our method produces high accuracy in spatially delineation of the two structures, while its performance in terms of maintaining the identity of individual structures along their axes is qualitatively illustrated in Figs. 3 and 4.

TABLE I.

Results of Quantitative Analysis of True Positive and False Negative Separations of Two Objects Using the Proposed Multiscale Topomorphologic Opening Algorithm for Different Phantom Images at Different Downsampling Rates

	Down-sampling rate=4		Down-sampling rate=5
Phantoms	True positive	False negative	True positive	False negative
Fig.3(a)	98.00%	0.70%	93.72%	5.06%
Fig.4(a)	99.11%	0.45%	98.64%	0.74%
Fig.4(d)	98.78%	0.45%	98.36%	0.65%
Fig.4(f)	98.43%	0.50%	97.98%	0.50%
Fig.4(h)	97.54%	1.79%	97.45%	1.78%
Fig.4(j)	99.23%	0.28%	96.49%	2.82%

B. In Vivo CT Image Data

The effectiveness of the method has also been examined both qualitatively and quantitatively on clinical pulmonary multidetector CT images. A result of application of the method separating pulmonary arteries/veins in a thoracic CT image of a healthy subject is qualitatively illustrated in Fig. 5. The thoracic region of a 22-year-old female was imaged using a Siemens Sensation 64 multidetector CT scanner at 120 kVp and 100 mA. The subject was scanned in feet-first supine position. The image was acquired at 0.75 mm slice thickness and was reconstructed with 0.5 mm slice thickness and (0.55 mm)² in-plane resolution. A coronal slice image of the CT data is presented in Fig. 5(a) in which the display intensity was manually chosen to emphasize pulmonary vasculature. The entire vascular tree inside the lung region was segmented in a CT image using a previously published method by Shikata et al. [42] that uses tree connectivity on CT intensity data after processing the image with a tubular enhancement filter [see Fig. 5(e)] and produces a binary image (vessels and background). A fuzzy representation of the vascular tree [Fig. 5(b)] was generated using the following three steps—1) determine the mean and standard deviation of intensity values over segmented vascular region; 2) dilate the binary vascular region by one voxel to include partial-volume voxels; and 3) assign fuzzy membership at each voxel over the dilated vascular region following a step up Gaussian function with mean and standard deviation parameters as computed in Step 1. Seed points for each of the arteries and veins were manually selected by an expert using a 2-D-slice-display graphical interface followed by the application of the multi-scale topomorphological separator algorithm. The subsequent automated process required 2–3 min to complete the A/V separation for each of left and right lungs. Finally, separated A/V trees were displayed with color coding [red for veins and blue for arteries; see Fig. 5(f)] through a 3-D surface rendition software tool developed in-house using VTK-based 3-D visualization filter classes. Results of A/V separation from CT image data of another healthy subject’s lung are illustrated in Fig. 6.

Results of the experimental study designed to quantitatively examine the performance of the proposed method are now reported. It may be noted that for any given manual artery/vein separation (considered as the reference), it is always possible to achieve 100% accuracy with the proposed method using a sufficiently large number of seed points. It is, therefore, justified to study the performance of the method in terms of its relation between needed user intervention and accomplished accuracy or error. Specifically, we compute the accuracy/error of artery/vein separation as a function of the number of seeds used. Two CT datasets presented in Figs. 5 and 6 are used for our quantitative experiment. For each lung data, approximately 8000–10000 voxels were manually labeled as arteries and veins by an expert; henceforth, this expert input will be referred to as input_A. The purpose of input_A was to separately trace arteries and veins in pulmonary vasculature and was accomplished by clicking several points on each slice inside corresponding structures. Input_A consists of approximately 2000–2500 points for arteries and another 2000–2500 points for veins in each of the left and right lungs. The first four-five levels of branching are thus described for each of the A/V trees. It should be noted that input_A is different from the expert input used to produce the A/V separations in Figs. 5 and 6; henceforth, the later expert input will be referred to as input_B. Input_B was generated by an independent user consisting of 25–35 seeds for each of the A/V trees for each of the left and right lungs. Now, we describe a seed selection scheme based on expert’s segmentation (input_A) of arteries and veins that is considered as the reference in our experimental design. Note that our seed selection scheme is based on expert’s segmentation and designed primarily to evaluate the method. Thus, the seed selection scheme is not intended to solve problems relating to “automatic seed selection” that is a research topic on its own.

Let = {(p, (p))|p ∈ } denote the fuzzy representation of a pulmonary vascular tree and let denote the FDT map derived from . In this section, the original FDT map without any local-scale normalization is used so that the seed selection starts separating arteries and veins at larger scales, and then, progresses into finer details. Let M_artery and M_vein denote the sets of voxels manually labeled by the expert as belonging to arteries and veins, respectively; note that M_artery and M_vein are the subsets of input_A. The first arterial seed is automatically selected as the voxel in M_artery with the highest FDT value, i.e., at the largest scale value. Let A₁ denote the set of the singleton seed voxel; the subscript index indicates the iteration number. The set V₁ of singleton seed voxel for veins is defined in the exact same way using M_vein. Let $C_{\text{artery}}^1$ and $C_{\text{vein}}^1$ denote the arterial and venous regions computationally segmented from using the proposed method with A₁ and V₁ being the sets of seeds for arteries and veins, respectively. Accuracy and errors of the proposed method for single seed may be computed by comparing $C_{\text{artery}}^1$ and $C_{\text{vein}}^1$ with the reference segmentation M_artery and M_vein.

Exact definitions of accuracy and error measures are described in the following paragraph; first, let us describe the iterative progression scheme for seed selection refining artery/vein separation. Let A_i and V_i denote the sets of seeds for arteries and veins in the ith iteration, and let $C_{\text{artery}}^i$ and $C_{\text{vein}}^i$ denote the regions assigned to arteries and veins, respectively; note that each of A_i and V_i contains exactly i number of seeds. Let denote the same fuzzy object as except that membership values over $(C_{\text{artery}}^i\cap M_{\text{artery}})\cup (C_{\text{vein}}^i\cap M_{\text{vein}})$, i.e., the set of correctly segmented artery/vein voxels, are set to zero. Let denote the FDT map of . During the(i + 1)th iteration, the set of arterial seeds is A_i+1 augmented by exactly one voxel p to A_i ∪ {p} such that $p\in M_{\text{artery}}-C_{\text{artery}}^i$ and it has the highest value in ; in case of multiple voxels with the highest FDT value, one is selected arbitrarily. The set of venous seeds V_i+1 is augmented in exactly the same manner. It may be noted that the seed selection scheme adopted for the current experiment is deterministic in nature, and therefore, generates only one sequence of seeds for a given segmentation of vasculature and A/V separation.

Accuracy of the method is defined by comparing its results with manual labeling of arteries or veins in input_A. Here, we need to mention the following three facts.

1. Each of manually labeled arteries and veins M_artery and M_vein is truly a set of voxels selected by an expert using mouse cursor clicks and not a painted region. On the other hand, our computerized method generates volumetric regions $C_{\text{artery}}^i$ and $C_{\text{vein}}^i$ for arteries and veins.

2. Segmented arterial and venous trees produced by the computerized method capture fine details. On the other hand, the expert could separately trace arterial or venous trees only for larger branches.

3. A large number of voxels, approximately 8000–10000, were labeled for arteries/veins for each thoracic CT image.

Based on these three observations, we have defined segmentation performance measures in terms of sensitivity, false detection, and missed locations as follows (arterial definitions given):

$$\text{Sensitivity}=\frac{\mid M_{\text{artery}}\cap C_{\text{artery}}^i\mid }{\mid M_{\text{artery}}\mid }$$ (11)

$$\text{False}=\frac{\mid M_{\text{vein}}\cap C_{\text{artery}}^i\mid }{\mid M_{\text{artery}}\mid }$$ (12)

$$\text{Miss}=\frac{\mid M_{\text{artery}}-C_{\text{artery}}^i\cap C_{\text{vein}}^i\mid }{\mid M_{\text{artery}}\mid }.$$ (13)

Sensitivity and false detection measures of the method for the two CT datasets in Figs. 5 and 6 are presented in Fig. 7(a) and (b), respectively. Assuming that both M_artery and M_vein are entirely included via initial segmentation of composite pulmonary vasculature, the measure of miss was expected to be close to zero and the experimental results confirmed our expectation. Therefore, we omit the graphical plots for miss locations in Fig. 7(a) and (b) keeping the graphs simpler. As illustrated in Fig. 7(a), 95% overall sensitivity was achieved for the CT dataset in Fig. 5 using 27 seeds at the cost of 4.2% false detections and 0.8% miss. It required 52 seeds to accomplish 99% sensitivity at false rate of 0.6% and 0.4% miss. For the CT dataset in Fig. 6, 95% overall sensitivity was achieved using 40 seeds at the false detection rate of 3.7% and 1.3% miss, and 99% sensitivity was obtained using 66 seeds at the false detection rate of 0.5% and 0.5% miss. To convey the quality of artery/vein separation at different levels of sensitivity, we present the results for the dataset in Fig. 6 at 80%, 90%, 95%, and 99% overall sensitivity. It may be interesting to compare the results of Fig. 7(f) produced from the expert input input_A with that of Fig. 6(c) (reproduced as Fig. 7(g) to facilitate the comparison) that was produced from input_B by an independent expert. It is encouraging to see the high level of agreement between such two independent expert-driven separations of arteries and veins using the current algorithm.

Fig. 7.

(a) and (b) Graphical presentations of results of quantitative analysis comparing required user intervention with accomplished sensitivity and false detection for the pulmonary CT data presented in Figs. 5(a) and 6(b). For the example of Fig. 5, the numbers of seeds, false, and miss at 95% and 99% overall sensitivity are: 27, 4.2%, 0.8% and 52, 0.6%, 0.4%, respectively; these numbers for the example of Fig. 6 are 40, 3.7%, 1.3% and 66, 0.5%, 0.5%, respectively. (c)–(f) Qualitative illustrations of artery/vein separation at sensitivity levels of 80%, 90%, 95%, and 99%, respectively. To facilitate visual comparison, the image of Fig. 6(f) is repeated in (g).

An intriguing observation in Fig. 7(a) and (b) is that the sensitivity of separating arteries/veins may reduce despite adding more seeds. For example, in Fig. 7(b), it starts with almost 100% sensitivity for arteries, and then, falls significantly after adding more seeds before rising again. What happens here is that initially, most of the vasculature is occupied as arterial territory leaving only a small region for veins. It leads to the initial high sensitivity for arteries, while low sensitivity is observed for veins. As more seeds are added, the segmentation loses falsely acquired venous territories along with some arterial territories that were connected through those previous false acquisitions of venous territories. This leads to the reduction in both arterial sensitivity and false measures, as presented in Fig. 7(a) and (b). A high sensitivity together with a low false detection generally indicates a good segmentation of the specific structure.

IV. Discussion and Conclusion

In this paper, a fundamentally novel topomorphologic multi-scale opening operator was presented for separating two isointensity objects mutually fused at various scales and locations. Experimental results demonstrated its performance. The method seeks morphological identities of each object at a specific scale and separates them without using any gradient- or edge-like features. Introduction of the ideas of morphological reconstruction and separator allows the method to seal the joining border at the current scale, and then, seek morphological features identifying different objects at finer scales. Let us first clarify the differences between the current method and the previous state-of-the-art algorithms related to fuzzy connectivity [12], [17], [39] and FDT [15]. The prior arts of FDT and fuzzy connectivity are used here only to solve the first step of the three basic steps used by our new method, as stated in Section II. Further, the novelty of demonstrating that FDT and fuzzy connectivity together with the seeds of two objects allow automatic determination of the optimum opening structure scale separating two mutually fused objects is part of the current work. Additionally, the morphological reconstruction introduced here to block the annular remainder of separation at the current scale is a simple yet effective step allowing morphological separations at lower scales. The A/V separation paper published by Lei et al. [12] was designed for contrast-enhanced MR images and uses intensity variations at vessel walls (which are expected to exhibit lower MR signal values as compared to contrast enhanced lumen) to separate arteries and veins. However, such assumption does not hold for CT images where intensity variation at regions of fusion between arteries and veins is almost absent. To demonstrate the improvements achieved by the current method, the algorithm by Lei et al. and an algorithm with a straightforward application of FDT into iterative relative fuzzy connectivity (IRFC) framework [18] were implemented and applied to several mathematical phantoms in Fig. 4. By comparing the results of these three different methods presented in Fig. 8, it is obvious that the current method is fundamentally different from prior state-of-the-art methods. In summary, the technical contributions of the current work are as follows.

Fig. 8.

(a) Few examples of mathematical phantoms each containing two tubular objects fused at different locations and scales with significant overlap. (b) Results of separations of two objects using the algorithm by Lei et al. [12], (c) simple FDT-based IRFC [18], [20], and (d) the multi-scale topomorphologic opening algorithm reported here.

1. The current work models the A/V separation task via non-contrast CT imaging as an entirely new multiscale topomorphologic opening task.

2. Our method provides a unique solution to the problem where existing methods fail.

3. The reported method provides a practical solution to a clinically important problem.

4. The presented theory opens a new avenue of multiscale topomorphologic operations.

An application of the proposed method for separating pulmonary arteries and veins in in vivo thoracic CT images has been studied. For pulmonary CT images, the geometry of coupling between arteries and veins is very challenging and quite unknown. However, the method has shown acceptable performance with a reasonable number of seeds. Approximately 25–35 seeds were manually selected by an expert on each of the A/V subtrees for each of the left and right lungs. It may be pointed out that the seeds were selected using a 2-D graphical tool. Often, multiple seeds were placed within the same locality of an object, and therefore, not all of the seeds contributed to true A/V separation. A quantitative experiment has demonstrated that the method can yield 99% accuracy for the entire pulmonary vasculature using approximately 60 seeds selected using a computerized blind seed selection scheme.

Both qualitative and quantitative results on phantoms as well as clinical pulmonary CT data have been presented. Results have established viability of the new method in resolving multiscale fusions of two isointensity structures with complex and unknown geometry of coupling, which are often present in biological objects. The experimental results, beyond any doubt, have established usefulness of the current method in separating arteries and vein in pulmonary CT images without any blood pool contrast. We have found that the seed selection is an important task in the entire process and the current seed selection tool is far from optimal. An effective seed selection process is demanding an appropriate 3-D graphical user interface coupled with the multiscale opening operator providing a transparent interactive environment to the user. Currently, we are developing such a graphical system for efficient seed selection system with appropriate GUI and evaluating the method in terms of accuracy, reproducibility, and efficiency on a larger dataset using multiple users. We believe that a large-scale evaluation of accuracy, reproducibility, and efficiency will further demonstrate the good performance of the method. We also expect that designing an effective seed selection system with a smart GUI will become a valuable research effort in its own right.