DEEP-LEARNING STRATEGY FOR PULMONARY ARTERY-VEIN CLASSIFICATION OF NON-CONTRAST CT IMAGES

Abstract

Artery-vein classification on pulmonary computed tomography (CT) images is becoming of high interest in the scientific community due to the prevalence of pulmonary vascular disease that affects arteries and veins through different mechanisms. In this work, we present a novel approach to automatically segment and classify vessels from chest CT images. We use a scale-space particle segmentation to isolate vessels, and combine a convolutional neural network (CNN) to graph-cut (GC) to classify the single particles. Information about proximity of arteries to airways is learned by the network by means of a bronchus enhanced image. The methodology is evaluated on the superior and inferior lobes of the right lung of twenty clinical cases. Comparison with manual classification and a Random Forests (RF) classifier is performed. The algorithm achieves an overall accuracy of 87% when compared to manual reference, which is higher than the 73% accuracy achieved by RF.

1. INTRODUCTION

Recent progresses in medical imaging applications and computed tomography (CT) allow identification and segmentation of lung pulmonary structures (e.g. lung vessels and bronchi). However, operations such as fully-automatic segmentation and separation of pulmonary artery/vein (AV) trees still represent a big challenge. Depending on their nature, different pulmonary diseases may affect either the arterial or the venous trees in specific ways. As an example, recent studies show that alterations of the arteries may be associated with chronic obstructive pulmonary diseases (COPD) [1]. Moreover, due to the high complexity of the vascular trees and intrinsic problems of CT images, such as scan resolution and partial volume effect, manual identification of the two trees represents a long and tedious job, even for expert radiologists. For this reason, the development of a (semi-)automatic method that allows for proper distinction and classification of arterial and venous vessels may be of great help for physicians.

Although many methods have been proposed for vessel segmentation [2], only a few studies have attempted to separate AV trees starting from non-contrast CT images [3–9]. In [3], distances of vessels to bronchi (segmented with a modified region growing method) and to pulmonary fissures (estimated by a Voronoi diagram) are computed and averaged to classify vessel sub-trees. In [4, 5], a morphologic multiscale opening operator is used to separate attached arteries and veins at various scales and locations. A method to reconstruct vascular trees by labeling each sub-tree manually and considering the strength between the marked points is presented in [6]. However, methods [3–6] require user interaction to manually correct mislabeled structures and use only a small amount of CT cases for evaluation. [7] proposed a fully-automated method which uses high-order functions to encourage sets of voxels to entirely belong to arteries or veins. Although fully-automatic, the method still lacks an extensive evaluation, as it was tested only on vessels with intensities higher than −200 HU, and only compared to manual reference.

Recently, two works have been published with the aim of improving available AV segmentation approaches [8, 9]. In [9], an algorithm which represents the vessels tree as a graph and uses local information to separate the graph in a set of small sub-tress is presented. The sub-tress are linked to each other under the assumption that arteries and veins approach a common alveolar sag and classification is done by simply considering the difference in the vessel subtree volumes. Although this method does not need information about airways, discrimination based only on volume of the tress may be not ideal, especially in patients with specific diseases such as COPD. [8] proposed a fully automatic AV separation algorithm where vessels are classified based on an arterialness measure which indicates their proximity to bronchi. Uniform distribution and reduction of vessel diameter moving towards peripheral areas are also considered. The method was tested on 25 non-contrast CT images and seems to outperform [6]. However, the method is highly sensitive to parameters, with those used for calculation of arterialness being the most affective.

In this work, we propose a fully automatic AV classification approach that combines a Frangi filter [10], used to extract local information of vessels and airways, with a convolutional neural network (CNN) [11] approach, followed by a final graph-cut (GC) strategy [12]. The neural network is trained to learn information about the vascular trees on small patches surrounding the single vessel candidates defined by the scale-space particles approach described in [13, 14]. As in [8], we use a bronchus enhanced image, but we let the net automatically learn the arterialness measure, thus reducing parameters sensitivity. Connectivity information of the single particles, based on their location and strength, is also taken into account. A quantitative evaluation is performed by assessing agreement between human observers and our method in twenty non-contrast thoracic CT scans. Comparison between the deep neural net and Random Forests (RF) is also accomplished.

2. MATERIAL AND METHODS

2.1. Vascular Segmentation

The automated AV separation is performed using the workflow described in Fig. 1. The first step consists in extracting the vascular tree from the CT image. We first segment the lung and then use a scale-space particle sampling methods which exploits the Hessian matrix response to identify and represent the single vessels as collection of particles, as described in [13, 14].

Fig. 1.

Overview of the proposed method

2.2. Artery-Vein Convolutional NN

Once the particles are extracted, an initial artery-vein classification is performed for each individual particle using a CNN architecture. We train the network to automatically identify the features that separate arteries from veins on three patches of 32×32×3 pixels extracted around the axial plane of the particles. To do so, we consider four relevant aspects: local information on the CT image, Frangi vesselness (strength), Frangi airwayness (proximity to bronchi), and connectivity between particles of a vessel. For local information, the first 2D slice is extracted from the original CT image. The second slice determines the vessel strength given by the CT image enhanced with a Frangi vesselness filter [10]. A Frangi filter, specifically tuned, is also used to create a bronchus enhanced image used as the third slice of the patch. Finally, to help the net take into account connectivity information, we include the patches of the two closest particles that have the most similar orientation to the sample of interest. For the classification, we employ a 16-layer net which consists of six convolutional layers separated and followed by a pooling and a dropout layer, and three fully-connected layers. We use a Nesterovmomentum update with a softmax function as output nonlinearity, which is a typical choice in classification tasks, and we train for 1000 epochs with a learning rate of 0.3 and batch of size of 128. To train the classifier, we separated four cases from the evaluation dataset described in Section 2.4 and extracted training samples belonging only to the upper lobes. A total of 69042 particles (37977 arteries 31065 veins) were used.

2.3. Artery-Vein Graph-Cut

Despite the connectivity information that is provided to the net, spatial inconsistency may still occur during classification, mainly due to the particle-base classification strategy and the presence of many touching and intertwined areas in the two vascular trees. For this reason, once the first initial classification is concluded we employ a GC strategy to refine the classification. To this end, we use the approach described in [15] that constructs a graph based on the probability of each particle to be an artery or a vein, and to find the minimal-cuts to assign the particle to one class or the other. In our case, the initial probability is given by the trained classifier.

2.4. Evaluation

Database.

Twenty CT scans from the COPDGene study were acquired using multi-detector CT scanners with at least 16 detector channels. COPDGene centres were approved from their Institutional Review Boards and all subjects provided written informed consent. For each subject, we consider only the right lung, which is segmented and separated into superior, middle, and inferior lobes. For this study, only upper and lower lobes are utilized. This allows us to have a total of 40 independent cases for AV classification, as the two lobes present specific and unique characteristics. For each of the two lobes, a manual labeling of arteries and veins have been performed with a 2-steps approach. Starting from the scale-space particles segmentation described in Section 2.1, two trained engineers performed a pre-classification using a a slice by slice manual labeling. Then, a radiologist with broad expertise in lung imaging went through all cases and corrected possible mis-classification errors. Also, points corresponding to other structures were removed. As a final result, a total of 777206 particles (432283 arteries, 344923 veins) have been labeled. For the training of the net we use the superior lobe of four cases and 16 cases here used for validation.

Reference method.

We compare our method with the method proposed in [15]. This approach employs a Random Forest (RF) algorithm as an initial classifier for a tree representation of vessels and airways using the same scale-space particles approach.

3. RESULTS

The proposed method was evaluated on all cases of the database described in Section 2.4 that were not used for training the net. In order to evaluate performance of the algorithm, we carried out two different experiments. First, we compared the results obtained for each case to the standard reference manually created. Then, to evaluate the accuracy of CNN, we compared the results of our neural network to those obtained using RF [15] as initial classifier with and without the final GC step. In both experiments, a per-particle accuracy measure, as determined by the Jaccard similarity coefficient, was used to evaluate results.

3.1. Comparison to Manual Classification

An overview of the results obtained for all considered cases are shown in Table 1. When compared to manual classification, the automatic algorithm achieves a mean accuracy of 87% (median: 88%, range: 66% to 975%). Analyzing the two lobes individually, the algorithm achieves a mean accuracy of 88% for the superior lobe (median: 88%, range: 70% to 97%) and 87% for the inferior lobe (median: 90%, range: 66% to 97%), indicating that the algorithm is able to generalize well across lobes. Analyzing arteries and veins separately, the method achieves a mean agreement with the manual reference of 92% for arteries (median: 95%, range: 60% to 99%) and 82% for veins (median: 87%, range: 36% to 96%), indicating a better performance in classifying arteries. Moreover, CNN without the GC is able to properly classify 72% of the vessels (median: 71%, range: 61% to 85%). An example of good and bad performing classification is shown in Fig. 2 to illustrate the performance of our algorithm.

Table 1.

Overview of results obtained with the proposed method in comparison to manual reference. Overall indicates analysis on both lobes, RSL stands for right superior lobe, and RIL indicates for right inferior lobe. A stands for arteries, V for veins.

	Mean (%)			Median(%)			Range(%)
	A+V	A	V	A+V	A	V	A+V	A	V
Overall	87	92	82	88	95	87	66–97	60–99	36–96
RSL	88	93	82	88	94	87	70–97	78–99	36–96
RIL	87	91	82	90	96	87	66–97	60–98	36–95

Fig. 2.

Classification results for the upper lobe of a good (a) and a bad (b) performing cases.

3.2. Comparison to Random Forests

Table 2 shows an overview of the comparison between classifying vessels using CNN or RF as first step in the algorithm described. Whereas the algorithm that uses CNN achieves an overall accuracy of 87% (median: 88%, range: 66% to 97%), the usage of RF gives an accuracy of 73% (median: 74%, range: 51%–89%). In the separate lobes, the usage of RF yields to an accuracy of 73% (median: 74%, range: 56%–87%) for the superior lobe and 72% in the inferior lobe (median: 74%, range: 51%–89%), as compared to 88% and 87% obtained for superior and inferior lobes, respectively, using CNN. Moreover, using RF an accuracy of 94% (median: 97%, range: 66%–100%) for arteries (similar to that obtained with CNN) is achieved, while an accuracy of 46% (median: 51%, range: 4%–80%) is obtained for veins (accuracy of 82% using CNN). Analysing the ability of the two machine learning approaches to classify arteries and veins without using GC, CNN outperforms RF with an overall accuracy of 72% (median: 71%, range: 61%–85%) against 51% (median: 51%, range: 49%–65%).

Table 2.

Comparison between results obtained with CNN and RF both with and without GC. A stands for arteries, while V indicates veins.

	Mean (%)			Median(%)			Range(%)
	A+V	A	V	A+V	A	V	A+V	A	V
CNN + GC	87	92	82	88	95	87	66–97	60–99	36–96
RF + GC	73	94	46	74	97	51	51–89	66–100	4–80
CNN	72	74	69	71	77	70	61–85	56–86	45–85
RF	51	59	42	51	59	42	49–65	55–73	35–54

4. DISCUSSION

In this work, an automated method for artery-vein classification on chest CT images which uses a CNN approach in combination with GC was presented. The algorithm was evaluated using the vessels of the right lung of twenty subjects with COPD, from which the superior and inferior lobes were extracted. The proposed method achieves an overall accuracy of 87% when compared to manual reference, with only two cases below 70%. This is lower than the 91.1% accuracy claimed in [8], although a direct comparison cannot be performed as different cases were used for evaluation. The method we propose is similar to [8] in the sense that both methods do not need airway segmentation, but they only require a bronchus enhanced image to exploit the knowledge of proximity of arteries to veins. However, while in [8] an arterialness measure needs to be computed, making the algorithm high sensitive to parameters, we let the CNN automatically define this value. To motivate the choice of the use of CNN as first classifier, we compared the performance of the algorithm to an RF approach which uses airway segmentation to define proximity of arteries to bronchi. This way, we can also evaluate whether airway segmentation add important information for the classification. The results showed that a CNN approach achieves much higher results, both in terms of overall classification (in combination with GC) and as single classifier. In general, our results are promising and pave the way to a future use of CNN for AV classification. An interesting approach might be to use a recently developed 3D CNN approach [16] to help the net learn features on vessel segments instead of reformatted views across the vessel axis.