An automated pathological class level annotation system for volumetric brain images

Abstract

We introduce an automated, pathological class level annotation system for medical volumetric brain images. While much of the earlier work has mainly focused on annotating regions of interest in medical images, our system does not require annotated region level training data nor assumes perfect segmentation results for the regions of interest; the time and effort needed for acquiring training data are hence significantly reduced. This capability of handling high-dimensional noisy data, however, poses additional technical challenges, since statistical estimation of models for such data is prone to over-fitting. We propose a framework that combines a regularized logistic regression method and a kernel-based discriminative method to address these problems. Regularized methods provide a flexible selection mechanism that is well-suited for high dimensional noisy data. Our experiments show promising results in classifying computer tomography images of traumatic brain injury patients into pathological classes.

Introduction

Medical imaging plays a significant role in the modern healthcare process; a huge number of digital images are generated every day. Consequently, there is an increasing need for efficiently accessing and retrieving relevant images for teaching, research and diagnosis. By indexing the correspondence of keywords and images, traditional text-based image retrieval system (TBIR) can be used to solve the image search problem. TBIR, however, requires a laborious manual annotation of the images with keywords. Content-based retrieval systems (CBIR), on the other hand, allow users to use a query image to search for images with similar low-level visual features. However, due to its reliance on the low-level features, CBIR is not suitable for answering abstract or high-level queries. Automatic annotation technology combines the advantages of both TBIR and CBIR by first annotating images with their semantic content and then allowing users to perform text-based search on image databases. The images and their corresponding semantic labels are automatically associated by a computer.

Medical image annotation can be divided roughly into two categories: anatomical annotation and pathological annotation. While training data for anatomical annotation may be obtained from the DICOM headers of the images, pathological training data, which usually contains semantic labels, is often harder to obtain, since it normally requires manual annotation of the regions of interest. The cost and time needed for manual region labeling remains a main obstacle for building automated content-based image retrieval systems.

In this work, we introduce a system that does not require labeling of each abnormal region, but works with training data that are class-labeled at the image set level. This relaxed requirement is very helpful in practice, but it also poses new challenges. Class-labeled training data may be obtained from medical text reports either manually or automatically utilizing natural language processing techniques. In addition, our system does not assume perfect segmentation of image like many previous systems do. However, building robust systems based on incomplete and uncertain high-dimensional data would usually require large training samples, which is often not easy to obtain in medical domains. To overcome these challenges, we propose a regularized logistic regression approach for annotating volumetric brain images with pathology-level information. We consider the annotation task as a classification problem for each patient’s image scan. Instead of requiring annotations for each abnormal region on the training image, our method works on class-labeled training data that consists of several images from each patient, all of which are assumed to belong to the same pathological class.

In principle, the proposed approach could be applied to assign pathological classes to any volumetric brain images created by computer tomography (CT), magnetic resonance imaging (MRI), etc. To illustrate the framework, we build a system to automatically annotate CT brain images for traumatic brain injury (TBI) that is caused by external mechanical forces such as falls, vehicle accidents and violence. TBI is a major cause of death and disability worldwide. TBI brain damages consist of several types of hemorrhages (or hematoma): extradural hematoma (EDH), subdural hematoma (SDH), intracerebral hemorrhage (ICH), subarachnoid hemorrhage (SAH) and intraventricular hematoma (IVH). Radiologists, trained specialists and senior residents usually categorize TBI by scrutinizing the CT images.

Related work

One of the earliest attempts to annotate CT brain images is from Cosic and Longaric [1] who proposed a rule-based approach for labeling intracerebral brain hemorrhage (ICH) on CT brain images. Liao et al. [2] implemented a pathology-level image classifier for CT brain images using a decision tree-based approach. Their training data consisted of 48 images classified into three hematoma types: epidural, subdural and intracerebral; the, authors assumed that all the images have hemorrhage regions. Zhang and Wang [3] proposed to classify normal and abnormal CT brain images by using global image features such as intensity, shape, texture and symmetry. They used the C4.5 decision tree algorithm and radial basis function neural network for the classification.

Obtaining annotated training data requires substantial effort. To address the problem of lacking annotated training data, Gong et al. [4] proposed to use a traditional machine translation paradigm to find the correspondence between pathology labels and the regions of interest. They used an unsupervised approach to label the images with keywords extracted from the associated reports. This work, however, did not address the case in which the extracted regions of interest from segmentation are not real hemorrhage regions. In addition, this work focused on classifying individual region of interests (ROIs) instead of annotating the pathological classes of single cases (each containing a series of images).

In our previous work [5], we proposed a generative-model approach for solving this same class of problems. In this approach the relationships among all the features have to be carefully modeled in order to achieve good performance. If the relationships are not obvious, discriminative approaches such as support vector machines (SVMs) [6] often yield better performance.

Our current work aims at annotating volumetric CT brain images with a most representative hematoma type by using regularized logistic regression feature selection technique and kernel-based discriminative methods. The proposed approach works without annotated region-level training data and does not assume perfect segmentation results of the regions.

Data

191 CT brain scans of patients with TBI were retrospectively selected from the database of the Neuroradiology Department in a tertiary referral hospital specializing in neurological diseases in Singapore. Institutional Review Board approval was obtained for this anonymized dataset for retrospective data mining. The dataset consists of three main types of TBI: SDH (110 cases), EDH (40 cases) and ICH (41 cases). Each volumetric brain scan is a stack of 18–30 images called slices (Figure 1).

Figure 1:

A volumetric CT brain scan with 19 slices

Each slice is an image with a resolution of 512 × 512 pixels. Each scan (not each image/slice) is manually assigned a hematoma type extracted from its relevant medical text report. Since the medical report is naturally subjective, thus possibly inconsistent and difficult for automated keyword extraction, the class-assignment is conducted manually. However, this process is still much less time-consuming than annotating individual regions of interest in images.

Methods

The proposed image classification system consists of two parts: a feature extraction component and a classification component. The feature extraction component segments out all the potential hemorrhage regions and extracts their description vectors. Since each TBI case may contain a different number of hemorrhage regions, descriptions of different cases may end up having different length feature vectors. Most discriminative classifying methods are incapable of handling such data, hence we need to transform the sets of features into uniform-length vectors. This transformation will create very high dimensional data vectors, simple models for which also require many parameters. In order to avoid estimating many parameters based on a relatively small sample of 191 cases, we propose to first deploy a regularized logistic regression method that reduces the dataset’s dimensionality and then apply SVM on the reduced dimension case descriptors to learn a classification model. Unlike other feature selection methods, regularized logistic regression provides a framework to easily adjusting the selection criteria in an interpretable manner. Theoretically, logistic regression and SVM form a good pair since they both measure similarity of vectors by an inner product.

We describe our system in more detail in the following sections.

Feature extraction component

The feature extraction component extracts a list of potential regions from a brain scan using an automated segmentation technique. Due to the limitations of the segmentation technique, potential regions can either be artifacts or true hemorrhage regions. In this work, for illustration purpose, we adopt a segmentation method adapted from Li et al. [7]. This technique is shown to achieve reasonable results for content-based retrieval purposes. Other segmentation techniques could potentially be used as well.

The segmentation process may be summarized into four main stages: 1) removing the skull area from an image by using a thresholding technique and a simple hole filling algorithm [8], 2) removing artifacts and reducing noise from the image by using a wavelet technique, 3) normalizing images (pose correction, intensity and size normalization), and 4) segmenting out potential hemorrhage regions by an automated thresholding algorithm. Figure 2 illustrates the segmentation process described above.

Figure 2:

Preprocessing and segmentation process. (a) original image, (b) image after skull removal (c) image after normalization process, (d) segmented region

In traumatic brain injury, the location and shape of hemorrhage regions are important for differentiating various types of hemorrhages. For instance, EDH usually has biconvex shape with well-defined margin. Meanwhile, SDH often appears with crescentic shape and less regular inner margin. Hence, for each segmented region, we automatically extract its shape descriptors and location (see Table 1) using a regionprop method from MATLAB. For example, features S, MajA, MinA, Ecc, and E are important indicators for differentiating convex, non-convex and crescentic regions. We will also summarize this set of region descriptors to derive a more global description of the patient’s brain, and we encode this information via the features as shown in Table 2.

Table 1:

Extracted features for each potential region

Feature name	Description
Location (L1, L2)	The centroid of the region in polar coordinates.
Slice index (Z)	The slice index of the region within the stack of images (order of increasing is from neck to head)
Area (A)	The actual number of pixels in the region
Solidity (S)	The proportion of the pixels in the convex hull that are also in the region
Major axis (MajA)	The length of major axis of the ellipse that has the same normalized second central moments as the region
Minor axis (MinA)	The length of the minor axis of the ellipse that has the same normalized second central moments as the region
Eccentricity (Ecc)	The ratio of the distance between the foci of the ellipse and its major axis length
Extent (E)	The ratio of pixels in the region to pixels in the bounding box
Euler number (EN)	The number of objects in the region minus the number of holes in those objects
Adjacency (Adj)	The indicator on whether a region is adjacent to the skull

Table 2:

Description of the global features

Feature name	Feature description
N	Total number of extracted regions
WL1, WL2, WZ	Weighted average centroid of extracted regions (weighted by size of a region)
H1, H2, H3, H4	Histogram of number of extracted regions (slice index is split into four ranges: regions with Z <= 12, regions with 12 < Z <= 18, regions with 18 < Z <= 24 and regions with Z > 24)

Classification system

Our classification system consists of three main components: feature standardization, feature selection and classification.

Before incorporating the region-based features into the classification model, they need to be standardized into a uniform-length format and go through feature selection. The global features are already in uniform-length format for each brain scan, thus they can be used directly to train a classifier. Although feature selection could also be performed on the global features, these two sets of features are kept separate to avoid over-fitting the model. The two sets of features will then be combined in the classifier.

Figure 3 illustrates the workflow of our classification system.

Figure 3:

Overview of classification system

Feature standardization

As mentioned earlier, each potential hemorrhage region is characterized by a vector V = (L₁, L₂, Z, A, S, MαjA, MinA, Ecc, E, EN, Adj) of 11 features. Each brain scan B_i consists of k_i potential hemorrhage regions, hence, we can denote a brain scan B_i = (V₁ … V_ki) where k_i may vary in different scans resulting in variable length feature vectors for brain scans. Since discriminative kernel-based algorithms require uniform-length training vectors, we need to standardize our dataset into uniform-length feature vectors.

Since the brain scans may have different resolutions (in our data 18 – 30 slices), we need to interpolate the slice index of each potential region to the finest resolution, which is 30 slices/brain in our dataset. In each slice, we assume that the largest extracted region (if any) is the representative region for the slice. The other regions in a slice are discarded during this process. With this assumption, we may now transform a variable feature set into a uniform-length vector input with 30 × 11 features.

In order to have a comparable vector of features for each brain scan, the mapping (Figure 4) is done as follows:

1. Create a new vector with 30 × 11=330 features for each brain scan.

2. Divide this vector into 30 slots corresponding to 30 slices, each with 11 features.

3. Map potential regions to the corresponding slots based on their slice index Z.

4. Fill the middle empty slots with the average values of the slices that are located immediately before and after them (for interpolation purpose).

5. Fill the rest of the slots with zeros.

Figure 4:

Mapping from a variable set of features to a uniform-length vector

This standardization produces a 330 dimensional feature vector. Even having just one parameter per dimension would require estimating 330 parameters from a sample of 150 cases. Such estimates are unreliable or over-fitting, so we need to perform feature selection before classification.

Feature selection

Our feature selection approach is based on the intuition that some slices in a stack are usually more informative than the others, and that some features in those slices are more relevant than others. Hence, we employ a regularized logistic regression method (specifically, the sparse group lasso method) for feature selection. The main advantage of this method is that it allows us to select the features in a flexible manner, either individually or in a group, thereby providing substantial insight into the dataset.

L1-regularized logistic regression

Logistic regression is a popular machine learning technique for classification problems. Its regularization is also well known for avoiding over-fitting when the number of features outnumber the training examples. Specifically, L1 regularized logistic regression [9] is shown to be effective in selecting good features in the presence of many irrelevant features [10]. It requires a sample size that grows logarithmically in the number of irrelevant features.

In the supervised learning task, we are given M training instances {(x⁽ⁱ⁾, y⁽ⁱ⁾), i = 1 … M}. Each x⁽ⁱ⁾ ∈ ℝⁿ is an n-dimensional feature vector, and y⁽ⁱ⁾ ∈ {0, 1} is a binary class label. The logistic regression model defines the probability of class label y given a feature vector x as follows:

$$\mathit{p}\left(\mathit{y}=\mathbf{1}|\mathit{x};\mathit{\theta }\right)=\mathit{\sigma }\left({\mathit{\theta }}^{\mathit{T}}\mathit{x}\right)=\frac{1}{1+\text{exp}\left(-{\mathit{\theta }}^{\mathit{T}}\mathit{x}\right)},$$ (1)

where θ ∈ ℝⁿ is the parameter vector of the logistic regression model, and σ(·) is the sigmoid function as defined above. Promoting sparsity, L1 regularizer formalizes the desire to have many coordinates of θ zero by adding a regularization term $\beta {\Vert \theta \Vert }_1=\beta {\sum }_{j=1}^n\left|{\theta }_j\right|\left(\lambda >0\right)$. This yields a penalized log-loss minimization problem:

$${\mathbf{min}}_{\mathit{\theta }}{\sum }_{\mathit{i}=\mathbf{1}}^{\mathit{M}}-\mathbf{log}\hspace{0.17em}\mathit{p}\left({\mathit{y}}^{\left(\mathit{i}\right)}|{\mathit{x}}^{\left(\mathit{i}\right)};\mathit{\theta }\right)+\beta {\Vert \mathit{\theta }\Vert }_{\mathbf{1}}.$$ (2)

This optimization problem is called L₁ regularized logistic regression. In this model, features are selected individually. By increasing or decreasing the value of β, we could tune the model to pick fewer or more variables respectively.

To deal with features in groups, Meier et al. [11] proposed a group lasso extension to do variable selection on predefined groups of variables in linear regression models. Suppose that our n features are divided into G groups with n_g elements in g-th group. Hence, ${\mathit{x}}^{\left(\mathit{i}\right)}={\left({\mathit{x}}_{\mathit{i},\mathbf{1}}^{\mathit{T}},\dots {\mathit{x}}_{\mathit{i},\mathit{G}}^{\mathit{T}}\right)}^{\mathit{T}}$ with the group of variables ${\mathit{x}}_{\mathit{i},\mathit{g}}^{\mathit{T}}\in {ℝ}^{{\mathit{n}}_{\mathit{g}}}$, g = 1, …, G. When n_g = 1 for all groups, the features do not form any group in the feature space. The grouping also partitions the parameter vector into groups, θ = (θ₁ … θ_G). We may express our preference for small number of groups by adding an L1-regularization also to squared lengths of group parameter vectors θ_g. The optimization problem for this sparse group lasso may then be rewritten as:

$$\underset{\mathit{\theta }}{\mathbf{min}}\sum _{\mathit{i}=\mathbf{1}}^{\mathit{M}}-\mathbf{log}\hspace{0.17em}\mathit{p}\left({\mathit{y}}^{\left(\mathit{i}\right)}|{\mathit{x}}^{\left(\mathit{i}\right)};\mathit{\theta }\right)+\mathit{\beta }{\Vert \mathit{\theta }\Vert }_1+{\mathit{\beta }}_{\mathit{G}}\sum _{\mathit{g}=1}^{\mathit{G}}{\Vert {\mathit{\theta }}_{\mathit{g}}\Vert }_2,$$

where β_G denotes the regularization weight for group selection. By adjusting the values of β_G and β we can force the model to select more or less important groups of variables and the variables in those groups. With β_G = 0 we get a normal L1-regularized logistic regression and with β = 0 we get a normal (non-sparse) group lasso.

We apply sparse group lasso logistic regression to our training dataset for feature selection. We define a group as a set of 11 features each (corresponding to a single slice). Since we expect the relevant features for separating different class pairs to be different, we convert our multi-class feature selection task into several binary feature selection tasks (one-versus-one approach). In our case, we turn our training dataset into three pairs of classes. Feature selection is performed separately for each pair of classes.

In our experiments, we have observed that the group lasso and sparse group lasso algorithms have chosen some very distinctive and important slices. When selecting features for two SDH and EDH classes, slices in middle of the brain (10–18) are usually chosen. On the other hand, slices nearer to the top of the brain (21–25) are more frequently selected when performing feature selection for the pairs EDH-ICH and ICH-SDH. By investigating our training dataset, these selections usually show differences among hematoma types. Slices at the bottom and the top of the head are often not selected since they are most noisy.

The L1-regularized logistic regression feature selection component described above was implemented using the Sparse Learning Package (SLEP) [12].

Classification

After the features are selected by the logistic regression method, we use SVM with linear kernel [6] to perform the classification task on the selected local features and the global features. SVM is a popular machine learning algorithm that is widely used for classification problems. SVM optimizes the margin between positive and negative examples. Since an SVM solver can be used to optimize for certain criteria (such as ROC curve, precision-recall), SVM yields better results for imbalanced data in terms of precision and recall measurements.

We train binary SVM classifiers for each pair of binary training datasets. In one-versus-one approach, classification is done by a max-wins voting strategy, in which every binary classifier casts its vote to one of the two classes. The class with most votes determines the instance classification.

The classification component was implemented using the SVM^perf package [13].

Evaluation

Without global features

We performed ten-fold stratified cross-validation to evaluate the performance of our classifiers. In this experiment, we compare the performance of three feature selection methods: L1-regularized logistic regression (select individual features), group lasso (select groups of features) and sparse group lasso (selecting groups of features and individual features within group). Features from the training set are first filtered by a feature selection method and subsequently used to build the classification model. A trained classifier is then evaluated using a different testing set. Our baseline SVM classifier is directly trained using the full feature set. Table 3 shows the average performances of our classifiers using each method. We use precision (also known as positive predictive value), and recall (or sensitivity) to illustrate the performance of our classifiers.

Table 3:

Precision and recall of classifiers using different feature selection techniques

	SDH		ICH		EDH
	Precision	Recall	Precision	Recall	Precision	Recall
w/o feature selection	66%	65%	56%	59%	24%	26%
L1-regularized	71%	83%	63%	60%	52%	30%
Group lasso	76%	88%	73%	55%	67%	55%
Sparse group lasso	79%	90%	82%	58%	72%	65%

All feature selection techniques show improvements on both the precision and recall. Without feature selection, baseline classifier suffers poor performance in classifying EDH. This is most likely caused by over-fitting. Sparse group lasso outperforms the other two feature selection methods in terms of precision and recall. L1-regularized approach only chooses features individually. We observe that selecting features in group-wise manner improves the precision in all three classes.

Since parameters β and β_G control different criteria of the feature selection, they are important for the performance of the classifier. Table 4 illustrates how β and β_G affect precision and recall. The most affected classes are the minority classes. Selecting too many or too few groups and features could seriously hurt the performance of these classes. In our work, β (for features within group) is selected toward reducing more features and β_G (group) is selected toward favoring selecting more groups.

Table 4:

Precision and recall of classifier for different β and β_G

	SDH		ICH		EDH
Sparse group lasso	Precision	Recall	Precision	Recall	Precision	Recall
β = 0.08 & βG = 0.2	79%	90%	82%	58%	72%	65%
β = 0.01 & βG = 0.01	68%	80%	54%	50%	50%	30%
β = 0.3 & βG = 0.4	67%	76%	62%	45%	28%	28%

There are two main reasons for the relatively poorer performances in classifying EDH and ICH as compared to SDH. First, we have an imbalanced training dataset. The distribution of each hemorrhage type in the real setting is naturally imbalanced. Although we could avoid the imbalance in our training set by equalizing the number of training samples for each type, we chose to resample the real imbalanced distribution to allow more realistic evaluation. Hence, the number of training samples for SDH is more than the total number of cases for both EDH and ICH. Apparently this introduces a bias to the predictions. The confusion matrix in Table 5 shows that most common errors are misclassifying EDH and ICH as SDH. Second, the manual labeling process of the data may have affected the results. For instance, a specific case could have been labeled as ICH, even if the underlying cause is SDH. Hence, even if the classifier has correctly identified the TBI type, it may still be counted as an error. This problem is amplified when the dataset is favored towards a specific class (in this case, SDH).

Table 5:

Confusion matrix

		Predicted class
		SDH	ICH	EDH
Actual class	SDH	100	3	7
	ICH	12	25	3
	EDH	11	2	27

With global features

In the second experiment we combined the selected region-features with the global features for classification. We also performed ten-fold stratified cross-validation to evaluate the performance in classifying three TBI types. With the global features and the features selected by sparse group lasso, there are some improvements in terms of precision and recall for each type (Table 6). Especially for ICH cases, the classifier is more sensitive.

Table 6:

Average precision and recall of classifiers and their standard deviations after integrating global features

	SDH		ICH		EDH
	Precision	Recall	Precision	Recall	Precision	Recall
Sparse group lasso with global features	81% (±2%)	90% (±3%)	83% (±4.8%)	63% (±5%)	73% (±5%)	68% (±6%)
Gong et. al [4]	74.1%	74.1%	68.4%	72.2%	91.7%	84.6%

Since most of the existing work focuses on classifying a single ROI instead of the pathological class for a brain scan, a direct comparison is not possible. To gauge the performance of our classifiers, we made an attempt to compare our method with the results from Gong et al. [4], in which a statistical machine translation framework (mapping from pathological terms to regions of interest) was used for classifying ROIs. In that work, the pathological terms were extracted automatically from text reports using natural language processing techniques. 450 cases were used to build a translation model and 50 cases for testing. Performance was evaluated based on the classification results on a set of test images retrieved by pathological keywords. In a rough assessment, our classification results are comparable to the reported performance as shown in in Table 6. Again, this comparison is mainly for general reference purposes due to the differences in classification targets and training and testing data.

Discussion

In this work, we have introduced a pathology-level automated annotation system for volumetric CT brain images of trauma brain injury. We focus on annotating pathological class at the whole brain level, with limited annotated training data and imperfect segmentation results for the relevant brain regions.

We have proposed a new framework that combines the sparse group lasso feature selection technique and SVM to address the problem of potential over-fitting in the classification with high-dimensional incomplete and uncertain data. Sparse group lasso provides a flexible selection mechanism that helps to capture the structure of our dataset. We have also combined both region-based features and global features in the classification. We have evaluated various approaches within the class of regularized logistic regression for performance comparison. Our classifier has shown very encouraging results performing comparably with a previous, state of the art approach. The proposed method could be generalized to other volumetric brain images. Since the feature extraction component is an independent component of the framework, the classification component will remain the same even if the segmentation algorithm is changed to suit other volumetric modalities.

One of the main disadvantages of sparse group lasso is its sensitivity to regularization parameters. Choosing inappropriate parameters may lead to poor performance. Regularization parameters are usually optimized heuristically using cross-validation. However, there are some theoretical solutions for the problem. For instance, Lin et al. [14] proposed a Bayesian hierarchical model to make automatic adjustment of parameters.

There are also some other limitations in our approach that leaves room for further improvement. For instance, in our training dataset, a brain scan is currently assigned to a single class although it could belong to more than one class. We could consider the extension to multi label classification problem to address this issue. Furthermore, in this work, we assume that the class boundaries form hyperplanes in the feature space. If this assumption does not hold, transforming the features into different feature space may enhance the performance of our system.

Future work should involve the two-way integration of the image classifier with radiology report. Text report could improve the segmentation results as well as the prediction power of the classifier. On the other hand, the classification system could improve the automatic natural language understanding of medical text reports. Comprehensive human evaluation of our system will also be carried out to assess the system performance.