Enhancing Image Analytic Tools by Fusing Quantitative Physiological Values with Image Features

Abstract

Computer-aided diagnosis systems (CADs) can quantify the severity of diseases by analyzing a set of images and employing prior statistical models. In general, CADs have proven to be effective at providing quantitative measurements of the extent of a particular disease, thus helping physicians to better monitor the progression of cancer, infectious diseases, and other health conditions. Electronic Health Records frequently include a large amount of clinical data and medical history that can provide critical information about the underlying condition of a patient. We hypothesize that the fusion of image and clinical–physiological features can be used to enhance the accuracy of automatic image classification models. In particular, this paper shows how image analytic tools can move beyond classical image interpretation models to broader systems where image and physiological measurements are fused and used to create more generic detection models. To test our hypothesis, a CAD system capable of quantifying the severity of patients with pulmonary fibrosis has been developed. Results show that CAD systems augmented with multimodal physiological values are more robust and accurate at determining the severity of the disease.

Introduction

During the last 10 years, advances in sensors and imaging devices have made possible the acquisition of medical images at increasing speed, resolution, and quality. Currently, the large number of multimodal images that can quickly be captured within a single radiology study have created challenges for radiologists trying to analyze, examine, and interpret large datasets within a short period of time [1]. Those challenges had motivated the integration of computer-aided diagnosis systems (CADs) and advanced information technology within the clinical workflow [2].

During the last decade, CADs have emerged as image analytic tools from which clinical physicians and radiologists can obtain quantitative information about the disease [3]. In general, CAD systems have shown to be effective at assisting physicians during the interpretation and decision-making process. However, given the significant amount of anatomical variance often found among subjects and the different anatomical patterns frequently found in medical images, most CAD systems are specially designed to work with a particular disease, image format and/or anatomical region. For instance, many CAD systems have been developed to assist in the diagnosis of breast cancer [2, 4, 5]. Among them, some CADs have been optimized to automatically analyze radiographic X-ray images while others are optimized to analyze magnetic resonance (MR) images. Similarly, many CAD systems have been developed for other diseases including lung diseases, polyps, and brain tumors [5–7].

Since each CAD system is designed to identify particular image patterns and anatomical regions, most of the research in the development of CAD systems has been focused on creating new image processing techniques, statistical models, and image features capable of accurately detecting abnormal regions. Although image analysis has proven to be crucial in the detection and monitoring of numerous diseases, existing techniques are limited by the quality of the input images and the generalization of the different parameters. Commonly, small adjustments to the feature extraction, statistical model, or feature selection process are needed to achieve optimal results with various datasets [8, 9].

Recently, advances in systems biology and bioinformatics suggest that many diseases may have biomarkers that potentially could be used to diagnose and measure the disease appearance, progression, and recurrence [10, 11]. The distinctive benefits of biomarkers and image features raise the possibility of integrating these variables for optimizing the yield of diagnostic information. However, since for many diseases unique biomarkers are not known and most findings are still under research, other multimodal information could be used to augment the automatic identification process.

Electronic Health Records contain a voluminous amount of information that is challenging to systematically organize and interpret during the diagnostic process. Additionally, the multimodal and heterogeneous characteristics of the data make it very challenging to integrate qualitative radiology reports with quantitative laboratory data [12]. However, data integration can move CAD systems from image analytic tools that mainly rely on the extraction and analysis of image features to more generic frameworks where images are analyzed within the context of the patients' clinical condition. In particular, data fusion enables the development of image analytic applications that are more clinical-like frameworks in which a number of weak biomarkers are combined and used to diagnose a patient. To test our hypothesis that the fusion of image and clinical–physiological features can be used to enhance the accuracy of automatic image classification models, a study has been carried out to test the accuracy of automatically determining the severity of patients with pulmonary fibrosis.

Pulmonary fibrosis is a life-threatening disease that causes shortness of breath, coughing, and diminished exercise tolerance. Characterized by scarring of lung tissue and the development of excess fibrotic tissue in the lungs, many forms of pulmonary fibrosis have poor prognoses for patient outcomes. Approximately 40,000 people die annually from pulmonary fibrosis and about 200,000 individuals are affected by fibrosis in the USA [13, 14]. The need to improve surveillance and clinical monitoring of patients' progression and symptoms patients warrants the need to better identify the severity of pulmonary fibrosis. The most commonly used method to diagnose lung fibrosis is by analyzing chest radiographs or computed tomography (CT) images to detect patterns that indicate fibrosis; however, in many cases such patterns are discernible only in the most advanced stages, thus making it too late for the advanced disease to be effectively managed.

In our study, laboratory-based blood measurements, pulmonary function tests (PFTs), and electrocardiogram (EKG) studies were performed on 66 subjects within 5 days of capturing a high-resolution CT (HRCT) of their lungs. Each of the 66 patients had a history of pulmonary fibrosis. A group of expert clinicians scored the severity of the patients and labeled them as being: none, minimum, mild, moderate, and severe [15]. Figure 1 shows a set of images highlighting the differences between each of the different stages. Note that as pulmonary fibrosis progresses, more fibrotic tissue appears within the lungs. Computer-aided diagnosis systems often use those patterns to quantify and score the severity of the disease.

Fig. 1.

A set of images highlighting the differences between each of the different stages. Note that as pulmonary fibrosis progresses, more fibrotic tissue appears within the lungs. The right image shows how computer-aided diagnosis systems often use those patterns to quantify and score the severity of the disease

Although radiological features are important for assessing pulmonary fibrosis, integration of quantified imaging variables with clinical variables could further optimize the diagnostic process. Recently, such data fusion techniques have been considered. Kumánovics et al. demonstrated how physiological measurements such as the levels of serum are correlated with lung fibrosis [16]. Depeursinge et al. showed how clinical and visual features could be fused to enhance image classification models of patients affected with interstitial lung disease [17]. Ye et al. presented a framework to fuse heterogeneous data to better predict and monitor Alzheimer's disease [18]

The aim of this paper is to study the associations between clinical and image features within patients with pulmonary fibrosis and to explore how physiological attributes such as laboratory-based blood measurements, electrocardiograms, and pulmonary function tests can be used to enhance the detection and quantification of the disease.

Method

Our framework to test the effectiveness of fusing clinical and image data consisted of four primary steps. First, the input high-resolution CT studies were segmented to obtain a mask of the lung. Second, a texture-based image analysis technique was used to model and obtain quantitative information from the CT images. Third, the image and clinical data were fused into a single vector and statistical analysis was performed to select a subset of image and clinical features capable of optimizing the yield of the automatic identification process. Finally, a multimodal model was created to take advantage of different sources of data to automatically diagnose the severity of the disease. Figure 2 summarizes our approach and method.

Fig. 2.

Pipeline followed by our system to analyze and fuse quantitative image features and clinical variables to better assess the severity of a particular disease

Lung Segmentation

Before employing any image analysis technique, segmentation is often the first step in most CAD systems. There are many well-established lung segmentation methods used in clinics [19, 20]. In this study, a region growing technique is used to automatically create a binary mask of the lungs. For this study, two seeds per volume (i.e., one for the right lung and one for the left lung) were automatically selected by considering the locations of small intensity valued voxels inside the body region. The segmentation technique recursively examined neighboring pixels of the initial seed points and determined whether a pixel should be in or out of the mask of the lungs. At the end, a post segmentation filtering was applied to remove any possible vessels contained within the mask.

Image Quantification

The second step of the pipeline is executed to obtain quantitative images features. At this step, texture analysis is used in conjunction with supervised machine learning techniques to estimate quantitative measurements of the images and create statistical models to automatically identify fibrotic patterns.

The image quantification step has two primary components: training and testing. During the training part, a small set of images are annotated by an expert radiologist and used by the computer to learn the statistical properties of the disease. The annotated regions are split into small patches or blocks of K² pixels with ϑ overlapping pixels. First-, second-, and high-order statistics are estimated for each resulting patch and the obtained measurements are then used to learn the statistical properties the patterns under consideration [21, 22]. Figure 3 (top) shows a diagram of the training step. In this example, fibrotic patterns are annotated in red while normal parts of the lungs are annotated in blue.

Fig. 3.

The image quantification module involves a training and a testing step. (Top) During the training step, annotations from expert radiologists are used to learn the statistical properties of the patterns under consideration. (Bottom) During the testing step, new images are inputted to the imaging model which is used to estimate quantitative image features for the study under consideration

First-order statistics are calculated from the probability of observing a particular pixel value at a randomly chosen location in the image. First-order statistics are among the simplest textural measurements that can be extracted from 2D/3D images, especially given that they are obtained from a histogram. A histogram is a simple measurement that maps intensity values into various disjoint bins. Some of the first-order statistics include mean, variance, kurtosis, skewness, and deviations.

A number of local statistical properties cannot be captured by first-order statistics, but instead can only be captured with more advanced texture-analysis techniques such as second- or high-order statistics. Second-order statistics measure the likelihood of observing a density value i and j at an average distance d = (dx, dy) apart. Frequently, second-order statistics are computed using co-occurrence matrices as demonstrated by Haralick et al.[23] Let D = (r, f) denote a vector in the polar coordinate of the image. We can compute the joint probability of the pairs of gray levels that occur at pairs of points separated by D. That joint probability can be stored on a matrix P(i, j) which contains the probability of observing the pair of gray levels (i, j) occurring at separation D. In our case, we estimate the co-occurrence matrices by computing different matrices for the angles f = (0°, 45°, 90°, 135°) within each axis. To guarantee rotation-invariant properties, we compute the average co-occurrence matrix and then extract statistical properties including energy, contrast, correlation, inertia, entropy, and sum of entropies.

To better identify regions with similar properties, we use run-length matrices of higher-order statistics to estimate the number of gray level runs within the image [24]. The run-length matrix PQ (i, k) (i = 1,..,m; k = 1,..,n) represents the frequency that k points with 32-bit quantized gray level i continue in the direction Q = (0°, 45°, 90°, 135°). From a run-length matrix textural statistics such as the amount of short (fine) runs, long (coarse) runs, and the uniformity of such runs can be estimated.

In total, our system used 26 different textural metrics to create a model capable of identifying the individual patterns. Table 1 lists the textural features that were extracted from each patch. Once the 26 measurements are extracted for all the patches under consideration, support vector machines (SVMs)—a supervised learning technique—are used to create a statistical model [25]. The general idea of SVMs is to use the observed measurements to estimate a hyperplane in high-dimensional space that can be used to separate two classes [25]. In our case, the resulting hyperplane maximizes the distance between fibrotic tissue and normal areas of the lungs.

Table 1.

List of the textural metrics used to obtain quantitative image features of the lungs

Histogram features	GLCM		Run-length matrices
Mean	Energy	Entropy sum	Short-run emphasis	Run-length non-uniformity
Variance	Inertia	Average sum	Long-run emphasis	Long-run low gray-level emphasis
Skewness	Inverse diff	Entropy diff	Run gray-level non-uniform	Short-run high gray-level emphasis
Kurtosis	Entropy	Average diff	Low gray-level run emphasis	Long-run high gray-level emphasis
Abs deviation	Correlation		High gray-level run emphasis	Run percentage
Std deviation			Short-run low gray-level emphasis

Once a model is trained, the textural measurements of any new patch are used to determine its corresponding group. Note that the training step only is executed once—the first time a disease is specified. After that, the same statistical model is applied to any new input data [26].

The second core component of the image quantification step is testing. During the testing step, previously unseen images are inputted to system and the statistical model classifies them based on their statistical properties. Figure 3 (bottom) shows the general idea of the testing step. At this step, the resulting segmentation of the lungs is used to define the region that is split into small blocks of K² pixels. Texture analysis is applied to each of the patches and the generated model classifies them according to their statistical properties. The last two components of Fig. 3 (bottom) show the results of the automatic identification process and how that classification is used to obtain quantifiable information about a given HRCT study.

Data Fusion and Subset Estimation

To move beyond classical CAD systems that estimate quantifiable information for an input image, we also incorporated to the framework clinical variables. By doing that, the image was considered within the context of the patient's condition. At this step, a number of statistical tests including t tests, analysis of variance (ANOVA), Pearson correlation, and linear regression were performed between the image and clinical features. Those tests were used to better understand the associations between clinical values and quantitative image features and to estimate a set of clinical values that could be used to improve the accuracy of the automatic identification process.

Before combining multimodal measurements into a single feature, Kolmagorov–Smirnov test—a normality test—was used to determine the specific statistical analysis needed to determine the associations between individual variables [27].

Multimodal Model

The last step of the pipeline is designed to create a statistical model from the multimodal input data. In our case, support vector machines (SVMs) used the labels defined by the expert clinicians (i.e. none, minimum, mild, moderate, and severe [15]) to determine a statistical model capable of automatically classifying the input data [25]. This modeling step also has training and testing components. During the training process, part of the data was used to learn the statistical model while the rest was used to test the accuracy of the resulting model. During the testing step, new image features and clinical variables were presented to the system which automatically determined severity. In our experiments, a radial basis function was used as the kernel function for the SVM model. The corresponding parameters were estimated from a grid space computed during the training process [25, 28].

Results

In this section, we present results from three core components of our pipeline. First, we tested the accuracy of the second step which uses textural properties to obtain quantifiable information about the study. After using the expert's annotations and splitting the regions into 8 × 8 pixel blocks (11,889 blocks), an SVM model was created to automatically identify fibrosis. Figure 4 shows the accuracy of the detection process as well as the variation in accuracy observed when different patch sizes were used. After a number of experiments and different texture sizes, we have concluded that, in HRCT images, fibrosis can be detected more accurately by using a K = 8 × 8 window size with an σ = 2 pixels offset on each side. When using a model created from K = 16 × 16 patches, the overall accuracy of the system dropped by 10%. Figure 4 (left) shows the ROC curves on different patch sizes. From the results, we can see that our imaging module can automatically identify patterns of fibrosis within HRCT images with 90% accuracy. Figure 4 (right) shows the results of an automatic identification process of a severe case. The fibrotic tissue is automatically identified and highlighted in gray.

Fig. 4.

(left) ROC curves comparing different texture sizes. When using 8 × 8 patch sizes, the accuracy of the identification process increased by almost 10%. (right) Results of the automatic identification process are overlaid on the input images. Gray areas are classified as fibrosis while black areas identified as normal

To further demonstrate the effectiveness of using textural features for the identification of pulmonary fibrosis, correlation between image features and the severity of the disease was studied. Figure 5 shows some of those associations between severity and texture measurements. Absolute deviation from first-order statistics was found highly correlated. This association is the results of an increase in the dispersion from the mean intensity value as the disease progresses. Within second-order statistics, inertia and entropy (among others) were also found to be highly correlated. This is, as fibrosis progresses, the areas of the lungs that are less uniform/similar, thus causing a larger difference in entropy. From run-length matrices, high gray-run emphasis—a measure of the frequency of observing gray runs (i.e. fibrotic tissue)—was shown to be highly correlated. In general, we found that texture analysis is an effective technique to obtain quantitative image features and identify pulmonary fibrosis [21].

Fig. 5.

A subset of the image features that were found to be highly correlated with the severity of pulmonary fibrosis

From our texture-based image model, we also wanted to test the accuracy of automatically determining the severity of the disease. Despite the 90% accuracy in detecting fibrotic patterns within HRCT images, the precision of determining the severity of the disease based on those patterns was 76.92%. The difference here is that our imaging module only looks at images, not at the images in the context of the patient's condition. This highlights the importance of fusing image and clinical features when designing tools to assist physicians during the diagnosis process.

Now that we have determined that texture analysis and the second step of the pipeline are useful for quantifying and measuring pulmonary fibrosis, we wanted to explore which clinical and physiological values are associated with the severity of pulmonary fibrosis and how image and clinical variables can be merged. At this step, multiple significant (p < 0.01) correlations were found, but—as expected—the correlations were not as strong as when using image features given that (as of today) pulmonary fibrosis does not have unique biomarkers. Table 2 summarizes some of the blood measurements we correlated with the severity of pulmonary fibrosis. Note that even those correlations that were significant, the correlation coefficients were not that strong. Figure 6 plots some of the clinical measurements that were found significantly correlated. Erythrocyte sedimentation rate (ESR)—a blood value measuring inflammation—was found correlated. Another blood feature found correlated with fibrosis disease was fibrinogen, a measure of the blood clotting ability. Multiple values obtained from PFTs were found correlated including forced vital capacity (FVC)—the volume of air that can be blown out after full inspiration and total lung capacity (TLC) which measure the maximum volume of air present in the lungs. When analyzing the correlation with EKG, multiple values were correlated including the QRS axis of the heart. Figure 6 shows some of our results.

Table 2.

A set of the laboratory blood features that were correlated with the severity of pulmonary fibrosis

Variable	R value	P value
Erythrocyte sedimentation rate	0.500	0.001*
Fibrinogen	0.383	0.005*
IgA Serum	0.379	0.006*
RDW	0.304	0.031*
Lactate dehydrogenase	0.290	0.040*
IgG serum	0.266	0.061
Uric acid	0.235	0.099
Alkaline phosphatase	0.201	0.160
Eosinophils	0.185	0.196
MCV	0.135	0.349
Lymphocyte absolute	0.107	0.458
WBC	0.098	0.494
Glucose	0.090	0.531
Total CO2	−0.053	0.712
Potassium	−0.074	0.608
Sodium	−0.085	0.556
Creatinine	−0.103	0.473
Monocytes	−0.150	0.297
Calcium	−0.169	0.238
Hgb	−0.177	0.217
Hct	−0.188	0.189
RBC	−0.218	0.126
Creatine kinase	−0.228	0.116
IgM serum	−0.285	0.044*
Magnesium	−0.285	0.044*
Albumin	−0.383	0.005*

Note that even the measurements that are significantly correlated (marked with an asterisk), the correlation coefficients are not that strong

Fig. 6.

A subset of the clinical and physiological features that were found to be correlated with the severity of pulmonary fibrosis. In our framework, such features are used as weak markers to create multivariate models capable of predicting the stage of the disease

Since the clinical features are relative weak markers of disease, they are not generally used independently to predict the stage of pulmonary fibrosis. They can instead be combined in a multivariate fashion and used to better estimate and/or monitor the severity of pulmonary fibrosis. By combining the four weak markers shown in Fig. 6, we were able to create a multivariate model with correlation coefficient R² = 0.626. This shows that multivariate models of weak clinical markers could potentially be used in conjunction with image features to accurately assess the severity of the disease. From these results we can see that image features are more correlated to the severity of pulmonary fibrosis than the blood and PFT features. In particular, we can see that with images we can create a predictive model that estimates severity with 76.92% accuracy while when creating a model with clinical variables the accuracy drops to 62.6%. Now we can compare the effectiveness of fusing image and clinical data.

To demonstrate how image analytic tools can be enhanced with physiological values, a statistical model was designed using images and clinical variables. When only textural features were used to automatic predict the severity of the disease, a 76.92% (0.17 std error) was observed when using an 8 × 8 window size. When using other suboptimal window sizes such as 4 × 4, the accuracy dropped to 70.76% (0.21 std error) as illustrated in Fig. 7b.

Fig. 7.

Results of using image features and clinical values to automatically predict the severity of the disease. a Results when using image features with an 8 × 8 window size. b Results when using image features with a 4 × 4 window size. From the results we can see the importance of fusing image and multimodal clinical variables to automatically diagnose pulmonary fibrosis

By adding four laboratory-based blood variables from Table 2 to our model, a 1–4% improvement was observed. These are blood measurements such as ESR and fibrinogen that provide additional information to the model about the patient's condition, thus improving the accuracy of the model. Furthermore, when two EKG features and two PFT measurements were added to the model, the accuracy of the model consistently increased between 2% and 6%. The improvement is even greater when suboptimal image patch sizes were used as demonstrated by Fig. 7b. On average, an improvement of 2–6% was observed in all analytics tools that were augmented with multimodal clinical variables. The 6% improvement might not seem like a significant enhancement, but when we take into consideration that even when a significant amount of effort is put into optimizing the parameters and image features such an improvement cannot be reached, then we can understand the benefits of using multimodal features. In particular, here we can see how a small subset of four to six clinical variables is capable of putting the image features in clinical context, thus enhancing the accuracy of the automatic classification model.

From our results we can see that as more weak biomarkers are included from different sources, the accuracy of the detection process increases. In addition we can see that as more multimodal features are used, the difference between using a suboptimal versus optimal window size decreases, underscoring the importance of considering and analyzing images within its clinical context.

Discussion

Diagnostic decision making among physicians generally entails integrating quantitative data, such as laboratory blood tests, with qualitative radiology reports. This study tests the hypothesis that fusing quantitative image measurements with quantitative laboratory variables can enable more accurate statistical models for computer-assisted radiologic diagnosis. This paper motivates the concept of how image analytic tools can move beyond traditional image processing models to broader multivariate systems that integrate imaging with clinical measurements.

The results presented in this paper show how multimodal features can be used to integrate imaging with clinical values. In addition, this paper shows how CAD systems can take advantage of hidden correlations between images and clinical variables to enhance automatic classification systems. A limitation of our work is that all the multimodal features are equally weighted in high-dimensional space and this study does not correlate with ultimate patient outcomes such as mortality or symptoms. Further research will seek to expand and improve the clinical variables that are used as weak biomarkers and will study methods to automatically determine individual weights for different clinical tests.

Conclusion

This paper illustrates the importance of considering multimodal clinical features during the task of image classification. Among patients with lung fibrosis, there are multiple correlations that can be used to monitor the progression of the disease as well as to improve automatic scoring systems. Quantified computer-assisted image analysis integrated with clinical laboratory test results through statistical analysis can improve diagnosis and staging of diseases.