Predicting Visual Semantic Descriptive Terms from Radiological Image Data: Preliminary Results with Liver Lesions in CT

Abstract

We describe a framework to model visual semantics of liver lesions in CT images in order to predict the visual semantic terms (VST) reported by radiologists in describing these lesions. Computational models of VST are learned from image data using high–order steerable Riesz wavelets and support vector machines (SVM). The organization of scales and directions that are specific to every VST are modeled as linear combinations of directional Riesz wavelets. The models obtained are steerable, which means that any orientation of the model can be synthesized from linear combinations of the basis filters. The latter property is leveraged to model VST independently from their local orientation. In a first step, these models are used to predict the presence of each semantic term that describes liver lesions. In a second step, the distances between all VST models are calculated to establish a non–hierarchical computationally–derived ontology of VST containing inter–term synonymy and complementarity. A preliminary evaluation of the proposed framework was carried out using 74 liver lesions annotated with a set of 18 VSTs from the RadLex ontology. A leave–one–patient–out cross–validation resulted in an average area under the ROC curve of 0.853 for predicting the presence of each VST when using SVMs in a feature space combining the magnitudes of the steered models with CT intensities. Likelihood maps are created for each VST, which enables high transparency of the information modeled. The computationally–derived ontology obtained from the VST models was found to be consistent with the underlying semantics of the visual terms. It was found to be complementary to the RadLex ontology, and constitutes a potential method to link the image content to visual semantics. The proposed framework is expected to foster human–computer synergies for the interpretation of radiological images while using rotation–covariant computational models of VSTs to (1) quantify their local likelihood and (2) explicitly link them with pixel–based image content in the context of a given imaging domain.

I. Introduction

Medical imaging aims to support decision making by providing visual information about the human body. Imaging physics has evolved to assess the visual appearance of almost every organ with both high spatial and temporal resolution and even functional information. The technologies to preprocess, transmit, store and display the images are implemented in all modern hospitals. However, clinicians rely nearly exclusively on their image perception skills for the final diagnosis [1]. The increasing variability of imaging protocols and the enormous amounts of medical image data produced per day in modern hospitals constitute a challenge for image interpretation, even for experienced radiologists [2]. As a result, errors and variations in interpretations are currently representing the weakest aspect of clinical imaging [3].

Successful interpretation of medical images relies on two distinct processes: (1) identifying important visual patterns and (2) establishing potential links among the imaging features, clinical context, and the likely diagnoses [4]. Whereas the latter requires a deep understanding and comprehensive knowledge of the radiological manifestations and clinical aspects of diseases, the former is closely related to visual perception [5]. A large–scale study on malpractice in radiology showed that the majority of errors in medical image interpretation are caused by perceptual misinterpretation [6]. Strategies for reducing perceptual errors includes the normalization of viewing conditions, sufficient training of the observers, availability of similar images and clinical data, multiple reporting, and image quantification [3]. The use of structured visual terminologies based on radiology semantics is also a promising approach to enable unequivocal definition of imaging signs [7], but is yet little used in routine practice.

Computerized assistance for image analysis and management is expected to provide solutions for the aforementioned strategies by yielding exhaustive, comprehensive and reproducible data interpretation [8]. The skills of computers and radiologists are found to be very complementary, where high–level image interpretation based on computer–generated image quantification and its clinical context remains in the hands of the human observer [9]. However, several challenges remain unsolved and require further research for a successful integration of computer–aided diagnosis (CAD) into the radiology routine [10]. A fundamental question for a seamless workflow integration of CAD is to maximize interactions between CAD outputs and human conceptual processing [9]. The latter relies both on the trust and intuition of the user of the CAD system. Trust can only be achieved when a critical performance level is achieved by the system [10]. Intuition still requires extensive research efforts to design computer–generated outputs that match human semantics in radiology [7]. The transparency of the computer algorithms should be maximized so that the users can identify errors.

A. Semantic information in radiology images

Radiologists rely on many visual terms to describe imaging signs relating to anatomy, visual features of abnormality and diagnostic interpretation [11]. The vocabulary used to communicate these visual features can vary greatly [12], which limits both clear communication between experts and the formalization of the diagnostic thought process [13]. The use of standardized terminologies of imaging signs including their relations (i.e., ontologies) has been recently recommended to unambiguously describe the content of radiology images [3]. The use of biomedical ontologies in radiology opens avenues for enabling clinicians to access, query and analyze large amounts of image data using an explicit information model of image content [14].

There are several domain–specific standardized terminologies being developed, including the breast imaging reporting and data system (BI–RADS) [15], the Fleishner society glossary of terms for thoracic imaging [16], the nomenclature of lumbar disc pathology [17], the reporting terminology for brain arteriovenous malformations [18], the computed tomography (CT) colonography reporting and data system [19], the visually accessible Rembrandt images (VASARI) for describing the magnetic resonance (MR) features of human gliomas as well as others [20]. Most of the standardized terminologies do not include relations among their terms (e.g., synonyms or hierarchical taxonomic structures), and as such, they thus do not constitute true ontologies. As a result, these terminologies cannot be used for semantic information processing leveraging inter–term similarities. Recent efforts from the Radiological Society of North America (RSNA) were undertaken to create RadLex¹, an ontology unifying and regrouping several of the standardized terminologies mentioned above [21], in addition to providing terms unique to radiology that are missing from other exiting terminologies. RadLex contains more than 30,000 terms and their relations, which constitutes a very rich basis for reasoning about image features and their implications in various diseases.

B. Linking image contents to visual semantics

The importance of ontologies in radiology for knowledge representation is well established, but the importance of explicitly linking semantic terms with pixel–based image content has only recently been emphasized² [7]. Establishing such a link constitutes a next step to computational access to exploding amounts of medical image data [22]. This also provides the opportunity to assist radiologists in the identification and localization of diagnostically meaningful visual features in images. The creation of computational models of semantic terms may also allow the establishment of distances between the terms that can be learned from data, which can add knowledge about the meaning of semantic terms within existing ontologies.

C. Related work

The bag–of–visual–words (BOVW) approach [23] aims at discovering visual terms in an unsupervised fashion to minimize the semantic gap between low–level image features alone and higher–level image understanding. The visual words (VW) are defined as the cluster centroids obtained from clustering the image instances expressed in a given low–level feature space. Its ability to enhance medical image classification and retrieval when compared to using the low–level features was demonstrated by several studies [24–28]. Attempts were made for visualizing the VWs, aiming at interpreting the visual semantics being modeled. In [24, 26], color image overlays are used to mark the local presence of VWs in image examples. In [25, 27–29], prototype image patches (those closest to the respective VWs) are displayed to visualize the information modeled. Unfortunately, VWs often do not correspond to the actual semantics in medical images, and they are therefore very difficult to interpret for radiologists.

The link between VWs and medical VSTs is studied in [30, 31]. Liu et al. used supervised sparse auto–encoders to automatically derive several patterns (i.e., VWs) per disease. However, albeit the learned patterns were derived from the disease classes, they did not correspond to visual semantics belonging to a controlled vocabulary and therefore did not have a clear semantic interpretation [31]. In the context of endomicroscopic video retrieval, André et al. use a Fisher–based approach to learn the links between VWs learned from dense–scale–invariant feature transform (SIFT) and 8 visual semantic terms (VST) [30]. Entire videos can be summarized by star plots reflecting the presence of VSTs, but the transparency of the algorithms remains limited as the occurrence of the VSTs are not localized in the images.

Other studies have focused on the direct modeling of VSTs from application–specific semantic vocabularies [32–37]. In the context of histological image retrieval, Tang et al. built semantic label maps that localize the occurrence of VSTs from Gabor and color histogram features [33]. The authors further refined their VST maps using spatial–hidden Markov models in [38]. In [34], the link between ad–hoc low–level image features based on gray–level intensity and VSTs can be tailored for each specific user for describing high–resolution CT (HRCT) images of the lungs. Shyu et al. evaluated the discriminatory power of low–level computational features (i.e., gray–level intensity) for predicting human perceptual categories in HRCT in [39]. In [40], a comprehensive set of low–level image features (i.e., shape, size, gray–level intensity, and texture) was used to probabilistically model lung nodule image semantics. Kwitt et al. [36] defined semantic spaces by assuming that VSTs in endoscopic images are living on Riemannian manifolds in a space spanned by SIFT features. They could derive a positive–definite semantic kernel that can be used with support vector machine (SVM) classifiers. Gimenez et al. [37] used a comprehensive feature set including state–of–the–art contrast, texture, edge and shape features together with LASSO (Least Absolute Shrinkage and Selection Operator [41]) regression models to predict the presence of VST from entire ROIs.

The above–mentioned studies demonstrated the feasibility of predicting VSTs from lower–level image features. However, both transparency and performance of most systems suffer from two limitations. First, the automatic annotation of global regions of interest (ROI) can be ambiguous [32]. Local quantifications of the VSTs can increase the transparency of the system by highlighting visual features that are recognized as positive inside the ROI. Second, most of the studies do not allow for rotation–invariant detection of the VSTs by relying on low–level computational features that are analyzing images along arbitrary directions (e.g., SIFT, oriented filterbanks, gray–level co–occurrence matrices). VSTs are typically characterized by directional information (e.g., lesion boundary, nodule, vascular structure), but their local orientation may vary greatly over the ROI. Optimal modeling of VSTs requires image operators that are rotation–covariant, enabling the modeling of the local relative organization of the directions independently from the orientation of the VST [42]. The importance of the local relative orientation of directions for classifying normal liver tissue versus cancer tissue has also been highlighted by Upadhyay et al. using 3–D rigid motion invariant texture features [43].

In this work, we learn rotation–covariant computational models of RadLex VSTs from the visual appearance of liver lesions in CT. The models are built from linear combinations of N–th order steerable Riesz wavelets, which are learned using SVMs (see Section II-C). This allows local alignment of the models to maximize their response, which can be computed analytically for any order N. The scientific contribution of the VST models is twofold. First, the models can be used to predict and quantify the local likelihood of VSTs in ROIs. The latter is computed as the dot product between 12 × 12 image patch instances and one–versus–all (OVA) SVM models in a feature space spanned by the energies of the magnitudes of locally–steered VST models. Second, Euclidean distances are computed for every pair of VST models to establish a non–hierarchical computationally–derived ontology containing inter–term synonymy and complementarity. This work constitutes, to the best of our knowledge, a first attempt to establish a direct link between image contents and the visual semantics used by radiologists to interpret images.

II. Methods

A. Notations

A VST is denoted as c_i, with i = 1,…, I while its likelihood of appearance in an image f is denoted a_i ∈ [0, 1].

A generic d–dimensional signal f indexed by the continuous–domain space variable x = {x₁, x₂,…, x_d} ∈ ℝ^d is considered. The d–dimensional Fourier transform of f is noted as:

$$f(\mathit{x})\stackrel{ℱ}{\leftrightarrow }\mathrm{f̂}(\mathit{\omega })={\int }_{{ℝ}^d}f(\mathit{x})e^{-j\langle \mathit{\omega },\mathit{x}\rangle }d{x}_1\dots d{x}_d,$$

with ω = {ω₁, ω₂,…, ω_d} ∈ ℝ^d.

B. Dataset: VSTs of liver lesions in CT

The institutional review board approved the retrospective analysis of de–identified patient images. The dataset consisted of 74 contrast–enhanced CT images of liver lesions in the portal venous phase with a slice thickness of 5mm [22, 37]. There are eight diagnoses of the lesions: metastasis (24), cyst (21), hemangioma (13), hepatocellular carcinoma (6), focal nodular hyperplasia (5), abscess (3), laceration (1), and fat deposition (1). A radiologist (C.F.B., 15 years of abdominal CT experience) used the axial slice f with the largest lesion area to circumscribe the liver lesions, producing image ROIs. Each lesion was annotated with an initial set of 72 VSTs from the RadLex ontology [22]. The presence of a term c_i did not imply the absence of all others, where each lesion can contain multiple VSTs. All terms with an appearance frequency³ below 10% and above 90% were discarded, resulting in an intermediate set of 31 terms. A final set of 18 VSTs describing the margin and the internal texture of the lesions was used, which excludes terms describing the overall shape of the lesion (see Table I). Each of the 74 ROIs was divided into 12 × 12 patches⁴ to analyze (A) the margin of the lesion (i.e., periphery) and (B) the internal texture of the lesion. The distinction between these zones of a lesion is relevant to understanding how the algorithm performs, and also parallels how radiologists interpret lesions, taking into consideration both the boundary and internal features. The peripheral patches (A) were constrained to have their center on the ROI boundary and the internal patches (B) had to have their four corners inside the ROIs (see Fig. 1). The patches were overlapping with a minimum distance between the centers equals to one pixel. A maximum of 100 patches were randomly selected per ROI (i.e., 50 peripheral and 50 internal). Each of the 18 VST were represented by every patch extracted from the corresponding ROIs. Peripheral or internal patches were used as image instances depending on the VST’s localizations (see Table I). Peripheral patches were used for enhancing and nonenhancing to model the transition of enhancement at the boundary of the lesion in the portal venous phase.

TABLE I.

VSTs from RadLex used to describe the appearance of the liver lesions in CT scans. The 18 VSTs describing the margin and the internal texture of the lesions are marked in bold.

category	VST	frequency	patch location
lesion margin	1) circumscribed margin	70.3 %	peripheral
	2) irregular margin	12.2 %
	3) lobulated margin	12.2 %
	4) poorly–defined margin	16.2 %
	5) smooth margin	45.9 %
lesion substance	6) internal nodules	12.2 %	internal
perilesional tissue characterization	7) normal perilesional tissue	43.2 %	peripheral
lesion focality	8) solitary lesion	37.8 %	—
	9) multiple lesions 2–5	21.6 %
	10) multiple lesions 6–10	20.3 %
	11) multiple lesions > 10	18.9 %
lesion attenuation	12) hypodense	72.2 %	internal
	13) soft tissue density	16.2 %
	14) water density	14.9 %
overall lesion enhancement	15) enhancing	62.2 %	peripheral
	16) hypervascular	14.9 %	internal
	17) nonenhancing	29.7 %	peripheral
spatial pattern of enhancement	18) heterogeneous enh.	13.5 %	internal
	19) homogeneous enh.	32.4 %	internal
	20) peripheral discont. nodular enh.	17.6 %	peripheral
temporal enhancement	21) centripetal fill–in	17.6 %	—
	22) homogeneous retention	18.9 %
	23) homogeneous fade	21.6 %
lesion uniformity	24) heterogeneous	41.9 %	internal
	25) homogeneous	56.8 %
overall lesion shape	26) round	25.7 %	—
	27) ovoid	45.9 %
	28) lobular	25.7 %
	29) irregularly shaped	12.2 %
lesion effect on liver	31) abuts capsule of liver	17.6 %	—

Fig. 1.

Location of the image patches for the characterization of the margin (left) and the internal texture (right) of the lesions.

C. Rotation–covariant VST modeling

Recent work showed that the Riesz transform and its multi–scale extension constitutes a very efficient computational model of visual perception [44], since it performs multi–directional and multi–scale image analysis while fully covering the angular and spatial spectrums. This constitutes a major advantage when compared to other approaches relying on arbitrary choices of scales or directions for analysis (i.e., Gabor wavelets, gray–level co–occurrence matrices (GLCM), local binary patterns (LBP)). In a first step, we create VST models using linear combinations of N–th order steerable Riesz wavelets. Then, the VST models are steered locally to maximize their response, which can be done analytically for any order N and yields rotation–covariant results [42].

1) Steerable Riesz wavelets

Steerable multi–directional and multi–scale image analysis is obtained using the Riesz transform. Steerable Riesz wavelets are derived by coupling the Riesz transform and an isotropic multi–resolution framework⁵ [45]. The Nth–order Riesz transform ℛ^N of a 2–D function f yields N + 1 components ℛ^(n,N−n), n = 0, 1,…, N that form multi–directional filterbanks. Every component ℛ^(n,N−n) ∈ ℛ^N is defined in the Fourier domain as:

$${ℛ}^{(n,\widehat{N-n})}\{f\}(\mathit{\omega })=\sqrt{\frac{N}{n!(N-n)!}}\frac{{(-j{\omega }_1)}^n{(-j{\omega }_2)}^{N-n}}{{\Vert \mathit{\omega }\Vert }^N}\mathrm{f̂}(\mathit{\omega }),$$ (1)

with ω_1,2 corresponding to the frequencies along the two image axes x_1,2. The multiplication with jω_1,2 in the numerator corresponds to partial derivatives of f and the division by the norm of ω in the denominator ensures that only phase information (i.e., directionality) is retained. The directions of every component is defined by Nth–order partial derivatives in Eq. (1).

The Riesz filterbanks are steerable, which means that the response of every component ℛ^(n,N−n) oriented with an angle θ can by synthesized from a linear combination of all components ℛ using a steering matrix A^θ as in [42]:

$${\mathbf{ℛ}}^N\{f^{\theta }\}(\mathbf{0})={\mathit{A}}^{\theta }{\mathbf{ℛ}}^N\{f\}(\mathbf{0}).$$ (2)

A^θ contains the respective coefficients of each component ℛ^(n,N−n) to be oriented with an angle θ.

2) Steerable VST models

VST models are built using linear combinations of multi–scale Riesz components. Such models characterize the organizations of directions at various scales that are specific to each VST. At a fixed scale, a VST model ${\mathrm{\Gamma }}_{c_i}^N$ is defined as:

$${\mathrm{\Gamma }}_{c_i}^N={\mathit{w}}_{c_i}^{\mathrm{T}}{\mathbf{ℛ}}^N=w_1{ℛ}^{(0,N)}+w_2{ℛ}^{(1,N-1)}+\cdots +w_{N+1}{ℛ}^{(N,0)},$$ (3)

where w_ci contains the weights of the respective Riesz components for the VST c_i. An example of the construction of a model for a synthetic pattern is shown in Fig. 2. By combining Eqs. (2) and (3), the response of a model ${\mathrm{\Gamma }}_{c_i}^N$ oriented by θ can still be expressed as a linear combination of the initial Riesz components as:

$${\mathrm{\Gamma }}_{c_i}^{N,\theta }={\mathit{w}}_{c_i}^{\mathrm{T}}{\mathit{A}}^{\theta }{\mathbf{ℛ}}^N.$$ (4)

Fig. 2.

Example of the construction of a model ${\mathrm{\Gamma }}_{c_i}^8$ using a linear combination of the Riesz templates ℛ^(n,N−n). ${\mathrm{\Gamma }}_{c_i}^8$ is visually similar to the pattern [42].

l₂–norm SVMs are used to learn the optimal weights in a feature space spanned by the energies of concatenated multi–scale Riesz components to be optimally discriminant in OVA classification configurations [42, 46].

D. From VST models to a computationally–derived ontology

Once multi–scale models ${\mathrm{\Gamma }}_{c_i}^N$ are learned for every VST using OVA configurations, the distance between every pair of VST can be computed as the Euclidean distance between the corresponding set of weights w_ci. The symmetric matrix Φ(c_i, c_j) can be considered as a non–hierarchical computationally–derived ontology modeling the visual inter–term synonymy and complementarity relations.

E. Rotation–covariant local quantification of VSTs

The orientation θ of each model ${\mathrm{\Gamma }}_{c_i,s_j}^N$ is optimized at each position x_p and for each scale s_j to maximize its local magnitude as:

$${\theta }_{\text{dom},s_j}({\mathit{x}}_p):=\underset{\theta \in [0,\pi ]}{\text{arg max}}(|{\mathit{w}}_{c_i,s_j}^{\mathrm{T}}{\mathit{A}}^{\theta }{\mathbf{ℛ}}_{s_j}^N\{f\}|)({\mathit{x}}_p).$$ (5)

The maximum magnitude of the model ${\mathrm{\Gamma }}_{c_i,s_j}^N$ steered using θ_dom,sj at the position x_p is computed as:

$$m_{c_i,s_j}({\mathit{x}}_p)={\mathit{w}}_{c_i,s_j}^{\mathrm{T}}{\mathit{A}}^{{\theta }_{\text{dom},s_j}}{\mathbf{ℛ}}_{s_j}^N\{f\}({\mathit{x}}_p).$$ (6)

For a given image patch, a feature vector υ can be built as the energies E of the magnitudes over the patch of every steered model as:

$$\mathit{\upsilon }=\{E(m_{c_1,s_1}),\dots ,E(m_{c_1,s_J}),\dots ,E(m_{c_I,s_1}),\dots ,E(m_{c_I,s_J})\}.$$ (7)

The dimensionality of υ is I × J. It worth noting that the features from Eq. (7) are not steerable anymore after using the energies of the steered model’s magnitudes. OVA SVM models u_i with Gaussian kernels ϕ(υ) are used to learn the presence of every VST in the feature space spanned by the vectors υ in Eq. 7. The decision value of the SVM for the image patch υ_p measures the likelihood a_i of a VST as:

$$a_i=\langle \varphi (\mathit{\upsilon }),\varphi ({\mathit{u}}_{\mathit{i}})\rangle +b=\text{exp}\left(\frac{-{\Vert \mathit{\upsilon }-{\mathit{u}}_{\mathit{i}}\Vert }^2}{2{\sigma }^2}\right)+b,$$ (8)

where b is the bias of the SVM model. Likelihood maps are created by displaying a_i values from all overlapping patches.

F. Experimental setup

The number of scales was chosen as J = ⌊log₂(12)⌋ = 3 to cover the full spatial spectrum of 12 × 12 patches. The order of the Riesz transform N = 8 was used, which we found to provide an excellent trade–off between computational complexity and the degrees of freedom of the filterbanks in [42, 47]. A leave–one–patient–out (LOPO) cross–validation (CV) was used both to learn the VST models and to estimate the performance of VST detection using OVA configurations. For each fold, the training set was used both for learning the models and to train the SVMs in the feature space spanned by the vectors υ in Eq. (7). No multi–class classification is performed since the VSTs are not considered as mutually exclusive [37]. The random selection of the patches was repeated five times to assess the robustness of the approach. The decision values a_i (see Eq. (8)) of the test patches were averaged over each ROI and used to build receiver operating characteristic (ROC) curves for each VST. The values a_i of the test patches were also used to locally quantify the presence of a VST and were color–coded to create VST–wise likelihood maps. CT intensities were used as additional features to model gray–level distributions in the patches, which we found to be complementary to Riesz models in [46]. The distribution of CT intensities in [−60, 220] Hounsfield Units (HU) were divided into 20 histogram bins that were directly concatenated with the features obtained from Eq. (7).

III. Results

A. VST models and computationally–derived ontology

Examples of scale–wise models ${\mathrm{\Gamma }}_{c_i,s_j}^N$ are shown in Fig. 5 for six VSTs. The distributions of the weights w_ci,sj for every N + 1 Riesz component ${ℛ}_{s_j}^{(0,N)},\cdots ,{ℛ}_{s_j}^{(N,0)}$ are shown with bar plots. The normalized computationally–derived ontology matrix Φ(c_i, c_j) containing the Euclidean distances between every pair of VST models is represented as a heatmap in Fig. 3.

Fig. 5.

Scale–wise models ${\mathrm{\Gamma }}_{c_i,j}^N$ for 5 VSTs and N = 8. The distributions of the weights w_ci,j learned by OVA SVMs are represented for each scale with bar plots. The red regions in the last column of the table are showing examples of likelihood maps a_i computed from Eq. (8) for every VSTs.

Fig. 3.

Computationally–derived ontology matrix Φ(c_i, c_j) containing the normalized Euclidean distances for every pair of VST models. Values closer to zero (black) indicate the shortest distances, or most similar terms. Clear relationships between groups of VSTs are revealed both for terms describing lesion margin and internal texture.

B. Automated VST detection and quantification

ROC curves were built by varying thresholds on the decision values a_i. Fig. 4 compares the detection performance using the energies of the magnitudes of steered models (i.e., Eq. (7)) versus the energies of the initial Riesz components. A paired T–test on a_i values is used to compare the two approaches. We also compared the performance of steered models combined with HU histogram bins with the best results obtained by Gimenez et al. on the same dataset as used here [37]. Examples of local quantification of the local presence a_i of VSTs (i.e., likelihood maps) are shown in the last column of Fig. 5.

Fig. 4.

Comparison of the automated detection performance between (A) initial Riesz components ℛ^N, (B) steered models ${\mathrm{\Gamma }}_{c_i}^N$, (C) steered models ${\mathrm{\Gamma }}_{c_i}^N$ combined with HU histogram bins, and (D) the best results obtained by Gimenez et al. [37] on the same dataset in terms of AUCs. (*) denotes p–values below 0.05 for the comparison between (A) and the (B). (B) are always higher or close to the best performance of the (A), which highlights the importance of rotation–covariance. The difference between the global AUCs of (A) and (B) (i.e., 0.76 versus 0.8) is associated with a p–value of 7.5446e-152. (C) and (D) were shown to be complementary.

IV. Discussions and Conclusions

We built computational models of human perceptual semantics in radiology. The framework identifies relevant organizations of image scales and directions related to VSTs in a rotation–covariant fashion using steerable Riesz wavelets and SVMs. The models were used both to (1) detect and quantify the local presence of VST and (2) to create a computationally–derived ontology from actual image content. In these early applications, our techniques for detection and quantification of VST enable automatic prediction of relevant semantic terms for radiologist consideration. This is an important step towards improving the accuracy of lesion description and diagnosis, with the aim of reducing overall interpretation errors. Our approach generates VST likelihood maps (see Fig. 5), which provide insights on the information modeled by the system, making it possible for the user to evaluate the amount of trust that can be put in its outputs and maximizes the transparency of the methods. Avoiding a “black box” type approach, this feature of our system is likely to be more intuitive to the ultimate end users. The automated detection and quantification of visual semantics grants access to large amounts of similar images by enabling interoperability with semantic indexing [22, 48]. The creation of a computationally–derived ontology from VST models can be used to refine and complement existing ontologies, where relations between terms are solely encoded by their hierarchical linguistic organization. The proposed computationally–derived ontology allows measuring inter–term synonymy and complementarity in the context of a given medical application adding additional useful knowledge to existing ontologies [49].

The visualization of the computationally–derived ontology matrix Φ(c_i, c_j) in Fig. 3 reveals clear relationships between groups of VSTs. Two homogeneous groups are found to model antonymous terms concerning the lesion margin: well– versus poorly–defined margin. Irregular, lobulated and poorly–defined margin are very close, and they are all distant from circumscribed and smooth margin. The lack of distinction between the shape of the lesion and the type of lesion margin shows the inability of the proposed models to characterize overall lesion shape. For instance, lobulated margin and poorly–defined margin are not expected to be close, as it is possible to have lesions with margins that are both lobulated and circumscribed. Shared synonymy and antonymy is also observed in VSTs describing the internal texture of the lesions. Heterogeneous and internal nodules are found to share synonymy, and they are both opposed to homogeneous, homogeneous enhancement and hypodense. It can also be observed that hypervascular is close to heterogeneous enhancement and peripheral discontinuous nodular enhancement, which all relate to the pattern of lesion enhancement. A few erroneous associations are observed (e.g., soft tissue density and hypervascular, or water density and enhancing). Overall, the computationally–derived ontology is found to be complementary to the RadLex ontology, because it allows connecting semantic concepts with their actual appearance in CT images. For instance, heterogeneous and homogeneous are very close to each other in RadLex because they both describe the uniformity of lesion enhancement, but they are opposed to each other in the computationally–derived ontology since they are visually antonymous in terms of texture characterization. The combination of the RadLex ontology and the computationally–derived ontology was shown to significantly improve image retrieval performance in [49]. Likewise, we would expect our computationally–derived ontology to be useful in combination with existing ontologies like RadLex in image retrieval and other applications.

Figure 4 details the VST detection performance. It shows that although the steered models are not improving the results for all VSTs, they are always higher or close to the best performance of the initial Riesz components (i.e., global AUC equals to 0.8 versus 0.76, p = 7:5446e-152). An overall complementarity of the features based on steerable VST models and HU intensities is observed (e.g., water density, soft tissue density, hypervascular, hypodense). However, the detection performance is little improved or even harmed for texture–related terms when compared to using only steerable VST models (e.g., heterogeneous, heterogeneous enhancement, homogeneous). The proposed approach appears to be complementary to Gimenez et al. [37], where the errors are occurring for different VSTs. This suggests that the inclusion of other feature types can improve our approach.

Fig. 5 displays the relevance of the information modeled by a subset of VST models. The visualization of models ${\mathrm{\Gamma }}_{c_i,j}^N$ reveals dominant scale–wise VSTs. The largest scale model of peripheral discontinuous nodular enhancement implements a detector of nodules surrounding the lesion boundary. The two smallest scale models of internal nodules are implementing circular dot detectors. These cases illustrate that the models of VSTs correspond to image features that actually describe the intended semantics. In general, the scale–wise distributions of the weights reveal the importance of Riesz components ℛ^(0;N) and ℛ^(N;0), which can be explained by their ability to model strongly directional structures (see Fig. 2). The stability of the models over the folds of the LOPO–CV is measured by the trace of the covariance matrices of VST–wise sets of weights w_ci over the CV folds. The values of VST–wise traces are all below 0.33% of the trace of the global covariance matrix of all models from all folds. This demonstrates the stability, and hence the generalization ability, of the learned models.

We recognize several limitations of the current work, including a narrow imaging domain, a relatively small number of cases, and the use of somewhat thick, 5mm CT sections. In future work, we plan to include additional cases to limit the risk of finding erroneous associations between terms caused by fortuitous co–occurrences of them. We plan to include additional image features that model the lesion shape. This will allow us to create separate computationally–derived ontologies based on the type of information modeled (i.e., intensity, texture, margin and shape).