A hierarchical knowledge-based approach for retrieving similar medical images described with semantic annotations

Abstract

Computer-assisted image retrieval applications could assist radiologist interpretations by identifying similar images in large archives as a means to providing decision support. However, the semantic gap between low-level image features and their high level semantics may impair the system performances. Indeed, it can be challenging to comprehensively characterize the images using low-level imaging features to fully capture the visual appearance of diseases on images, and recently the use of semantic terms has been advocated to provide semantic descriptions of the visual contents of images. However, most of the existing image retrieval strategies do not consider the intrinsic properties of these terms during the comparison of the images beyond treating them as simple binary (presence/absence) features. We propose a new framework that includes semantic features in images and that enables retrieval of similar images in large databases based on their semantic relations. It is based on two main steps: (1) annotation of the images with semantic terms extracted from an ontology, and (2) evaluation of the similarity of image pairs by computing the similarity between the terms using the Hierarchical Semantic-Based Distance (HSBD) coupled to an ontological measure. The combination of these two steps provides a means of capturing the semantic correlations among the terms used to characterize the images that can be considered as a potential solution to deal with the semantic gap problem. We validate this approach in the context of the retrieval and the classification of 2D regions of interest (ROIs) extracted from computed tomographic (CT) images of the liver. Under this framework, retrieval accuracy of more than 0.96 was obtained on a 30-images dataset using the Normalized Discounted Cumulative Gain (NDCG) index that is a standard technique used to measure the effectiveness of information retrieval algorithms when a separate reference standard is available. Classification results of more than 95% were obtained on a 77-images dataset. For comparison purpose, the use of the Earth Mover's Distance (EMD), which is an alternative distance metric that considers all the existing relations among the terms, led to results retrieval accuracy of 0.95 and classification results of 93% with a higher computational cost. The results provided by the presented framework are competitive with the state-of-the-art and emphasize the usefulness of the proposed methodology for radiology image retrieval and classification.

1. Introduction

1.1. Context

Diagnostic radiologists need to maintain high interpretation accuracy while maximizing efficiency in the face of increasing volumes of images per study. They are now confronted with the challenge of efficiently and accurately interpreting cross- sectional studies that often contain thousands of images [1]. Currently, this is largely an unassisted and time-consuming process, and a radiologist's accuracy is established through training and experience. Despite this training, there is substantial variation in interpretation among radiologists [2], and accuracy varies widely [3]. A promising approach to maintain interpretative accuracy in this “deluge” of data is to integrate computer-based assistance into the image interpretation process. Many general-purpose image retrieval systems have been proposed in the literature [4]. Among these systems, an emerging technique that may assist radiology interpretation is content-based image retrieval (CBIR) [5]. This framework assists users in finding similar images within large collections of images. For medical purposes, the role of CBIR is powerful: in addition to enable similarity-based indexing, it could provide computer-aided diagnostic support based on image content and on other meta-data associated with images.

The main idea of CBIR is to search for similar images based directly on their visual contents. Image retrieval is usually performed by image example, where a query image is given as input and an appropriate distance is used to find the best matches in the corresponding feature space [6]. In general, images are indexed using quantitative features extracted from regions of interest (ROI) of the images (e.g., lesions) and focus on their contents (e.g., shape, texture) [7]. Although these features are powerful to describe the image content in an automated fashion, they are often not discriminating enough to comprehensively characterize medical images. In addition, the performance of most CBIR systems is constrained by the low-level properties of these features because they cannot efficiently model the user's high-level expectations [8] (referred to as the semantic gap problem). Since this problem remains unsolved, research in image retrieval focuses on new methods to characterize the image content with higher level semantics, closer to that familiar to the user and potentially more useful in retrieving similar-appearing images [9].

In recent work on image retrieval that incorporates semantics, the images were characterized using a set of semantic terms [10, 11, 12] in a process referred to as “semantic annotation”. Such terms can be directly derived from the terminology provided by the radiologists in their reports [13] or automatically predicted from computational imaging features [14]. The semantic terms can be used to describe a variety of information about the image content (e.g., lesion shape, patterns of enhancement), and they are directly linked to the user's high-level understanding and descriptions of image features [15] (Figure 1). These terms can improve diagnostic decision making by enabling radiologists to search databases of images for cases that are similar in terms of shared high-level imaging features to the cases which they are working on. Based on these considerations, incorporating semantic features into CBIR systems can be a promising attempt to bridge the semantic gap between the visual description of an image and its meaning [16]. However, most of the existing CBIR strategies do not consider the intrinsic semantic properties of the terms during the comparison of the images. Consequently, there is an unmet need that we propose to address by presenting a new framework that includes semantic features in images and that enables retrieval of similar images in large databases.

Figure 1.

A CT image of the liver annotated with semantic terms. The boundaries of the lesion are depicted in red.

1.2. Motivations

When images are described using semantic terms, they are usually modeled as a set of terms, referred to as “bag-of-words” (BOW) [17]. BOW models have been successfully used in natural language processing [18] to capture a summary of the semantics of text based on word content. BOW models are represented as vectors of numerical (or binary) values, where each element represents the probability of occurrence (or presence/absence) of a term. Most of the classical BOW approaches assume that every term describing an image is independent of other features – that there are no intrinsic relations between the words that are contained in a bag. However, such an assumption is often oversimplified; while in text the independence assumption is reasonable, in the case of images annotated with diverse semantic features, this assumption is problematic [19]. The relations among terms are crucial since these features usually have a strong semantic correlation with each other (e.g., relations between anatomy, imaging observations).

Consider an example of medical image clustering with three images ${\mathcal{I}}_{A-C}$. Each image contains a lesion in the liver. We use a basic vocabulary to describe the lesion shape appearing in these images: {ovoid, round, irregular}. The three images are annotated as presented: ${\mathcal{I}}_A$ is annotated with ovoid; ${\mathcal{I}}_B$ is annotated with round; ${\mathcal{I}}_C$ is annotated with irregular. By considering a classical BOW strategy, these three images could be represented in a 3-dimensional space as ${\mathcal{I}}_A=(1,0,0)$, ${\mathcal{I}}_B=(1,0,1)$ and ${\mathcal{I}}_C=(0,0,1)$. Figure 2(a) shows this representation in a classical Euclidean space. By considering this representation, it is difficult to group the three images into relevant clusters of interest because each image is equidistant to every other image. However, we know intuitively that the images ${\mathcal{I}}_A$ and ${\mathcal{I}}_B$ are more similar than the images ${\mathcal{I}}_C$ and ${\mathcal{I}}_C$ (resp. ${\mathcal{I}}_B$ and ${\mathcal{I}}_C$) since the terms ovoid and round are semantically closer than terms ovoid and irregular (resp. round and irregular). Therefore, if we no longer hold the terms in the bag as orthogonal, it seems natural to “bend” the axis and thus enable easy clustering them into two groups (Figure 2(b)).

Figure 2.

Three images characterized with semantic terms related to the shape of the lesions. These images can be modeled by vectors of semantic features into two different spaces.

This basic example highlights the need of considering the semantic relations of the terms for image retrieval purpose. Although several efforts have been conducted in computer vision [20, 21] and medical imaging [22] to integrate semantics into image retrieval applications, most of the proposed approaches are dedicated to automatic annotation of the images with semantic terms, and they do not focus on the potential relations among the semantic terms during the retrieval step. Consequently, there is an opportunity to improve image retrieval applications by considering these semantic aspects. To this end, our approach to improving CBIR considers the semantic relations between the terms when assessing the distance between images described with BOW. Indeed, assessing the distance between vectors is the basis of determining the similarity in most medical information retrieval applications [23, 24]. However, to enable the integration of such semantic relations into a distance function, we have to find solutions to three major problems: (1) how to model the relationship between the terms, (2) how to quantify a semantic proximity between terms, and (3) how to use these relations in computing the distance between images described as BOW.

Recent works in information retrieval [25] have shown that considering controlled vocabularies, such as ontologies, for image annotation can open up new research directions to deal with these problems. Ontologies can be used to model the relations between terms and can provide a solution to deal with our first issue. In addition, the problem of quantifying a semantic proximity between terms belonging to an ontology has been studied in the field of natural language processing [26]. Numerous kinds of measures have been proposed to assess the semantic similarity between terms and can be used as robust solutions to our second issue. Finally, in the domain of histogram comparison, new distances resting on hierarchical merging strategies have been proposed to consider the relative proximity between the bins avoiding bins correlation issues. The cornerstone of these distances is to consider the (intrinsic) multilevel semantic correlations between the distributions modeled by the histograms. Such distances could then be adapted to consider semantic similarity between terms when assessing the distance between BOW and can be considered as a solution to our last issue. Consequently, it appears that, by coupling these three strategies there is an opportunity to consider term semantic relations when comparing BOW in order to improve image retrieval applications.

We describe a new semantic framework, devoted to the retrieval of similar medical images in large databases. The main idea is to annotate the images using terms belonging to a medical ontology and to consider the semantic ontological relations among these terms when comparing the images for visual similarity. By exploiting linguistic relationships between semantic features, this framework could lead to more accurate radiology reporting and diagnoses.

1.3. Outline

This article is organized as follows. Section 2 proposes a state of the art about semantic proximity and vector distances, as well as the different contributions that we present in this article. Section 3 describes the proposed semantic framework, dedicated to the comparison of medical images described with semantic terms. Section 4 gathers experiments enabling to assess the relevance of this framework. Conclusions and perspectives will be found in Section 5.

2. Linguistic proximity and semantic distances

2.1. Image annotation with a structured vocabulary

Image annotation is a complex task that has been widely studied in the domains of computer vision and image retrieval. System performance relies on the choice of the terms being used to describe the content of the images: this choice is highly dependent of the application, the users needs, and the user experience. Consequently, it leads to different possible descriptions for a same image, thwarting good performance of CBIR systems based purely on semantic image descriptions. To deal with this issue, recent works in the semantic domain [27] used controlled vocabularies for annotating the images. A controlled vocabulary provides a set of pre-defined terms with definitions that can facilitate the annotation of large sets of images since it provides standard terms for describing the features in images. In medical imaging, recent works [11] have investigated computer-aided methods to support diagnosis by providing a database of annotated images that can be retrieved by similarity, which further indicates that semantic annotations from a controlled vocabulary can lead to more accurate diagnoses.

Ontologies, which are related to controlled terminologies but also provide explicit specification of relations among terms, provide a formal way to model knowledge [28]. As they are machine-accessible and usually built from a consensus of domain experts, they represent a powerful way to structure semantic terms belonging to a particular knowledge source. In the context of medical imaging, numerous ontologies are being developed to organize biomedical concepts in a comprehensive manner (e.g., Medical Subject Headings (MeSH), International Classification of Diseases (ICD taxonomy), Systematized Nomenclature of Medicine – Clinical Terms (SNOMED CT), Unified Language of Radiology Terms (RadLex)) [29]. Ontologies specify different kinds of taxonomic relations among the terms (e.g., subtype/supertype, homonyms, synonyms relationship) and can be seen as an oriented graph in which semantic concepts are linked by taxonomic relations. As most medical ontologies contained subtype/supertype relations (i.e., is_a relations), we will focus on such relations in the remainder of the article.

An important aspect of ontologies that makes them particularly valuable for CBIR is that their relations can be leveraged to compute semantic likeness between terms. Indeed, their hierarchical structures make it possible to directly assess a semantic proximity value between two terms belonging to the tree structure [30]. Figure 3 presents an extract of the RadLex ontology and the idea of evaluating similarity between terms belonging to a hierarchy. In the next section, we study and compare different measures that have been proposed in the literature to evaluate the proximity between semantic terms contained in an ontology.

Figure 3.

An extract of the RadLex ontology. This figure illustrates the idea of using the ontology structure to evaluate the semantic proximity between two terms of interest denoted as Term a and Term b (highlighted in red).

2.2. Proximity between semantic terms

For the last decades, numerous kinds of measures have been proposed to assess the semantic similarity between terms belonging to an ontology [31]. Theses measures are generally gathered in three groups: Edge-based measures consist of directly inferring the semantic similarity between terms from the ontology structure [32]; Feature-based measures compute the semantic similarity between terms regarding the degree of overlap between sets of ontological features [33] and Measures based on information content exploit the notion of information content, by associating appearance probabilities to each concept in the taxonomy, computed from their occurrences in a text corpus [34].

As specific ontological features and text corpora are not always available in the medical context (and their definitions require considerable human efforts), both feature-based measures and measures based on Information Content may not be well adapted in our applicative context in which more automated strategies are required. In addition, the authors of [35] have experimentally shown that the use of edge-based measures could lead to better results than the ones obtained with information content measures in the biomedical context. For all these reasons and for the sake of genericity, we will focus on the remainder of this section on the use of edge-based approaches to assess the semantic likeness between semantic terms belonging to an ontology.

In order to quantify a semantic similarity value between terms, an intuitive edge-based method has been originally proposed by Rada et al. in [36]. It calculates the similarity between the terms by computing the minimum path length connecting their corresponding ontological nodes via taxonomic links. The underlying idea is that the longer is the path, the more semantically far the terms are. Since the original definition of this measure, several improvements have been proposed in the literature [32]. One of the major improvement has been to also consider the relative depth of the terms in the ontology. Indeed, since a concept becomes more specialized as long as it is recursively refined in the hierarchy, the depth of a term in the ontology is an important dimension. Consequently, most of the proposed measures act by counting the number of taxonomic links from each term to their Least Common Subsumer (LCS) (i.e., the most concrete taxonomical ancestor that subsumes these two terms) and also the number of links of the LCS to the root of the ontology. Among them, Al-Mubaid and Nguyen have proposed in [37] a measure based on a cluster-based strategy that combines both the minimum path length and the taxonomical depth of the considered branches. The definition of this measure was also extended to deal with terms belonging simultaneously to multiple ontologies [38], enabling to evaluate the term similarity from complementary sources of knowledge.

Since these measures enable to consider both the relative path length between the terms and their relative depth in the ontology, they present powerful and comprehensive properties. Furthermore, it has been shown in [35] that these measures outperform other existing ones for the comparison of semantic terms belonging to biomedical ontologies.

2.3. Distance between vectors of semantic features

In general, the similarity between images described with BOW is evaluated by computing the distances between these bags. The classical approaches are based on the vector space model (VSM) that has proved to be very popular in the domain of text retrieval [39]. Such approaches consider a BOW as a fixed-dimensional vector where each vector element represents the probability of occurrence of a term within a document.

In general, vector similarity is evaluated using “element-to-element” distances that only compare the contents of the corresponding elements of the vectors (e.g., Manhattan D_L1 or Euclidean D_L2 distances). As they only compare corresponding vector elements, these distances can be computed linearly and can be used to measure similarities for large datasets. However, these distances ignore the potential semantic proximity between neighboring elements.

Another way of evaluating vector similarity is to consider “cross-element” distances that compare corresponding vector elements as well as non-corresponding ones. Based on this property, these distances account for the semantic proximity between the vector elements. The Cosine-Similarity Measure (CSM) defines the similarity between two vectors to be the cosine of the angle between them, which is identical to the normalized inner product of the two vectors. This measure has proven to be very popular for query-document and document-document similarity in text retrieval [39]. Several extensions of the CSM dealing with hierarchical domain structure and semantic proximity have been proposed in the literature [40, 41]. However, as their computation requires to compare each vector element to all the elements of the corresponding vector (quadratic in the worst case), their computational costs remain hardly tractable when measuring similarities for large datasets.

In the domain of histogram comparison, “cross-bin” distances have been proposed to compare histograms in a “cross-element” fashion [42, 43]. Practically, this can be done by assigning, to each pair of bins, a weight (i.e., a numerical value, called “ground-distance”) modeling the degree of semantic proximity between the compared bins. These weights are usually stored by using a (dis)similarity matrix [42]. Based on this paradigm, some of the authors of this article have recently proposed a cross-bin distance called HSBD (Hierarchical Semantic-Based Distance) resting on a hierarchical merging strategy [44, 45]. Its computation relies on the iterative merging of the semantically closest bins of the histograms to create coarser histograms of higher semantic levels. This hierarchical strategy is the cornerstone of this distance: it enables to consider the (intrinsic) multilevel semantic correlations between the distributions modeled by the histograms. In addition, as the distance value is obtained by computing iteratively a chosen element-to-element distance, its computational cost is lower than quadratic costs required for cross-bin distances.

This distance, which has been involved in image retrieval applications, has provided encouraging results. However, this distance has been proposed in the context of low-dimensional histogram comparison, and the ground-distances between the elements of the vectors have to be manually defined by the user using his/her background knowledge.

2.4. Contributions

Based on these methodological considerations, we propose in this article a new semantic framework, devoted to the retrieval of similar medical database images described with high-level semantic annotations. The contribution of the current study is threefold:

• we employ semantic terms, which belong to a controlled vocabulary extracted from a biomedical ontology, for the annotation of radiological images. These terms can improve diagnostic decision making by enabling radiologists to search databases of images for cases that are similar in terms of high-level visual features to a new query image. The use of a controlled vocabulary for image annotation guarantees the comparability property of the database images characterized with semantic terms;

• we propose to consider the semantic similarity between terms during the retrieval of similar database images. To this end, we use an ontological edge-based measure, which enables to automatically quantify the semantic term similarity from the ontological structure, coupled to the HSBD distance, which takes into account these term similarities when comparing images described as BOW. This strategy provides a potential solution to the actual issues of BOW approaches that assume that every term describing an image is independent of other features;

• we extend the HSBD distance to enable the comparison of high-dimensional vectors of semantic features. The main advantage of this vector distance is to consider the semantic multiscale similarities among the terms when comparing images characterized with BOW, with a lower computational cost than the ones induced by the classical approaches in CBIR.

Use of semantic information associated with images is not new. However the incorporation in medical image retrieval systems of semantic similarities in an ontological fashion is innovative and can be considered as a robust solution to the semantic gap problem. To show the interest of considering the relations among terms in the context of the retrieval of medical images, we propose to apply it to two different tasks: ranking and classification of computed tomographic (CT) images of the liver. We evaluate the effectiveness and the gain of considering the HSBD distance combined to ontological relations to retrieve relevant similar images in a database compared to other distances of the state-of-the-art.

3. Methodology

3.1. Workflow

Our workflow is divided in two main steps: (1) annotation of the query image with semantic terms from an ontology and using them to create vectors of semantic terms that describe the visual content of the image, and (2) comparison of the query to other previously annotated images by computing the vector similarity using the HSBD distance coupled to an ontological measure. The reader may also refer to Figure 4 for a visual outline of this workflow.

Figure 4.

Workflow of the proposed semantic framework for image comparison purpose.

3.2. Notations

An interval on $\mathbb{R}$, bounded by $a,b\in \mathbb{R}$, will be noted [a, b] while an interval on $\mathbb{Z}$, bounded by $a,b\in \mathbb{Z}$, will be noted [[a, b]]. A set $\mathcal{S}$ of k unordered elements e_i with i ∈ [[0, k − 1]] is denoted by {e₀, e₁ …, e_k−1}. A list (vector) L of k ordered elements e_i with i ∈ [[0, k − 1]] is denoted by 〈e₀, e₁, …, e_k−1〉.

3.3. Image annotation and ontologies

Let ${\mathcal{I}}_A$ be a query image. An abnormality in the query image ${\mathcal{I}}_A$ is first identified and circumscribed (either manually or automatically) to capture the boundary of a region of interest (ROI). The next step is to describe the visual features of the ROI using a set of semantic terms. For the sake of generality, the proposed semantic framework can be instantiated with two strategies for choosing these terms: (1) automatic annotation using a machine learning algorithm to predict the presence (or the absence) of the terms from computational imaging features derived from the ROI [46] or (2) manual annotations based on observations made by a radiologist. As the first strategy is out of the scope of this paper, we focus in this work the second strategy. Each region of interest is then presented to the radiologist for manual annotations using terms belonging to a specific ontology denoted as Θ. This task is performed by the physician regarding his/her background knowledge and the semantic terms are captured by a semantic image annotation tool called the electronic Physician's Annotation Device (ePAD) [47].

In order to speed up the manual annotation task and to make the image descriptions comparable, we created data capture templates that contain pre-defined lists of terms taken from the ontology Θ. These terms can be potentially used to describe the content of the images in the considered application. A template example is presented in Table 1. We denote as $\mathcal{X}=\{x_0,x_1,\dots ,x_{k-1}\}$ where the k terms x_i ∈ Θ models the vocabulary used for annotating the images.

Table 1.

The 72 semantic terms used to describe the visual appearance of the liver lesions from CT images. These terms are grouped into specific categories related to different properties of the lesion appearance.

Category	Semantic term	Category	Semantic term
lesion effect on liver	abuts capsule of liver abuts hepatic vein abuts portal vein bulges capsule of liver displaces portal vein distorts hepatic vein infiltrates wall into lumen	lesion rim or capsule	absent rim incomplete rim irregular rim nonuniform rim thick rim thin rim
lesion focality	coalescent multifocal multiple lesions >10 multiple lesions 1–5 multiple lesions 6–10 satellite lesions present solitary lesion	perilesional tissue characteristics	feeding arteries normal perilesional tissue perilesional perf. alteration perilesional steatosis
lesion margin	circumscribed margin cluster of grapes irregular margin lobulated margin mixed margin obscured margin poorly-defined margin smooth margin	lesion brightness	high signal intensity hyperdense hypodense soft tissue density water density
lesion substance characteristics	central scar hemorrhage internal nodules necrosis scar steatosis visible internal vessels	lesion shape	irregularly shaped linear lobular ovoid polygonal rectangular round wedge-shaped
lesion enhancement	enhancing hypervascular nonenhancing	spatial pattern of enhancement	heterogeneous homogeneous mosaic mottled peripheral cont. rim peripheral discont. nodular reticular target
uniformity of enhancement	heterogeneous homogeneous mixed	temporal pattern of enhancement	centripetal fill-in homogeneous fade homogeneous retention homogeneous washout mixed pattern peripheral retention

Once the query image ${\mathcal{I}}_A$ has been annotated, we can straightforwardly build a vector of semantic features A = 〈a₀, a₁, …, a_k−1〉 where each element a_i ∈ A is a binary value representing the presence or the absence of the term $x_i\in \mathcal{X}$. This vector is then considered as a representation of the query image in the semantic feature space.

3.4. Semantic distance between vectors of terms

We describe hereinafter how the HSBD distance has been extended to enable the comparison of vectors of semantic terms by considering the ontological relations of these terms.

3.4.1. Ontological similarity between terms

The computation of the HSBD distance between two vectors A and B of k elements, requires the definition of a k × k dissimilarity matrix ${\mathcal{M}}^{\mathit{dis}}$ modeling the semantic proximity (from 0: equal, to 1: totally different) between the k terms of the considered vocabulary $\mathcal{X}$.

Such dissimilarity matrix ${\mathcal{M}}^{\mathit{dis}}$ that models the semantic relations between all the terms of the considered vocabulary x ∈ X = {x₀, x₁, …, x_k−1} can be defined as

$${\mathcal{M}}^{\mathit{dis}}=\left[\begin{array}{ccc}\hfill \alpha (x_0,x_0)\hfill & \hfill \cdots \hfill & \hfill \alpha (x_0,x_{k-1})\hfill \\ \hfill ⋮\hfill & \hfill \ddots \hfill & \hfill ⋮\hfill \\ \hfill \alpha (x_{k-1},x_0)\hfill & \hfill \cdots \hfill & \hfill \alpha (x_{k-1},x_{k-1})\hfill \end{array}\right]$$ (1)

where α_(xi,xj) ∈ [0, 1] represents the semantic dissimilarity between x_i and x_j that has been provided by the structure of the ontology Θ. To this end, we define α_(xi,xj) as

$${\alpha }_{(x_i,x_j)}={\alpha }_{(x_j,x_i)}=\{\begin{array}{cc}0\hfill & \text{if}{x}_i=x_j\hfill \\ S_{ϴ}(x_i,x_j)\hfill & \text{otherwise}\hfill \end{array}\phantom{\}}$$ (2)

where s_Θ is an extension of the semantic dissimilarity measure proposed by Al-Mubaid and Nguyen in [37]. The original version of this measure and its proposed extension are presented hereinafter.

The computation of this measure relies on a cluster-based strategy that combines both (1) the minimum path length between the terms within the ontological structure and (2) the taxonomical depth of the considered branches.

The minimum path length between the terms is evaluated by counting the number of taxonomic links between the terms relatively to the hierarchical structure of the ontology. Let x_i, $x_j\in \mathcal{X}$ be two semantic terms belonging to the ontology Θ. We define path(x_i, x_j) = {l₀, …, l_n−1} as a set of links connecting the terms x_i and x_j in Θ. Let |path(x_i, x_j)| ≥ 0 be the length of this path. Then, if we consider all the possible paths from x_i to x_j, the minimum path length between these terms is defined as min_∀p |path_p(x_i, x_j)| where p denotes a possible path from x_i and x_j.

The taxonomical depth of the term branches is evaluated using a cluster-based strategy. Starting from the root of the hierarchy, the computation of this measure requires the creation of clusters for each main branch of the ontology structure (i.e., each branch starting after the root is considered as a cluster of terms). The idea is then to assess the common specificity (CS) of two terms by subtracting the depth of their Least Common Subsumer (LCS) from the depth D_c of their common cluster. Such a strategy leads to the following definition CS(x_i, x_j) = D_c − depth(LCS (x_i, x_j)). The common specificity is used to consider that lower level pairs of concept nodes are more similar than higher level pairs. In some particular situations, this feature is useful to counterbalance the behavior of the the minimum path length that is not relevant to realistically evaluate the semantic similarity between terms.

In this work, we have extended the original definition of this measure to normalize it and to give an equal weight to the path length and the common specify features. So, the proposed definition becomes

$$S_{ϴ}(x_i,x_j)=\frac{\log \left(\sqrt{({\min }_{\forall p}\mid {\mathrm{path}}_p(x_i,x_j)\mid -1)\times CS(x_i,x_j)}+\gamma \right)}{{\omega }_{\mathit{norm}}}$$ (3)

where ${\omega }_{\mathit{norm}}={\max }_{\forall i,j}\left(\log (\sqrt{({\min }_{\forall p}\mid {\mathit{path}}_p(x_i,x_j)\mid -1)\times CS(x_i,x_j)}+\gamma )\right)$ is a normalization factor computed as the maximal semantic dissimilarity value between two terms x_i, x_j belonging to Θ. In addition, we set γ = 1 to force the proposed measure to be positive.

3.4.2. Computation of HSBD

Before actually computing the distance between the two vectors A and B of k terms, it is necessary to define a way to hierarchically merge the different semantic terms into clusters (i.e., groups of terms of higher semantic levels). To this end, it is possible to automatically build a dendrogram $\mathfrak{D}$ induced by ${\mathcal{M}}^{\mathit{dis}}$, modeling this merging hierarchy, by using the Ascendant Hierarchical Clustering (AHC) algorithm [48]. This algorithm performs in four steps:

• Step a. Begin with groups containing only one basic term (i.e., k groups where k is the number of terms).

  px

• Step b. Compute the dissimilarity values between every group couples, and update the matrix ${\mathcal{M}}^{\mathit{dis}}$.

  px

• Step c. Merge the two closest groups (i.e., the groups which have the lowest dissimilarity value in ${\mathcal{M}}^{\mathit{dis}}$ modify ${\mathcal{M}}^{\mathit{dis}}$ accordingly (by merging the two lines/columns associated to these two groups).

  px

• Step d. If there are more groups than desired (generally, one group), go to Step b.

This algorithm hierarchically builds clusters of terms while minimizing their intra-group dissimilarity. To compute the dissimilarity values between every group couple, it is necessary to choose a linkage criterion that is the unique parameter of this algorithm. In this work, we have chosen to use the well-known Average Linkage criterion that generally provides satisfactory results [48].

Each one of the s stages of $\mathfrak{D}$ corresponds to a specific clustering results whose clusters model groups of terms at different semantic levels. It is then possible to straightforwardly determine from this hierarchy an order of fusions between the terms of the considered vocabulary in a multiscale fashion (from the leaves to the root). The dendrogram obtained for the vocabulary presented in Table 1 is presented in Figure 8.

Figure 8.

Dendrogram modeling the merging order of the semantic terms presented in Table 1. This dendrogram was automatically built using the algorithm defined in Section 3.4.2. The original terms are represented by color rectangles while the generated terms of higher semantic level are represented by blue disks.

Once the dendrogram $\mathfrak{D}$ has been built (Figure 5–①)), the HSBD distance can be computed. This computation is organized in two main steps:

• Step 1. Hierarchical element-to-element sub-distances computation (Figure 5–②))

  During an iterative merging process (scanning each stage of the dendrogram from the leaves to the root), the vectors A^v and B^v associated to A and B, which are induced by the merging of the terms composing each cluster of the stage S_v, are built. After each iteration, the classical Manhattan “sub-distance” D_L1 is then computed between the couple of (coarser) vectors A^v and B^v created previously. The resulting series of element-to-element sub-distances d⁰, … , d^s–1 enables assignment of vector similarities at different semantic levels.

  px

• Step 2. Element-to-element sub-distances fusion (Figure 5–③3))

  The sub-distances d^v computed for all the stages of $\mathfrak{D}$, and the “energy of merging” required to go from one stage to the next (i.e., the intra-cluster dissimilarity values computed by AHC), are then fused into a function d^inter which is integrated to provide the HSBD distance. Practically, this distance can be obtained by computing the area ${\mathcal{A}}_{{\mathbf{d}}^{\mathit{inter}}}$ under the curve representing the function d^inter. The principle of this step is illustrated in Figure 6. One can note that for semantically similar vectors (Figure 6(a)), the d^v values tend to decrease more rapidly than for dissimilar ones (Figure 6(b)).

For additional details about the computation of the HSBD distance, the reader may refer to [44, 45].

Figure 5.

Computation workflow of the HSBD distance.

Figure 6.

Graphical representation of the d^v values and the function d^inter, computed between example vectors. Depending on the contents of the vectors A and B, the behavior of the function d^inter widely differs.

4. Experimental study

4.1. Experiments

To assess our proposed semantic framework, we applied it in a system for retrieving liver lesions from computed tomography (CT) images. Liver lesions stem from a variety of diseases, each with different (though sometimes overlapping) visual manifestations. Some liver lesions are benign while others may be malignant. The ability to differentiate these lesions efficiently and accurately is important to patient treatment and outcome. We have used the proposed semantic framework for two specific image retrieval tasks: (1) image ranking and (2) image classification. The first task consists of considering an input image as a query and to compare it to the other images of the database to rank them based on their similarities. The second task consists of automatically predicting the diagnoses of the images based on their similarities.

4.2. Materials

4.2.1. Image database and semantic annotations of the lesions

We used 77 CT images of liver lesions in the portal venous phase, including six types of lesion diagnoses (Cyst (# 25 images), Metastasis (# 24 images), Hemangioma (# 14 images), Hepatocellular carcinoma (# 6 images), Focal nodular hyperplasia (# 5 images) and Abscess (# 3 images)) that were used in a previous research study [27]. These types of lesions are common and span a range of visual appearances. Scans were acquired during the time period February 2007 and August 2008 and used the following range of parameters: 120 kVp, 140–400 mAs, and 2.5–5 mm slice thickness. For each scan, the 2D axial slice with the largest lesion area was selected for analysis. We received institutional review board approval for retrospective analysis of deidentified patient images. Figure 7 shows representative examples of this dataset with different lesion diagnoses.

Figure 7.

6 CT images of liver lesions in the portal venous phase. The boundaries of the lesions are highlighted in red.

Our approach requires that lesions on CT images be circumscribed by a 2D ROI drawn manually or automatically. In this experimental study, a radiologist drew and recorded a ROI around the lesion on these images (Figure 7).

Each lesion was manually annotated by a radiologist, who provided semantic terms from the RadLex ontology [49] to describe the visual features of the lesions. A subpart of the RadLex ontology is presented in Figure 3. We extended the RadLex terminology to include a broader array of descriptive terms for this study to more comprehensively describe liver lesions. Our resulting vocabulary was composed of 72 terms that are presented in Table 1. Note that these terms can be grouped into 12 semantic categories (e.g., lesion effect on liver, lesion focality, lesion margin, lesion brightness) related to different visual properties of the lesions. The radiologist used the ePAD system for annotating the images. We created a template to provide the radiologist with a pre-specified set semantic terms for image annotation. All image annotations created by ePAD are stored in compliance with standards developed by the National Cancer Institute's Annotation and Image Markup (AIM) standard format for image metadata [50]. An example of a segmented and annotated image is presented in Figure 1. Semantic annotations required approximately 3 minutes per lesion and the number of terms chosen per lesion ranged from 8 to 15. From each annotation we created a binary semantic feature vector of length 72 to indicate presence or absence of specific observations. Table 2 presents an example of 50 annotated lesions. This table enables to visualize how much overlap in semantic feature space the classes of images belong.

Table 2.

Example of 50 annotated lesions (in column) belonging to six different types. Each lesion is vertically represented as a vector of semantic terms where each vector element models the presence (1) or the absence (-) of a specific term. For the sake of readability, only the terms with an occurrence higher than 2 are represented.

4.2.2. Initialization of HSBD

The computation of the HSBD distance between two vectors, requires the definition of a dissimilarity matrix ${\mathcal{M}}^{\mathit{dis}}$ which models the semantic proximity between the terms of the controlled terminology. We have applied the proposed semantic dissimilarity measure s_Θ (Equation(3)) on the RadLex ontology to fill this matrix in order to evaluate the ontological relations between each possible couple among the 72 terms.

From this matrix, a dendrogram was automatically built (Section 3.4.2) to define the merging order of these different terms (Figure 8). This dendrogram was visually analyzed by a radiologist to assess the thematic validity of the generated terms of higher semantic levels. One can visually note that this structure starts by merging semantic terms belonging to the same semantic categories (i.e., with similar colors).

4.2.3. Comparison to other distances

To assess the relevance of the proposed semantic framework, and in particular the use of the HSBD distance for image comparison for similarity assessment, we compared the results obtained with this distance to the results obtained by using other existing distances (Table 3) that are detailed hereinafter.

Table 3.

Distances used for comparison purpose.

Symbol	Implementation	Distance	Type
D L1	[51]	Manhattan distance	Non-semantic
D L2	[51]	Euclidean distance
D ∩	[52]	Intersection distance
		px
EMD	[43]	Earth Mover's Distance	Semantic
HSBD	–	Hierarchical Semantic-Based Distance

To highlight the benefits of considering the semantic relations between the features composing the vectors, we have compared our results to the results obtained with the classical Manhattan D_L1, Euclidean D_L2 and Intersection D_∩ distances that do no take into account the semantic similarities. For two vectors A and B with k elements, these distances can be formulated as $D_{L_1}(A,B)={\Sigma }_{i=0}^{k-1}\mid a_i-b_i\mid$ and $D_{L_2}(A,B)={({\Sigma }_{i=0}^{k-1}\mid a_i-b_i{\mid }^2)}^{1∕2}$, where ∩_max(A, B) denotes the 344 maximal intersection value between two vectors A and B (e.g., A = B).

To highlight the benefits of using the proposed distance instead of using an alternative semantic distance, we have also compared our results to the ones obtained with the Earth Mover's Distance (EMD) [43]. The principle of this distance is to estimate the cost of mapping two vectors. In the EMD, two features are considered as the “earth” and the “holes”, respectively. Then, the distance measure problem is transformed into the earth moving problem, where the minimum cost of moving all the “earth” into the “holes” is calculated. To make EMD and HSBD comparable, this cost of mapping two vector elements has been computed using the ontological proximity values stored in ${\mathcal{M}}^{\mathit{dis}}$ (in the same way as for the computation of HSBD). The main drawback of the EMD is its quadratic computational cost.

In the two experiments presented here, namely image ranking and image classification, we repeated the experiments by varying the distance used for comparing the images.

4.3. Task 1: Image ranking

Image ranking consists of considering an input image as a query and to compare it to the other images of the database to rank them based on their similarities. This task could be very useful since it can help radiologists to interactively search the database to find similar medical cases and patient cohorts based on the visual appearance of the images. We designed our evaluation to assess the ability of our semantic framework to rank CT images of liver lesions stored in the database in order of similarity relative to a query lesion. We evaluated the effectiveness and the gain of considering the HSBD distance combined to ontological relations to retrieve relevant similar images compared to other distances of the state-of-the-art. We considered two strategies to evaluate the performance of our system for image ranking: the first strategy consists of evaluating the ranking results relatively to a similarity reference standard defined for pairs of images while the second strategy consists of evaluating these results relatively to the diagnoses of the retrieved images. We describe hereinafter the two performance evaluation protocols and the results obtained.

4.3.1. Evaluation with a reference standard

Performance evaluation

As it was not conceivable in the context of this study to generate a similarity reference standard for the N = 77 considered images (would have required 2926 “manual” comparisons of image pairs), we used the reference standard of image similarity that has been proposed in [11], enabling evaluation of image retrieval performances by using the semantic features. This reference standard is composed of a subset of N = 30 CT images of liver lesions described previously (13 cysts, 7 hemangiomas and 10 metastases). These 30 images were selected based on being the first set of cases collected. Two radiologists viewed each pair of images twice and reached a mutual consensus opinion on a similarity measure for the pair (1, very similar; 2, similar; 3, not similar) by addressing similarity of texture, boundary shape, and sharpness. Thus, with this reference standard, a perfect retrieval system would return a sequence of images with similarities monotonically increasing from 1 to 3.

We used normalized discounted cumulative gain (NDCG) [53] to evaluate performance. The NDCG index is a standard technique used to measure the effectiveness of information retrieval algorithms when ground truth is available, as represented by our three-point similarity scale defined previously. NDCG is used to measure the usefulness (gain) on a scale of 0 to 1 of K retrieved lesions on the basis of their positions in the ranked list compared with their similarity to the query lesion according to a separate reference standard. The discounted cumulative gain (DCG) is evaluated with the weight of each retrieved lesion discounted at lower ranks. The DCG at a particular rank position K is defined as:

$${\text{DCF}}_{\mathrm{K}}=\sum _{i=1}^K\frac{2^{{\mathit{rel}}_i}-1}{{\log }_2(i+1)}$$ (4)

where rel_i is the graded similarity of the result at position i in the ranked list. Comparing a search engine's performance from one query to the next cannot be consistently achieved using DCG alone, so the cumulative gain at each position for a chosen value of K should be normalized across queries. This is done by sorting retrieved images of a result list by relevance (relatively to the reference standard), producing the maximum possible DCG till position K, also called Ideal DCG (IDCG). For a query, the NDCG is then computed as:

$${\text{NDCG}}_{\mathrm{K}}=\frac{{\mathit{DCG}}_K}{{\mathit{IDCG}}_K}$$ (5)

Thus, for a given K, higher NDCG(K) means more lesions similar to the query image are ranked ahead of dissimilar ones, with NDCG(K) equal to 1 implying perfect retrieval of K images.

The NDCG values for all queries can be averaged to obtain a measure of the average performance of a search engine's ranking algorithm.

Experiments and results

We sequentially withheld each image from the 30-images dataset and ranked the remaining 29 images according to the HSBD distance and to the other distances used for comparison purpose (Table 3). This guarantees that each image of the dataset was compared to all the other images. For each query image, we computed the mean NDCG value (and the best and worst retrieval values) at each K = 1, …, N – 1.

Figure 9 shows the NDCG scores obtained for the different distance metrics that were considered. It should be noticed that the experiment with the Manhattan distance D_L1 corresponds to the same experiment proposed in [11] with only semantic features. We consider these previous results as an experimental baseline. From this graph, one can note that the HSBD distance appeared to yield the best overall results, with mean case retrieval NDCG greater than 0.96 and greater than 0.91 for all values of K. For K = 10, the mean case retrieval NDCG was 0.96 for the HSBD distance, implying nearly perfect retrieval of 10 images. Results obtained with the EMD distance yielded excellent overall results, with mean case retrieval NDCG greater than 0.95 and greater than 0.90 for all values of K. The results obtained with the Manhattan distance produced a mean case retrieval NDCG of approximately 0.94, and greater than 0.91 for all values of K. For K = 10, the Manhattan distance produced a mean case retrieval NDCG of 0.92. Manhattan and Euclidean distances had the same ranking behavior. The Intersection distance appeared to yield the worst overall results, with mean case retrieval NDCG lower than 0.88 and lower than 0.95 for all values of K. Finally, the NDCG best and worst retrieval values were higher when the HSBD distance is used for most values of K.

Figure 9.

NDCG(K) plots, where K is the number of images retrieved for different distances in a set of 30 images containing cysts, metastases and hemangiomas.

We also used the paired Wilcoxon (sign-rank) statistical test [54] to test the null hypothesis that there is no significant difference in the NDCG scores obtained from highest-score distances (Manhattan, EMD and HSBD distances). As we only compare two paired groups (HSBD versus Manhattan and HSBD versus EMD), we do not need to use the Friedman test that is required for the comparison of three or more matched groups. Figure 10 presents a plot of the Z-score comparing HSBD to EMD/Manhattan, by varying K (where K is the number of retrieved images). The horizontal line is at Z = 1.96, the value for a (two-sided) 0.05 p-value; the Z-score should be above the line for HSBD to be significantly better than other metrics. For low values of K (K ≤ 3), HSBD is not better than the Manhattan distance, and for high values of K (K ≥ 24), it is not better than EMD. However, for intermediate values of K (K ∈ [[4, 23]]), HSBD is significantly stronger than the other compared distances (except for K = 7 and K = 12). For K = 5 and K = 10 (that represent usually cases of interest for radiologists), the Z-score is equal to Z = 2.9 for HSBD versus EMD and equal to Z = 2.1 for HSBD versus Manhattan. We propose potential methodological explanations to these statistical results:

1. for low values of K, we believe that the Manhattan distance performs better to retrieve similar images because of its inherent element-to-element property enabling to retrieve exactly similar vectors of terms;

2. for intermediate values of K, we believe that the HSBD distance performs better to retrieve semantically similar images (sharing semantically similar annotations) because of its ability to retrieve vectors of terms that are not strictly equal but are semantically similar (e.g., round and ovoid);

3. for high values of K, we believe that the EMD distance performs better to retrieve less semantically similar images (but with a considerably higher computational cost) because of its ability to deal with outliers by matching terms that are not semantically similar (e.g., internal nodules and normal perilesional tissue) but that are occasionally used to describe images belonging to the same category.

These statistical results confirm that the proposed semantic framework could lead to excellent information retrieval performance and the importance of considering the semantic relations among the terms when comparing and ranking images characterized by semantic annotations.

Figure 10.

Z-score from the paired Wilcoxon (signrank) test comparing HSBD to EMD/Manhattan (in terms of NDCG scores), by varying K (where K is the number of images retrieved).

While the NDCG curves show performance over all possible query images, Figure 11 exemplifies retrieval results by using two types of lesion (cyst and metastasis). Perfect retrieval would result in a ranked order of images with monotonically increasing reference standard dissimilarities. Although this example does not yield a perfect ranking, it yields a reasonable one. Images in the reference standard that were judged to be dissimilar to the query image (similarity of 3) rarely appeared ahead of images judged to be similar (similarity of 1) to the query image.

Figure 11.

Examples of retrieval for a cyst query (top left column) and a metastasis query (top right column). Dissimilarity rankings go from lowest (upper left corner) to highest (lower right corner). For each image, data are HSBD computed dissimilarity scores, and data in parentheses are dissimilarity scores from the reference standard. Both are measured on a scale of 0 to 3.

4.4. Evaluation with the diagnoses

Performance evaluation

To evaluate the ability of our retrieval system to find similar images, we also tested the sensitivity and specificity of retrieving images of the same diagnosis in our database of N = 77 lesions belonging to six different types. We performed a leave-one-out test on the retrieval algorithm by querying each lesion against the remaining lesions in the database. We then assessed the sensitivity and specificity for retrieving images having the same diagnosis as the query image. The sensitivity and specificity of the top K retrieval results, with K = 1, …, N – 1 were computed. For a fixed value of K, the sensitivity (i.e., true-positive rate) was calculated by the number of identical diagnoses in the K query results divided by the total number of images in the database with the same diagnosis as the query image. The specificity (i.e., true negative rate) is calculated by the number of non-identical diagnoses in the K query results divided by the total number of non-identical diagnoses in the database. The performance was analyzed using mean receiving operator characteristic (ROC) curves. Individual ROC curves for each lesion were defined by the values of the sensitivity and specificity computed for each value of K. We used piecewise linear interpolation to define 21-point curves to compute the mean values at each of the 21 sensitivity coordinates. The area under the ROC curve indicates the potential effectiveness within the framework of a decision support system, with the maximum area of 1 being optimal.

Experiments and results

Figure 12 presents the results of our evaluation using the ROC and AUC scoring functions. Each mean ROC curve represents the effect of varied numbers of similar images retrieved on the sensitivity and specificity of the retrieval results. Graphs a and b show the results for the two diagnoses containing more than twenty samples (cyst and metastasis); graph c shows the results for all types of lesions combined. The AUC values are shown parenthetically. It should be noticed that the experiment with the Intersection distance D_∩ corresponds to the same experiment proposed in [27]. We consider these previous results as an experimental baseline. All queries yielded ROC curves above the diagonal (dashed line), indicating that the retrieval results favor lesions with similar diagnoses. For all types of lesions and for each considered distance the observed mean ROC curve area was greater than 0.8. The HSBD distance appeared to yield the best overall results, with mean ROC curve areas equal to 0.985 and 0.869 for cyst and metastasis. Manhattan and Euclidean distances had the same ranking behavior and appeared to yield the worst overall results, with mean ROC curve areas equal to 0.972 and 0.813 for cyst and metastasis. The Intersection and EMD distances appeared to lead to intermediate results with mean ROC curve areas equal to 0.961, 0.836 and 0.981, 0.847 for the two considered diagnoses. We also combined the 21-point ROC curves for all queries encompassing six different lesion image types. The AUC of the mean ROC curve was 0.929 for the Intersection distance and 0.976 for HSBD, suggesting that our semantic-based system is highly accurate, overall.

Figure 12.

Mean ROC curves. Plots show the sensitivities and specificities for retrieving images having the same diagnosis as a query image using the HSBD distance.

In order to assess the statistical significance of AUC scores, we computed the difference in the AUC scores obtained from the different distances (Table 3) for the two diagnoses containing more than twenty samples (cyst and metastasis). The AUC differences have been computed by considering all the individual ROC curves obtained for each query image belonging to each diagnostic category. Figure 13 presents histograms showing the AUC difference scores that compare HSBD to Manhattan/Euclidean/Intersection/EMD when retrieving cyst and metastasis lesions. From these histograms, one can note that for cyst retrieval, the HSBD AUC scores are always higher than the ones obtained from the Manhattan, Euclidean, Intersection and EMD distances (except for one case where EMD performs better). When retrieving metastases, HSBD performs always better than the Manhattan and Euclidean distances, but performs sometimes worst than the EMD and Intersection distances for particular cases.

Figure 13.

Difference of AUC values comparing the HSBD distance versus the Manhattan, Euclidean, Intersection and EMD distances for cyst and metastasis. The AUC differences have been computed by considering all the individual ROC curves obtained for each query image belonging to each diagnostic category.

We also used the paired Wilcoxon (sign-rank) statistical test to test the null hypothesis that there is no significant difference in terms of AUC between the different distances. This statistical test led to 18 sub-tests, with a Bonferroni significance level of 0.05/18 = 0.0028. So any comparison with a p-value lower than 0.0028 can be considered significantly different. For the retrieval of cysts we obtained p-values equal to p ≤ 0.0021 for HSBD versus Manhattan, Euclidean, Intersection and EMD distances. For the retrieval of metastases we obtained p-values equal to p ≤ 0.0026 for HSBD versus the other distances. From these comparisons we deduce that the ROC curves obtained with the HSBD distance are statistically better than those obtained with the Manhattan, Euclidean, Intersection and EMD distances. These AUC comparison results confirm the relevance of our semantic-based system in terms of image retrieval.

4.5. Task 2: Image classification

Image classification consists of automatically predicting the class/category of a query image. This task could be very useful since it can help radiologists to determine the diagnosis of a new image by searching databases of radiological images that are similar in terms of shared imaging features.

We designed our evaluation to assess the ability of our semantic framework to assist us in classifying CT images of liver lesions stored in the database. In this study, we have applied the k-NN (k-nearest neighbor) classification algorithm, which can be easily adapted to perform the prediction task by considering a specific distance.

4.5.1. Performance evaluation

We have computed both standard global and local evaluation indexes to evaluate classification performance. We have used the six classes inferred by the diagnoses of the lesions.

To assess the global relevance of the prediction results, we calculated global classification accuracy indexes. For each experiment, we computed the classification accuracy, and the Kappa index $\mathcal{K}$. The Kappa index takes value in [0, 1] and decreases as the classification is in disagreement with the ground-truth. We also computed different indexes that represent the error of prediction.

Local evaluation indexes independently assess the prediction of each class. We computed: the rate of false positives, denoted by f^(p), the rate of false negatives, denoted by f⁽ⁿ⁾, and the rate of true positives, denoted by t^(p). These measures were used to estimate the precision $\mathcal{P}$ and the recall $\mathcal{R}$ of the results. To fuse these measures, we computed the standard F-measure $\mathcal{F}$ which is the harmonic mean of precision and recall. We also computed the AUC corresponding to the area under the ROC curve. All these indexes take values in [0, 1].

4.5.2. Experiments and results

The HSBD distance was integrated into the k-NN classification algorithm to compare the classified vectors representing the images of the 77-images dataset. To compare HSBD to other existing distances, we also ran the k-NN algorithm using the classical element-to-element distances D_L1, D_L2, D_∩ and the EMD distance (Table 3). We use leave one out classification: we train on 76 images, and test on a single image (for each of the 77 images). We ran the k-NN algorithm using odd integer k ∈ [[1, 5]] as the neighbor count.

The global evaluation results obtained on our dataset are presented in Table 4. From this table, one can see that the best global evaluation scores obtained appear to be always higher when the k-NN algorithm is run with the HSBD distance instead of the Manhattan, Euclidean, Intersection and EMD distances. The worst global evaluation scores have been obtained when the k-NN algorithm is run with the Manhattan and Euclidean distances. From this table, we also experimentally found that setting k = 1 led to the best overall results: this value was the best compromise to either minimize the error classification rate and to only include the nearby samples.

Table 4.

Global evaluation results for the image classification task.

k-NN classification (k = 1)
Index	D L1	D L2	D ∩	EMD	HSBD
Classification accuracy (%)	87.41	92.36	93.02	93.83	95.88
Kappa index $\mathcal{K}$	0.82	0.86	0.87	0.89	0.92
Mean absolute error (%)	0.06	0.04	0.04	0.03	0.03
Root mean squared error (%)	0.19	0.14	0.14	0.13	0.12
Relative absolute error (%)	27.15	15.65	14.23	13.48	11.78
k-NN classification (k = 3)
Index	D L1	D L2	D ∩	EMD	HSBD
Classification accuracy (%)	86.32	87.32	89.12	94.61	94.61
Kappa index $\mathcal{K}$	0.78	0.79	0.82	0.91	0.91
Mean absolute error (%)	0.08	0.06	0.06	0.04	0.04
Root mean squared error (%)	0.21	0.18	0.17	0.14	0.14
Relative absolute error (%)	29.12	25.61	19.66	15.87	15.87
k-NN classification (k = 5)
Index	D L1	D L2	D ∩	EMD	HSBD
Classification accuracy (%)	83.01	84.77	85.39	89.12	93.83
Kappa index $\mathcal{K}$	0.75	0.76	0.77	0.79	0.87
Mean absolute error (%)	0.11	0.09	0.07	0.06	0.03
Root mean squared error (%)	0.28	0.27	0.25	0.17	0.13
Relative absolute error (%)	35.12	27.16	23.49	19.66	13.48

The local evaluation results of the k-NN algorithm (with k = 1) are presented in Table 5. From this table, one can see that the F-measure and AUC scores obtained for most of the extracted classes appear to be higher when the k-NN algorithm is run with the HSBD distance instead of the Manhattan, the Euclidean, the Intersection or the EMD distances. In particular, classification results obtained with the HSBD distance show nearly perfect prediction values for images belonging to the cyst, metastasis, hemangioma and abscess categories. Such local evaluation results mean that the HSBD distance enhances the precision and the recall of the results.

Table 5.

Local evaluation results for the image classification task.

k-NN classification (k = 1)
Class	Index	D L1	D L2	D ∩	EMD	HSBD
Cyst	t (p)	1.00	1.00	1.00	1.00	1.00
	f (p)	0.23	0.23	0.06	0.06	0.02
	$\mathcal{P}$	0.68	0.68	0.89	0.89	0.96
	$\mathcal{R}$	1.00	1.00	1.00	1.00	1.00
	$\mathcal{F}$	0.81	0.81	0.94	0.94	0.98
	AUC	0.97	0.97	1.00	1.00	1.00
Metastasis	t (p)	0.67	0.67	0.67	0.92	0.92
	f (p)	0.04	0.04	0.03	0.03	0.04
	$\mathcal{P}$	0.89	0.89	0.88	0.91	0.92
	$\mathcal{R}$	0.67	0.67	0.65	0.90	0.92
	$\mathcal{F}$	0.76	0.76	0.74	0.90	0.92
	AUC	0.98	0.98	0.95	0.98	0.99
Hemangioma	t (p)	0.93	0.92	0.93	1.00	1.00
	f (p)	0.05	0.06	0.05	0.03	0.00
	$\mathcal{P}$	0.81	0.80	0.81	0.88	1.00
	$\mathcal{R}$	0.93	0.91	0.93	1.00	1.00
	$\mathcal{F}$	0.87	0.85	0.87	0.93	1.00
	AUC	0.95	0.93	0.95	1.00	1.00
Hepatocellular carcinoma	t (p)	0.17	0.50	0.28	0.33	0.50
	f (p)	0.00	0.03	0.01	0.01	0.01
	$\mathcal{P}$	1.00	0.60	0.64	0.67	0.75
	$\mathcal{R}$	0.17	0.50	0.32	0.33	0.50
	$\mathcal{F}$	0.29	0.55	0.42	0.44	0.60
	AUC	0.80	0.80	0.90	0.91	0.93
Focal nodular hyperplasia	t (p)	0.20	0.40	0.59	0.60	0.60
	f (p)	0.03	0.01	0.06	0.06	0.04
	$\mathcal{P}$	0.33	0.67	0.42	0.43	0.50
	$\mathcal{R}$	0.20	0.40	0.59	0.60	0.60
	$\mathcal{F}$	0.25	0.50	0.48	0.50	0.55
	AUC	0.86	0.76	0.81	0.83	0.91
Abscess	t (p)	0.67	0.33	1.00	1.00	1.00
	f (p)	0.00	0.00	0.00	0.00	0.00
	$\mathcal{P}$	1.00	1.00	1.00	1.00	1.00
	$\mathcal{R}$	0.67	0.33	1.00	1.00	1.00
	$\mathcal{F}$	0.80	0.50	1.00	1.00	1.00
	AUC	1.00	1.00	1.00	1.00	1.00
Weighted average	t (p)	0.75	0.87	0.91	0.92	0.94
	f (p)	0.10	0.04	0.01	0.01	0.01
	$\mathcal{P}$	0.78	0.86	0.91	0.92	0.94
	$\mathcal{R}$	0.75	0.87	0.92	0.92	0.94
	$\mathcal{F}$	0.73	0.86	0.91	0.92	0.93
	AUC	0.93	0.94	0.94	0.92	0.97

We have also compared the classification results obtained with the HSBD distance to the ones obtained by classifying the data using the Support Vector Machines (SVM) classifier algorithm trained on the dataset using a Gaussian kernel. We empirically selected the best kernel parameter values. During the learning process, this algorithm automatically optimizes the weight given to the attributes from the data. We obtained a global classification accuracy of 84.81% with this classification algorithm while the k-NN algorithm combined to HSBD led to an accuracy of 95.38%. This suggests that standard attribute selection and weighting strategies embedded into the SVM algorithm performs worse than our semantic strategy to consider the semantic correlations between the terms used to describe the visual contents of the lesions.

4.6. Discussion

We have developed an image retrieval system based on the HSBD distance that enables considering the semantic relations among the terms describing the images. Due to its theoretical low computational cost [44, 45], this framework can be used to deal with large datasets. Use of semantic information associated with images is not new. However the incorporation in medical image retrieval systems of semantic similarities in an ontological multiscale fashion is innovative. Finally, our validations in the context of medical imaging suggest the potential relevance and usefulness of HSBD for medical image retrieval and classification tasks: retrieval accuracy NDCG score of more than 0.96 was obtained on a 30-images dataset while classification results of more than 95% were obtained on a 77-images dataset. For comparison purpose, the use of the EMD distance, which is an alternative distance metric that considers all the existing relations among the terms, led to results retrieval accuracy of 0.95 and classification results of 93%. These comparison results suggest that the proposed framework can be considered as a wise trade off between the high computational cost of EMD and the granularity property of HSBD. The use of classical distances (e.g. Manhattan or Euclidean) that do not take into account relations among the terms led to lower accuracy results, showing again the interest of our framework. Our results are competitive with the state-of-the-art methods and demonstrate that the proposed HSBD distance may outperform other distances when comparing vectors of semantic terms.

Our work has some limitations. The current implementation of our semantic framework requires full manual annotations of the query image for retrieval. We plan to integrate a machine learning approach [14, 22, 55] to automatically predict the annotations from quantitative imaging features extracted from the images. Another consequence of the difficulty of the data collection is the limited size of the dataset considered in this study (77 CT 2D images were annotated by one radiologist). This dataset of mono-annotated images can lead to a potential bias in the presented retrieval results. In addition, the reference standard used in the image ranking task was small and contained only three diagnoses. Consequently, the performance results may be under-represented because of the challenge to develop larger reference standards. In the future we plan to increase the number of cases and the number of diagnoses. Another limitation is that the similarity between the terms relies both on the quality of the ontology itself and the considered term similarity measure. We plan to enhance the current framework by considering simultaneously different ontologies besides RadLex. Finally, we will explore additional approaches to evaluate the semantic similarity between the terms, such as using other ontology similarity measures (i.e., feature-based measures and measures based on information content) and including quantitative features extracted from the images that describe the visual appearance of the terms. The intuition behind using two distinct ways to compute term similarity is that coupling high-level knowledge extracted from an ontology to low-level information extracted from the image could be complementary and could lead to a semantic similarity metric closer to that familiar to the user.

5. Conclusion

We present a new semantic framework that enables comparing the similarity of images based on high-level semantic annotations from an ontology. A unique aspect of our approach is the consideration of semantic distance between ontological terms that describe the semantic image contents. Such annotations could in future be supplemented by computational features that are often not sufficient to characterize complex objects of interest in an accurate and comprehensive fashion. In addition, thanks to a hierarchical distance coupled to an ontological metric, this framework incorporates the semantic correlations among the terms used to describe the images. To validate this framework, we applied it to the ranking and the classification of medical images of the liver. The results obtained show reasonable retrieval scores compared with an independently constructed pairwise image similarity of liver lesions visible on portal venous CT images. The semantic framework we have developed is generalizable and can be easily adapted to other anatomic and diagnostic scenarios in which CT and other imaging modalities are used. Ultimately, our approach could provide real-time decision support to practicing radiologists by showing them similar images with associated diagnoses and, where available, responses to various therapies and outcomes.

Research highlights

• We propose an image retrieval system that considers the semantic of medical images.

• It relies on semantic annotation of the images and evaluation of their similarities.

• Image similarity is computed using ontological relations in a hierarchical fashion.

• It provides a means of capturing the semantic correlations between image contents.

• Evaluation on the retrieval of liver CT images proved the interest of this system.