Dynamic “Inline” Images: Context-Sensitive Retrieval and Integration of Images into Web Documents

Abstract

Integrating relevant images into web-based information resources adds value for research and education. This work sought to evaluate the feasibility of using “Web 2.0” technologies to dynamically retrieve and integrate pertinent images into a radiology web site. An online radiology reference of 1,178 textual web documents was selected as the set of target documents. The ARRS GoldMiner™ image search engine, which incorporated 176,386 images from 228 peer-reviewed journals, retrieved images on demand and integrated them into the documents. At least one image was retrieved in real-time for display as an “inline” image gallery for 87% of the web documents. Each thumbnail image was linked to the full-size image at its original web site. Review of 20 randomly selected Collaborative Hypertext of Radiology documents found that 69 of 72 displayed images (96%) were relevant to the target document. Users could click on the “More” link to search the image collection more comprehensively and, from there, link to the full text of the article. A gallery of relevant radiology images can be inserted easily into web pages on any web server. Indexing by concepts and keywords allows context-aware image retrieval, and searching by document title and subject metadata yields excellent results. These techniques allow web developers to incorporate easily a context-sensitive image gallery into their documents.

Introduction

The Internet has opened new avenues for creating, integrating, and distributing medical knowledge. Many of the recent innovations—termed “Web 2.0”—seek to integrate existing web resources and make them more dynamic, collaborative, and interactive^1–3. Integrating relevant images into web-based information resources adds value for research and education. The present work tested the feasibility of a simple, automated, context-sensitive platform to integrate relevant radiology images into a target web page’s content. There were two specific aims: (1) to automatically integrate relevant images based on the content of the target document, and (2) to make the process virtually transparent from the standpoint of the web site developer.

Materials and Methods

An online radiology reference, CHORUS (Collaborative Hypertext of Radiology; http://chorus.rad.mcw.edu), was selected as the collection of target documents. The 1,178 textual documents in this web-based collection describe diseases, radiological findings, and pertinent anatomy⁴. Each CHORUS document included Dublin Core metadata elements⁵, including the “title” and “subject” metadata. The subject metadata were assigned by the SAPHIRE system⁶ and had been incorporated into the CHORUS web site for more than 10 years as part of an effort to improve medical document retrieval using metadata⁷. The subject metadata terms were chosen from the Medical Subject Headings (MeSH®) vocabulary developed by the US National Library of Medicine (NLM) to index the biomedical literature in MEDLINE and PubMed. For example, the CHORUS document on optic nerve glioma incorporated the metadata (“META”) tags:

<META NAME="DC.title" CONTENT="optic nerve glioma">

<META NAME="DC.subject" CONTENT="(SCHEME=MeSH) Cranial Nerve Neoplasms">

American Roentgen Ray Society (ARRS) GoldMiner™ (http://goldminer.arrs.org), a radiology image search engine, was employed to identify pertinent images. As of February 2008, GoldMiner incorporated 176,386 images from open-access articles in 228 online peer-reviewed journals. The images were indexed by keywords and by concepts from the NLM’s Unified Medical Language System (UMLS) Metathesaurus® (http://umlsinfo.nlm.nih.gov)^8,9. Concepts were extracted from the unstructured text of figure captions using the NLM’s MetaMap Transfer® (http://MMTx; mmtx.nlm.nih.gov) software¹⁰. A web-based search engine interface provided access to the GoldMiner image library¹¹.

The web server software scanned each target document to identify the document’s title and topic metadata. The title text was used as a query string; if no matching images were found, then the subject metadata string was used. Up to four images were selected. The system precompiled Hypertext Markup Language (HTML) text to display the images and their captions. The system stored that text string in the database and indexed it by the uniform resource locator (URL) of the referring web page. In this way, the server needed only to search for the referring page’s URL and to return the corresponding precompiled HTML text string.

Preliminary analysis was conducted to assess the relevance of the retrieved images to the target documents. A strict definition of “relevance” was applied. It was not sufficient for a figure caption to include one or more words from the query term; a retrieved image was considered relevant only if it displayed the imaging finding, anatomy, or disease described in the title or subject of the document. For example, an image whose caption included the words “renal” and “calculi” might not be considered relevant to a document on renal calculi unless the image showed the requisite finding. The author manually reviewed the relevance of the retrieved images was for 20 randomly selected CHORUS documents.

The CHORUS documents were modified on their server to insert a single line of HTML text into each document.

<SCRIPT TYPE="text/javascript" SRC="http://goldminer.arrs.org/inline.php">

</SCRIPT>

This command invoked a server-side program that returned the precompiled JavaScript code to incorporate up to four images as an “image gallery” within the document. This image gallery was inserted “inline” into the web document; that is, the images were incorporated in real-time into the document, rather than being viewed through a link to a separate web page. When the server detected a known referring web page, it returned the HTML instructions to display a set of images within the body of the referring document. The command line did not, itself, include any parameters to specify the search; the GoldMiner system determined which document had called the command by examining the HTTP_REFERER parameter, which specifies the URL of the referring document. This approach greatly simplified the task of modifying the target documents to include the special JavaScript command. The web server was based on open-source technologies: the software was written in the PHP programming language (version 5.1.2; www.php.net), and used the MySQL relational database management system (version 5.0; MySQL AB; www.mysql.org).

Results

The client- and server-side scripts functioned properly and successfully integrated the GoldMiner Image Gallery into the target documents when images were identified. At least one potentially relevant image was retrieved in real-time for display for 1,081 (92%) of the 1,178 CHORUS documents. For 1,015 CHORUS documents (86%), four or more images were identified; a maximum of four images were displayed.

Of the 20 CHORUS documents reviewed, two had no images; the other 18 documents each included four displayed images. Of the 72 images displayed, 69 were judged to be relevant to the target document; hence, the estimated precision was 96% (95% confidence interval, 91–100%). For example, of four images retrieved for the CHORUS document on “spleen cyst,” three showed splenic cysts and were counted as relevant. The fourth image was not considered relevant: it did not show a splenic cyst even though its caption contained the words “spleen” and “cyst”. For the CHORUS document on “bagassosis,” although the four retrieved images did not contain that word in their captions, all four were considered relevant because they showed hypersensitivity pneumonitis (of which bagassosis is a subtype) and were pertinent to the document’s MeSH subject term, “Alveolitis, Extrinsic Allergic.”

The image gallery included JavaScript commands to display an abstract of the figure caption when a user hovered over an image. (Fig. 1). Each thumbnail image was linked to the full-size image at the original web site (Fig. 2). Users could click on the “More” link to search the image collection more comprehensively and, from there, link to the full text of the article.

Fig 1.

A screen shot shows how images from the ARRS GoldMiner™ image library are integrated into a document. When a user “hovers” (places the mouse) over an image, the browser displays the source of the image and an abbreviated caption, as shown in the figure. If the user clicks on an image, the browser is directed to the full-size image at the original source web site.

Fig 2.

The user has clicked on one of the thumbnail images in the Image Gallery display. A new browser window displays the full-size image. Reprinted with permission from Takanashi et al.¹².

The ARRS GoldMiner server retrieved images from its database using a proprietary algorithm that matched the target document’s metadata with words and concepts from figure captions and article titles in GoldMiner's database. The MeSH vocabulary is one component of the medical knowledge model used to index the GoldMiner collection, so these terms provided a natural bridge between the CHORUS documents and GoldMiner images. Such direct matching is helpful, but not required, as GoldMiner can search using arbitrary terms.

Discussion

This work demonstrates the feasibility of real-time integration of text with potentially relevant images for education and clinical decision support. A gallery of relevant radiology images can be inserted easily into web pages on any web server. Indexing by concepts and keywords allows context-aware image retrieval, and searching by document title and subject metadata yields excellent results. The use of a single, uniform line of HTML code makes it extremely easy for web developers to incorporate an image gallery into their documents. The current application represents an example of a web “mashup,” a hybrid application that combines data from more than one source¹³.

This system updates the displayed images automatically without intervention by the owner of the source web documents. The GoldMiner server periodically crawls the target web pages to check for updated content and also recompiles the code that displays the inlined images. Thus, as the image library is updated, the most recent relevant images are selected. Precompilation also allows the server to operate with maximal efficiency. For example, more than 15,000 CHORUS documents are viewed daily. Precaching the search results reduces overhead on GoldMiner’s database system; it is far more efficient to retrieve a precompiled HTML string than to execute a potentially complex database query each time a set of inlined images is requested.

GoldMiner retrieved images based on the appearance of the query words or related concepts within the associated figure caption. Preliminary analysis of GoldMiner’s search results has shown very high precision: of the images retrieved, 88% included the words or concepts queried¹⁴. Using strict criteria, this study found the retrieved images to be highly relevant to their target documents.

GoldMiner also can retrieve a set of images in real-time using a user-specified query string. For example, the specification:

<SCRIPT TYPE="text/javascript"

SRC="goldminer.arrs.org/inline.php?query=kidney+stones">

</SCRIPT>

generates a GoldMiner search for images that match the keywords and/or concept of “kidney stones.” This approach offers additional flexibility and allows developers to integrate ad-hoc search queries into a web page.

The system can integrate relevant images on demand using Asynchronous Javascript and XML (AJAX) technologies. AJAX is a group of interrelated techniques used to create interactive web applications¹⁵. AJAX typically combines: (1) XHTML or HTML and Cascading Style Sheets to define how information is to be displayed; (2) a client-side scripting language, such as JavaScript or JScript, to dynamically present and modify the web page’s Document Object Model; and (3) a mechanism, typically the XMLHttpRequest object, to exchange data asynchronously with the web server. The goal of AJAX technologies is to the client (browser) software to interact “behind the scenes” with the web server to make the web application more responsive and interactive. For example, AJAX has been used to develop PubMed Interact, which allows users to refine search parameters and interact with the search results to retrieve and display relevant information¹⁶.

An overarching theme of Web 2.0 is to facilitate collaboration through web technology¹⁷. Successful approaches “afford users the added advantage of reducing the technical skill required to use these features by allowing users to focus on the information and collaborative tasks themselves.”¹ A goal is to make the technology “transparent,” so that the user or web author can concentrate on the subject matter, rather than on the technological platform.

Several “Web 2.0” applications in radiology seek to provide interactive, collaborative web-based resources. A wiki is a website that allows visitors to add, remove, and edit content; the most famous wiki is the online encyclopedia Wikipedia (http://wikipedia.org). RadiologyWiki (www.radiologywiki.org)¹⁸, RadsWiki (www.radswiki.net), and Radiopaedia (www.radiopaedia.org) are example of multi-author information resources in radiology. ClubPACS (http://clubpacs.com)¹⁹ and Women’s Imaging Online (http://womensimagingonline.arrs.org) provide web-based communities of practice for PACS administrators and women’s imaging professionals, respectively.

The use of semantic information, such as that provided by the UMLS Metathesaurus, can improve navigation within a web-based network of images by relating medical concepts^20,21. The UMLS Metathesaurus is critical to GoldMiner’s ability to retrieve images not only by matching strings of text but also to understanding the underlying medical concepts. The MeSH terms form one component of the UMLS Metathesaurus. The RadLex vocabulary (http://radlex.org)²² promises to be an important addition to promote the uniform indexing of radiology content. The American Roentgen Ray Society is actively continuing the development of ARRS GoldMiner™ and provides this service freely on the Internet. Policies and instructions for those who wish to incorporate the ARRS GoldMiner Image Gallery into their web pages are available at the ARRS GoldMiner web site.