Effects of Porting Essie Tokenization and Normalization to Solr

Abstract

Search for information is now an integral part of healthcare. Searches are enabled by search engines whose objective is to efficiently retrieve the relevant information for the user query. When it comes to retrieving biomedical text and literature, Essie search engine developed at the National Library of Medicine (NLM) performs exceptionally well. However, Essie is a software system developed for NLM that has ceased development and support. On the other hand, Solr is a popular opensource enterprise search engine used by many of the world’s largest internet sites, offering continuous developments and improvements along with the state-of-the-art features. In this paper, we present our approach to porting the key features of Essie and developing custom components to be used in Solr. We demonstrate the effectiveness of the added components on three benchmark biomedical datasets. The custom components may aid the community in improving search methods for biomedical text retrieval.

Introduction

A search engine is a type of information retrieval system that was developed to assist users in finding information relevant to the users’ queries. It has the ability to quickly find the precise information in a cost-effective manner. In the context of biomedical information search, the search engines play a crucial role in helping users to find relevant and reliable information related to biomedical topics such as diseases, treatments, drugs, and medical procedures. The search engines also helped in COVID-19 pandemic, where researchers utilized the appropriate search engines to retrieve the relevant literature to speed up the COVID-19 research¹.

Essie is a search engine developed specifically for biomedical information retrieval for ClinicalTrials.gov at the National Library of Medicine². Later, Essie started supporting other NLM services, including Open-i®³. For the searches over biomedical and clinical text, Essie comes with two major advantages over the generic search engines, such as Apache Solr, a community supported opensource search platform built on Apache Lucene^TM and managed by the Apache Software Foundation⁴. First, Essie is a phrase-based search engine that indexes documents using the UMLS concepts and also maps the queries to the concepts. Second, it has an effective tokenization strategy especially designed for the biomedical terms. The unique probabilistic scoring mechanism gives an additional boost to performance of Essie. Essie, however, is a software developed for NLM. Moreover, it is no longer developed and supported. On the other hand, Apache Solr is a popular and widely used search engine in the opensource community. It uses Lucene as its core and provides full text search capabilities including phrase, wildcards, joins, grouping, nested search, and other features. The major advantage of Solr is that it is a community project with great support, and it is continuously developed and improved. Solr can be configured and deployed as an enterprise software in a highly scalable and distributed environment. This enables high availability and fault tolerance. Solr also allows for customization of the search engine. Customization can be performed at the configuration level and also by creating custom components, which can be used as custom plugins.

Tokenization plays a critical role in improving the retrieval performance and accuracy of search engines⁵. Formally, tokenization refers to the breaking of raw text into small chunks or simple forms. Tokenization can be as simple as splitting the strings of text into words on white space or getting sequences of characters as various lengths tokens (n-grams). Alternatively, tokenization may be as complex as is needed for correct identification of spelling variants. The use of tokenization, stemming, and stop word removal, which is known to improve the open-domain retrieval results, does not improve the retrieval performance in biomedical text⁵. The specialized language, the use of biomedical symbols, inconsistent spelling, abbreviations, synonymy, and use of inconsistent lexical variants have amplified the challenges of retrieving biomedical data^5,6. Since Essie offers the specially designed tokenization strategy for the biomedical text, in this work, we focused on porting the Essie Tokenization and Normalization strategy to Solr and using the tokenizer as a custom component in Solr.

To analyze the effect of Essie tokenizer on retrieval of biomedical documents using Solr, we performed extensive experiments on the test collections created for the evaluations of biomedical literature retrieval within the Text Retrieval Conference (TREC): TREC 2004 and 2006 genomics tracks^7,8, and TREC-COVID 2019¹. We evaluated and compared the retrieval performance of Essie, Solr with standard configuration and Solr with Essie tokenizer. Our experiments show that Solr enhanced with the custom Essie tokenization and normalization performs better across all collections and evaluation metrics.

Background and Significance

The National Library of Medicine has been using Essie for image and document retrieval in Open-i for the past 15 years. Open-i system is a multimodal biomedical search engine used to search and retrieve open-source articles and images. Search is performed using text and image queries. Currently, Essie is serving 4.2 million images and 1.4 million articles from 5 different sources including PMC (Pubmed Central), Chest X-ray collection from the Indiana University hospital network, The Orthopedic Surgical Anatomy Teaching Collection at USC Digital Library, Images from the History of Medicine Division of NLM, and MedPix, a collection of radiology teaching cases. Open-i site serves about 57,000 users a month. Essie indexes the article text and image features to retrieve the relevant text and images. As support for Essie is no longer available, we need to find an alternative search engine, and Solr appears to be the best opensource alternative, as it is supported and developed by the community. The upcoming Open-i update will have roughly 5 million articles and 20 million images. With a much bigger collection, it needs to maintain or improve over the current high-quality performance in terms of both the quality of the results and real-time response to the users’ searches. While Solr and Lucene have shown state-of-the-art performance in biomedical retrieval, it was often outperformed by Essie⁹. An ablation study of Essie retrieval results for TREC 2003 has shown that the most significant improvement in the results was due to Essie tokenization and normalization¹⁰. We, therefore, decided to invest an effort in porting the Essie Tokenizer to Solr, hoping it will improve the retrieval results in comparison to using the Standard Solr tokenizer. Our experimental results confirm that the specialized tokenizer improves the biomedical literature retrieval results, which motivated us to package and make the specialized Solr plugin publicly available at https://github.com/soumyagayen/solressietokenizer. Additionally, our ongoing research on consumer health question answering currently uses Solr as a starting point for a traditional information-retrieval based question-answering pipeline. We will use the ported Essie Tokenizer in this project too.

Related Work

The wide variety of biomedical text in the literature demands an effective tokenization and normalization strategy to retrieve the relevant biomedical information against user queries^11,12. In the Genomics Track¹³ of TREC 2003, Tomlinson¹⁴ noted that combining alphabetic and numeric characters into a single token performs better than separating them. To accommodate hyphenation in gene names, both Pirkola and Leppanen¹⁵ and Crangle et al.¹⁶ split alphabetic and numeric strings in their submissions toTREC 2003 and 2004, respectively. In their TREC 2006 system description, Urbain et al.¹⁷ reported normalizing the gene and protein terms with suitable variants for document and passage retrieval tasks. To analyze the effect of various tokenization and normalization strategies, Jiang et al.⁵ carried out a detailed evaluation on TREC biomedical text collections performing ad hoc document retrieval tasks. Specifically, they defined and investigated the role of the three break point normalization methods and a Greek alphabet normalization method for document retrieval task. Trieschnigg et al.⁶ analyzed the effect of various tokenization schemes for document retrieval task in the biomedical and newswire domains and confirmed that biomedical retrieval is more sensitive to subtle shifts in the tokenization methods. Claveau¹⁸ introduced unsupervised and semi-supervised methods for decomposing a term into its morphological components (morphs) and assigning semantic labels to each of those morphs. These experiments on biomedical document retrieval show significant performance improvement over the counterpart baseline systems on multiple retrieval metrics.

Recent work in the field of information retrieval has focused on neural word embedding approaches, which have proven effective in improving ad hoc retrieval performance, thanks to their ability to capture the semantic meanings of vocabulary terms. In the open domain, Roy et al.¹⁹ studied the effect of term normalization on ad hoc retrieval performance with word2vec²⁰ and fastText²¹ embeddings. Inspired by the success of sub-word²² based tokenization in neural machine translation task, recently, Hang et al.²³ compared the results of traditional token-level representation, Sentence Piece subword²⁴, and character-level representations in traditional IR system (BM25) for cross-lingual information retrieval on a large-scale dataset²⁵ derived from Wikipedia. In their study, Sentence Piece tokenization was found to be less effective compared to the traditional token-level representation using the BM25 scoring approach. Odunayo et al.²⁶ explored the role of three tokenization mechanisms: whitespace, a language-specific Lucene analyzer, and the multilingual BERT (mBERT) tokenizer on a multilingual retrieval benchmark, Mr. TyDi²⁷ on 11 typologically diverse languages. Their results demonstrate that the mBERT tokenization outperformed the whitespace tokenization and even showed better results compared to the custom Lucene analyzer for some languages. This study also suggests combining the mBERT with Lucene analyzers to provide additional performance gains, Solr has been used as an off-the-shelf search engine for indexing and retrieving biomedical documents^28,29,30. Essie tokenization and normalization proved to be more effective^2,31 than the other search engine strategies for tokenizing the biomedical text. To the best of our knowledge, no other efforts have been made to overcome the Solr tokenization limitations with respect to processing of biomedical documents and queries. This work bridges the gap between Essie and Solr search engines for processing biomedical information.

Methods

In this section we first describe the specifics of the latest Essie tokenization and normalization. We then describe how this tokenizer was introduced as a module in Solr architecture. We then describe the evaluation of the Solr-Essie tokenizer on three medical information retrieval datasets.

A. Essie tokenizer

Given an input text the Essie tokenizer performs the following tokenization and normalization steps on input text.

1. Case conversion:

   Flattening case to lowercase and flattening white space to a single blank space.

2. Converting unicode characters to ascii:

   Essie uses a lookup table to convert unicode characters to ascii characters. The steps taken during conversion are expanding ligatures, replacing variants of special characters (quotes, apostrophes, hyphens, etc.) with ascii form, stripping accent marks, dropping various special characters such as trademarks. In some cases, a single unicode character is mapped to multiple ascii characters.

3. Tokenization

   Essie treats each token as a run of letters, numbers or a run of the same punctuation mark. For example, “...”

   is 1 token. But “(QED.)” is 4 tokens “(“,”QED”,”.”,”)”.

4. Rule based transformations:

   • Number variants: Dropping trailing zeros after decimal, trailing decimals, embedded commas.

   • Possessives: Dropping all the trailing “ ‘s ” and all embedded “ ‘ “. For example, “O’Malley’s ” becomes “omalley”

   • Hyphens: Remove hyphens if possible, but making sure negative numbers are not converted to positive.

   • Punctuation: runs of identical punctuation are shortened to max length of 4.

5. Normalization using the UMLS SPECIALIST Lexicon³²:

   The tokens are mapped to a preferred form of inflectional variants limited to plurals; other variants (e.g., derivational) are not used, as in the preliminary experiments this more aggressive normalization did not improve the results. The spelling variants and compound words are used.

Figure 1 shows the examples of character-based normalizations performed by Essie. Figure 2 shows the term-based normalizations, which also includes the character-based normalizations.

Figure 1.

Examples of Essie Character Normalization.

Figure 2.

Examples of Essie Term normalization.

B. Solr Tokenization

Solr provides various options for inbuilt tokenizers and filters that can be configured. Solr also provides an option for creating custom filters and configuring them in an analyzer. A Solr analyzer examines the text of fields and generates a token stream. An analyzer can also be decomposed into a series of a tokenizer and a chain of many filters connected into a pipeline in the order they should run.

Figure 3 shows an example configuration of Solr analyzer that consists of a chain of filters in the order they are applied, starting with StandardTokenizer, StandardFilter, LowerCase Filter, and StopWord Filter.

Figure 3.

An example of a configuration of the text field analyzer in Solr.

Figure 3. An example of a configuration of the text field analyzer in Solr.

We have extracted the tokenization and normalization logic from Essie code and compiled it as an independent Jar file that can be placed in the Solr setup. To be used, it needs to be declared as a custom library in the solrconfig.xml as follows: <lib path="../custom-lib/customfilter.jar"/>. In the Solr schema.xml where the configuration is defined, we define our custom field (essie_custom_field) and custom analyzer. The first step in creating an analyzer is to define a Solr tokenizer. Solr provides various options, e.g., a StandardTokenizer, which splits the text into tokens based on white space, and many others. For our purpose, we use SolrKeywordTokenizer as it treats the entire text field as a single token and passes it on to our custom filter. This is the best Solr tokenizer for our purposes because we do not want Solr to tokenize the text. Tokenization will be performed by our custom filter (Essie Tokenizer and Normalizer). Next in the chain, we fit our custom filter which will tokenize and normalize the text and pass it to the subsequent filters in the chain. Figure 4 shows side by side the block diagram of Solr analyzer configuration with custom Essie tokenizer and the corresponding xml configuration.

Figure 4.

Solr configuration with Essie tokenizer. The left side shows a block diagram of Solr custom text field. The right side is the configuration in schema.xml.

Figure 4. Solr configuration with Essie tokenizer. The left side shows a block diagram of Solr custom text field. The right side is the configuration in schema.xml.

Evaluation

To evaluate the contribution of the Essie tokenizer ported to Solr, we compare the retrieval performance of Essie, Solr with the default tokenizer, and Solr with Essie tokenizer on three different test collections: TREC 2004 genomics track, TREC 2006 genomics track and TREC-COVID, which uses the COVID-19 Open Research Dataset (CORD-19)³³.

A. Collections

We have used the following benchmarked datasets to evaluate the performance of Essie tokenizer ported to Solr.

1. TREC 2004 Genomics Track⁷: It consists of 4,591,008 records that are a subset of the MEDLINE database (1993-2003). The ad-hoc retrieval task had 50 topics, which were derived from interviews eliciting information needs of real biologists.

2. TREC 2006 Genomics Track¹⁷: It consists of 162,259 documents that were curated from full text html documents of 49 journals. For the retrieval tasks, there are 28 topics that are posted as questions.

3. TREC-COVID: The TREC-COVID challenge was conducted over 5 rounds that used the CORD-19 collection. For our evaluation purpose we have used the cumulative data from all 5 rounds. The data were compiled using around 140k publications about COVID-19 and related historical coronavirus like SARS and MERS. For our evaluation we have used the dataset published by BEIR (Benchmarking IR)³⁴. The BEIR CORD-19 collection has 171,332 documents and 50 topics.

For each TREC collection the users’ information needs are called topics. Each topic traditionally consist of three parts: a short, 2-3 words, search that a user would submit to a search engine; a longer description, sometimes called question; and a long narrative part that explains the topic to a human, e.g., a reference librarian that might help with the search. In our evaluations we used the shortest field, e.g., for TREC-COVID we used the “query” field. Figure 5 shows an example TREC-COVID topic.

Figure 5.

An Example TREC-COVID topic

B. Evaluation Metrics

We used the official evaluation metrics as required by the particular challenge. For TREC 2004 and TREC 2006 we used Mean Average Precision (MAP), and for TREC-COVID we used Precision at 5 (P@5) and Normalized Discounted Cumulative Gain at 10 (NDCG@10).

Precision is defined as the ratio of relevant documents to the number of retrieved documents. When we consider a particular number of retrieved documents or the maximum cut-off rank, then we describe it as precision at a particular rank. For example, P@5 is the ratio of the relevant documents among the top five retrieved documents.

Mean Average Precision (MAP) is the arithmetic mean of the average precision values for an information retrieval system over all N topics in the collection. Average Precision is the average of the precision values obtained after each relevant document is retrieved³⁵.

NDCG, normalized discounted cumulative gain, is a metric for an evaluation which considers both relevance and rank of retrieved documents. Gain is the relevance score of each document. Cumulative Gain is the total of all the gains for given K retrieved documents. Discounted Cumulative Gain (DSG) is where higher weight is assigned to the gain of the top ranked document and lower weight is given to the highly relevant documents that are ranked lower. The Normalized Discounted Cumulative Gain (NDCG) is normalizing the score by dividing it by the ideal DSG score. Ideal DSG score is calculated from the ground truth³⁶.

We used the trec_eval package to compute the measures for the TREC 2004 and TREC-COVID evaluations³⁷. We used the evaluation package developed for the TREC 2006 evaluation³⁸.

Validation and Results

All collections provide an official file that contains relevance judgments for all judged documents for the given task. We indexed the above document collections using Essie, Solr with default settings, and Solr with Essie tokenizer. We retrieved documents using the short searches provided in TREC topics. For the purpose of comparing the three different search engines, our focus was on creating identical conditions for all three search engines and generating results for an unbiased comparison, where only the tokenizers were different. We did not apply any advanced techniques for improving the query, nor did we use any query reduction techniques or other techniques known to improve search results. MAP, precision at 5 (P@5) and precision at 100 (P@100) scores for collections TREC 2004 and TREC 2006 comparing Essie, Solr Standard and Solr with Essie Tokenizer are presented in Table 1. Table 1 also presents a comparison of Essie, Solr Standard and Solr with Essie Tokenizer for TREC-COVID collection measuring NDCG@10, P@5 and P@100.

Table 1.

Evaluation Results for TREC 2004, TREC 2006 AND TREC-COVID.

	TREC 2004			TREC 2006			TREC-COVID
	MAP	P@5	P@100	MAP	P@5	P@100	NDGC@10	P@5	P@100
Essie	0.149	0.392	0.180	0.165	0.142	0.083	0.175	0.195	0.207
Solr (Standard)	0.219	0.470	0.204	0.388	0.464	0.205	0.465	0.528	0.390
Solr (Essie Tokenizer)	0.278	0.529	0.251	0.435	0.528	0.232	0.554	0.612	0.416

Discussion

Essie being a search engine developed at the National Library of Medicine for the internal project has come to an end of life. All the development and support work on Essie has ceased. In most projects that used Essie, it has been replaced with Solr or ElasticSearch. The search engines like Solr and ElasticSearch with default settings produce average results when used to retrieve biomedical literature. Essie has outperformed these results in TREC evaluations and successfully served multiple NLM projects. In an effort to preserve our information retrieval performance on biomedical data, we ported the tokenization and normalization techniques used by Essie to Solr. Our results show that when Solr is configured with Essie Tokenizer it gives better performance than using the standard Solr tokenizer.

The precision at five retrieved documents is an important metric for our question answering research, as this is the number of potential answers the users see first in a traditional information retrieval-based system. Precision at 100 is important for Open-i results that present 100 images in each page of the retrieval results. As Table 1 shows, the users will see more relevant images in the first page of Open-i results due to Essie tokenization in Solr, and the question answering systems will also work with a set of candidate documents that are more relevant to the question.

Using Essie Tokenizer in Solr opens opportunities for a search engine that combines the good performance of a domain-specific tokenizer on biomedical text with the state-of-the-art features of Solr. Solr provides extensive support to the community. It is known to be reliable, scalable, fault tolerant, and ready to operate on distributed environment. Solr provides automated failover and recovery, options to be used in docker containers and support for the systems for automating deployment, scaling, and management of containerized applications, e.g., Kubernetes. These features make it a powerful application and explain its wide adoption in the field of biomedical information retrieval. Easy customization and setup of Solr along with Essie tokenizer makes it useful for the research community by providing a strong domain-specific baseline.

Our work has some limitations. Due to our primary interest in literature retrieval, we have not tested whether the Solr-Essie tokenizer will improve processing and retrieval of clinical text. We tested the tokenizer only with the BM25 retrieval model and have not tested other scoring functions provided by Solr. Finally, while tokenization and normalization have been shown the strongest contributors to Essie retrieval, additional smaller benefits come from handling polysemy using the UMLS synonymy and Essie document scoring function.

In the future we intend to extend our work to porting Essie synonymy to Solr. Synonymy is known to be important for biomedical search engines as it bridges the gap between medical literature and non-medical end users. For example, a user searching for “heart attack” will not find the majority of related documents without query expansion using synonymy. Synonymy also compensates for some of the derivational cases that are not covered by the conservative normalization. Our initial work on implementing UMLS synonymy in Solr did not show significant improvement in the evaluation results with the above collections. This might be explained by the standard medical language that was universally used both in the collection queries and in the documents, however a more thorough investigation with collections that are known to benefit from the UMLS synonymy is needed. The Essie scoring function is another potential venue to explore and add to the set of scoring functions already available in Solr.

While Essie synonymy and the scoring may improve over those already available in Solr, the researchers can already benefit from using Essie tokenization as Solr Filter, which is available along with the code on GitHub at: https://github.com/soumyagayen/solressietokenizer. The users can download the Jar and use the Essie normalization and tokenization in their custom Solr Anlyzer. Guidelines are available on how to use it in their Solr implementation.

Conclusion

We present the Essie Tokenizer for Solr that can be configured and used as a filter in the Solr analyzer by the biomedical community. Our experiments on TREC 2004, TREC 2006 and TREC-COVID datasets clearly show that there is an improvement in performance for biomedical literature retrieval when Solr is configured with Essie Tokenizer. The results also show that Solr with Essie tokenizer works better than Standalone Essie. We believe the customization which Solr provides at the index and query time, paired with Essie Tokenizer filter will significantly improve the performance of information retrieval on biomedical data.