Preliminary comparison of the Essie and PubMed search engines for answering clinical questions using MD on Tap, a PDA-based program for accessing biomedical literature

Abstract

MD on Tap, a PDA application that searches and retrieves biomedical literature, is specifically designed for use by mobile healthcare professionals. With the goal of improving the usability of the application, a preliminary comparison was made of two search engines (PubMed and Essie) to determine which provided most efficient path to the desired clinically-relevant information.

The MD on Tap (MEDLINE^® Database) system has been developed by the National Library of Medicine to facilitate real time searches of and retrieval from biomedical literature via a wireless PDA^1,2. Currently, it uses the PubMed search engine with the MEDLINE literature database. The client interface was designed specifically to facilitate information retrieval for clinical questions using a variety of settings, limits, and display options to help clinicians efficiently locate information at the point of care.

The PubMed search engine is widely used and effective for biomedical research. However, for the specific uses and setting for which MD on Tap is designed, i.e. a small device in the clinical environment, we wish to compare alternatives to identify the search engine that most efficiently identifies information of interest for the clinical user. To this end, we conducted a preliminary comparison of the PubMed and Essie³ search engines. Essie, currently under development by the National Library of Medicine, is an experimental probabilistic search engine that ranks results by relevance.

The initial comparison was based on 87 de-identified user sessions culled from the PubMed on Tap server records. By noting the search terms entered and the specific citations the user selected to view and save for 11 of those sessions, we inferred the clinical question for which an answer was being sought. Using the same search terms with the Essie search engine, we compared the position of the selected citations in the Essie return order with the position in the PubMed return order. We also noted additional useful citations that appeared near the top of the Essie returns but not viewed by the original MD on Tap user. As a result of the need to infer the clinical question, a direct comparison of the information retrieved by the two search engines was difficult to evaluate. There was no obvious advantage to either search engine in the resulting analysis.

We then used clinical questions from actual hospital rounds, obtained during a related one-week usability study that tested the usefulness of MD on Tap for answering clinical questions in a hospital setting. The clinicians who asked the questions also gave feedback on the information retrieved, thus enabling us to judge the value of the information retrieved by either the PubMed or Essie search engines in a subsequent in-depth comparison of 5 clinical questions. As in the first study, the evaluation considered the position of the clinically useful articles within the returned results, but also considered the value of the retrieved information. For 2 of the 5 questions, SEER and PubMed performed similarly. For 1 question, SEER performed better than PubMed, providing and ranking higher sources of more relevant information. For 2 of the 5 questions, SEER performed much better than PubMed, either providing answers where PubMed did not or supplying the information more efficiently. Thus, we found that, for those clinical questions, the Essie search engine identified the desired information more readily than the PubMed search engine. We conclude that a more thorough usability study is warranted, with the goal of identifying the search engine most helpful for clinicians in efficiently accessing biomedical literature at the point of care. In addition, our results suggest that the Essie search engine, currently under development by the National Library of Medicine, may improve the effectiveness and ease-of-use of biomedical literature searches performed using PubMed on Tap.