Abstract
Medical resident physicians used MD on Tap in real time to search for MEDLINE citations relevant to clinical questions using three search engines: Essie, Entrez and Google™, in order of performance.
MD on Tap (MDoT), developed by an R&D group at the National Library of Medicine (NLM), supports access to MEDLINE citations via special client software for Palm and Pocket PC PDAs with wireless access to the Internet1. Users are allowed a choice of three search engines for MEDLINE queries: Entrez2, the Boolean search engine used by the NLM PubMed® system, Essie3, the probabilistic search engine developed for the NLM clinicaltrials.gov web site, and Google. Entrez ranks results by publication date; Essie ranks results by relevance to search terms.
Medical residents participated in a structured evaluation of MDoT in conjunction with a medical informatics elective. Each physician accompanied, as a knowledgeable observer, medical teams on rounds in a community teaching hospital, using MDoT to search for MEDLINE citations judged relevant in answering any clinical question that arose. Residents submitted daily summaries to NLM that included, among other items, a list of the relevant citations.
All three search engines were evaluated from February 14 through March 8, 2006. One or more relevant citations were found for 55 of the 79 clinical questions that arose during rounds in an intensive care unit. The distribution of found citations among search engines is shown in Table 1.
Table 1.
Number of questions for which relevant articles were found and not found using three search engines. (Q = questions, F = found, NF = not found)
| Entrez | Essie | |||||
|---|---|---|---|---|---|---|
| F | NF | F | NF | F | NF | |
| # of Q | 32 | 12 | 9 | 2 | 14 | 10 |
| % of Q | 73% | 27% | 82% | 18% | 58% | 42% |
Thirty citations were selected for detailed comparison, ten from each list of relevant citations for the questions using each of the search engines. For Entrez and Google, only the first relevant citation was selected for any one question for which answers were found. For Essie, two relevant citations for one question were used, for a total of ten citations. Using the same search terms and options as in the original query, the other two search engines were used to seek the same citation. Target citations not in the first 250 citations returned were counted as not found. Tables 2 and 3 summarize results.
Table 2.
Average position of thirty relevant citations in the results of three search engines.
| Avg. orig. pos. of 10 in | Number of orig. 10 found in | Avg. pos. of those found in | Number of orig. 10 found in | Avg. pos of those found in |
|---|---|---|---|---|
| Entrez | Essie | |||
| 7.4 | 8 | 29.4 | 3 | 2.7 |
| Essie | Entrez | |||
| 6.8 | 5 | 14.6 | 5 | 55.2 |
| Entrez | Essie | |||
| 15.1 | 6 | 69.3 | 8 | 19.9 |
Table 3.
Average position of twenty relevant citations originally found with the other two search engines.
| Search Engine | Number found (of 20) | Percent found (of 20) | Avg. pos. of those found |
|---|---|---|---|
| Entrez | 11 | 55% | 62.9 |
| Essie | 16 | 80% | 24.7 |
| 8 | 40% | 10.1 |
Essie found relevant citations to the highest percentage of questions, both for original searches and with searches originally used for other engines, and ranked the target citations higher than Entrez. Google found far fewer relevant citations than the other search engines, but overall ranked the found target citations higher than other engines.
References
- 1.Hauser SE, Demner-Fushman D, Ford G, Thoma GR. PubMed on Tap: discovering design principles for online information delivery to handheld computers. Medinfo. 2004;11(Pt 2):1430–3. [PubMed] [Google Scholar]
- 2.U.S. National Institutes of Health, National Library of Medicine. NCBI Databases. [Online] http://www.ncbi.nlm.nih.gov/Database/
- 3.McCray AT, Ide NC. Design and implementation of a national clinical trials registry. J Am Med Inform Assoc. 2000 May–Jun;7(3):313–23. doi: 10.1136/jamia.2000.0070313. [DOI] [PMC free article] [PubMed] [Google Scholar]