Evidence-based Medicine Search: a customizable federated search engine

Abstract

Purpose: This paper reports on the development of a tool by the Arizona Health Sciences Library (AHSL) for searching clinical evidence that can be customized for different user groups.

Brief Description: The AHSL provides services to the University of Arizona's (UA's) health sciences programs and to the University Medical Center. Librarians at AHSL collaborated with UA College of Medicine faculty to create an innovative search engine, Evidence-based Medicine (EBM) Search, that provides users with a simple search interface to EBM resources and presents results organized according to an evidence pyramid. EBM Search was developed with a web-based configuration component that allows the tool to be customized for different specialties.

Outcomes/Conclusion: Informal and anecdotal feedback from physicians indicates that EBM Search is a useful tool with potential in teaching evidence-based decision making. While formal evaluation is still being planned, a tool such as EBM Search, which can be configured for specific user populations, may help lower barriers to information resources in an academic health sciences center.

Highlights

• A tool for federating search of clinical resources was developed through a quick and iterative process involving librarians and physicians. The tool embeds the search knowledge of reference librarians into a simple search interface that appeals to busy clinicians.

• The tool is highly configurable and can be customized for different clinical and education environments.

Implications

• Useful tools for enhancing user access to library resources can be developed in a quick manner if user needs and workflow are well understood. These tools can be implemented by staff in the library with appropriate skills.

• Iterative design processes can speed the design of innovative library services.

• Expert search skills can be an important part of designing technologies that provide more accurate search results to clinicians in a convenient manner. The ability of searchers to understand the changing needs of their users, as well as the changing nature of the information environment, can provide new opportunities to apply their knowledge.

INTRODUCTION

Background

Libraries and other information organizations have been grappling with the challenge of developing more efficient tools to allow users to discover information relevant to their needs, with many efforts focused on creating tools that facilitate access to a broad range of content to broad audiences. Web search engines from Alta Vista to Google, sometimes referred to as horizontal search engines, exemplify these approaches. Federated search engines implemented by many libraries provide a similar approach to enhancing access to the content of licensed and free article databases and other resources [1–4]. Recently, however, interest has increased in developing vertical search engines that provide access to targeted content to better meet the needs of particular communities or subject domains [5]. On the open web, Google Custom Search is one example of this approach [6].

At the same time, there has been significant interest in clinical medicine and other health professions in better integrating evidence from the medical literature and other sources into clinical practice. The complexities of knowledge translation [7] and the time constraints in clinical settings [8], however, create barriers to adopting evidence-based practice (EBP). Studies have shown that the user experience provided by commercial clinical information platforms can have a positive impact on both clinicians' perception of a search tool and their ability to find answers to clinical questions [9].

In addition to challenges specific to EBP, the convenience of general search engines has created a standard for ease of search, demonstrated in studies of clinician search behavior [10]. In response, many libraries have implemented federated search tools with single, integrated results sets [1–4]. Several projects have also created sophisticated search engines that federate clinical evidence in a single result set [11–14]. Additionally, other health sciences libraries have undertaken projects to create specialized search interfaces for specific types of resources [15, 16].

Librarian and clinician collaboration

The Arizona Health Sciences Library (AHSL) felt these approaches could be improved by organizing the user experience around one of the visual constructs used to teach evidence-based medicine (EBM) at the University of Arizona [17]. Further, clinicians and librarians at the institution felt that the commercial clinical search products available to them were inherently limited by not integrating content from multiple publishers equally. This was especially true of electronic book resources.

To develop an approach to the problem of integrating information searching and library expertise efficiently into clinical practice, the team chose local development of a tool because no commercial product was available to the institution that provided federated searching of licensed and freely available clinical evidence, provided supported customized display of results, and supported the embedding of power-searching strategies in a manner that was transparent to users. In light of this context, the AHSL, as a library dedicated to improving the integration of clinical evidence into clinical practice, sought to explore the following question: Is it possible to create a search tool that is both simple and convenient to use and sufficiently powerful to provide effective access to clinical evidence?

As described elsewhere, it became apparent to the authors that synergies between clinical departments and the health sciences library clearly provided an opportunity for integrating evidence into clinical decision making [18]. Librarians had been involved in EBM efforts for several years in an instructional capacity, and while the team felt that teaching clinicians to effectively search and evaluate information had merits, instruction also seemed to be a limited approach. Given the time constraints inherent in the clinical workplace, it was also seen as unlikely that physicians would apply this instruction in practice. Likewise, models such as the informationist that have been implemented in other settings were seen as unrealistic in the University of Arizona environment, both for financial reasons and the unlikelihood that local clinicians would relinquish control of query formulation at the point of clinical decision making.

The goal of the project was to create a scalable model for integrating librarian expertise into clinical decision making, without requiring their availability in real-time. To meet this goal, the team decided to develop a search tool that would allow physicians to control query formulation and result evaluation, but that would embed the expertise of librarians in resource selection and search construction. It would also embed clinical knowledge into the interface design to assist users in making relevance judgments.

Evidence-based medicine search development

The search tool, EBM Search, was programmed using Cold Fusion with a Microsoft SQL server database backend between February and August 2006. A rapid development process was employed, which focused on prototype development followed by an iterative feedback and modification process. The first iteration of the end-user search tool, developed in two weeks, provided the initial proof-of-concept and incorporated several design considerations. First, the search interface needed to be simple. Second, for practical and philosophical reasons, the team decided that providing a results screen that integrated results from all resources included in EBM Search into one single set was undesirable, in part, because the amount of development that would be required to manage a single results set was seen as unrealistic. More importantly, both clinical and library partners felt that an integrated results set was not optimal for enabling users to quickly select a resource at the appropriate level of evidence for their needs. Accordingly, the interface was organized around the evidence pyramid [16], a well-understood concept among local clinicians. Third, the search tool needed to incorporate librarian-designed search hedges to improve relevance for the target group, thereby reducing the need for expert searching skills from physicians. Fourth, EBM Search would use each resource's native interfaces for vocabulary mapping and relevance ranking, as these features were better developed in the individual resources than could be developed in this project. Finally, where feasible and meaningful, the number of results from each provider was displayed to enhance usability.

The initial implementation was designed for the department of emergency medicine (EM) and allowed local emergency department (ED) physicians to execute searches on topics of their choice. Resources were selected through consultation with the EM residency director and interviews with EM residents. Based on this input, the resulting search provided access to pre-executed queries in a number of resources: PubMed Clinical Queries, the Cochrane Database of Systematic Reviews, the Database of Abstracts of Reviews of Effects (DARE), topic reviews from UpToDate, and textbooks from Stat!Ref. The links to PubMed Clinical Queries were limited in several ways, including a limit to core clinical and EM journals. In this case, core journals were determined by local clinicians to be those most useful for finding clinical answers.

Figure 1 illustrates the tool's processes. When a search is submitted, the search tool executes several tasks. First, the physicians' search is post-pended with EBM search filters that have been developed by a reference librarian and modified with input from an ED physician. Filters have been customized for each database to take advantage of the native functionality. Second, extensible markup language (XML) web services are used to obtain the number of results for display, where supported. Third, uniform resource locators (URLs) have been constructed to link users to pre-executed searches in the native interface. For example, a search for “myocardial infarction” would, among other things, link to a PubMed search for randomized control trials on myocardial infarction using the following strategy:

(“myocardial infarction”[mh] OR myocardial infarction[tw]) AND randomized controlled trial[pt] OR randomized[Tiab] AND controlled[Tiab] AND trial[Tiab]

Figure 1.

Evidence-based medicine (EBM) search tool processThe process involved in searching and configuring EBM Search to provide access to abstracting and indexing (A&I) services, textbooks, and other sources of clinical evidence.

Finally, the results were displayed using a ranking system based on an EBM pyramid (Figure 2). Query sources were grouped for display according to level of evidence, accompanied by an illustration of the level of evidence provided for each grouping. For example, the EBM Search tool for “emergency medicine” includes groupings for Systematic Reviews, PubMed Clinical Studies, PubMed (All), Evidence-Based Guidelines, and Textbooks. The Systematic Reviews grouping includes links to searches in the Cochrane Database of Systematized Reviews and DARE, as well as three different searches for systematic reviews in PubMed that apply progressive limits to narrow the search. This method provides an interface that utilizes a two-tiered relevance ranking system to allow physicians to quickly select a desired level of evidence. Sources are organized by relevance to clinical decision categories such as therapy, diagnosis, and treatment, then the relevance-ranking features of each source are used to sort at the article and chapter level.

Figure 2.

Search and results screenThe search results organize sources by relevance to clinical decision categories, and users are linked to pre-executed queries in each source.

INITIAL FEEDBACK AND CONTINUED DEVELOPMENT

The prototype was well received by the EM department and generated interest from other departments after presentation in a grand rounds session. Although formal evaluation was still needed, the system clearly had potential and merited further development to be configurable for specialties other than EM. The system should also be modifiable by librarians without the assistance of a programmer.

In subsequent phases of development, the user interface remained essentially unchanged. Minor formatting changes were made, and additional resources such as the National Guideline Clearinghouse were added. Some adjustments were also made to the search hedges, primarily to refine the limits applied to PubMed searches. For example, searches for PubMed Clinical Studies were disaggregated to provide separate access to articles on therapy, diagnosis, and prognosis, with each of these further limited to EM journals. Most development at this point focused on the backend tools that would allow librarians, or other managers, to create and configure search tools for specific audiences (Figure 3). The goal was to create a framework that would allow librarians or end users to develop specialized search tools with little understanding of the mechanics of the EBM Search. The management features supported several basic functions: creating a search tool, adding resource groupings, assigning a level of evidence to each resource grouping, adding resources to each grouping, modifying the labeling of each resource, modifying the search parameters of resources, adding resources to multiple groups, and adding multiple instances of a resource to a single group, each with different search parameters.

Figure 3.

Configuration toolThe configuration features allow librarians and other nontechnical staff to develop their own search tools.

Over the course of three months, the backend tools needed to manage EBM Search were developed and refined through a process in which initial prototypes were quickly developed and then modified in an iterative fashion, as reference librarians reviewed and critiqued initial functionality and suggested next steps. The result was a system that allowed nontechnical individuals to create customized federated search tools with only one point of intervention from technical staff: the addition and configuration of new searchable resources. This phase of development was completed in June 2006, although bug fixes and other minor enhancements have continued in the time since.

DISCUSSION AND FUTURE DIRECTIONS

The design of the EBM Search tool is important for several reasons. First, clinician-initiated queries are enriched with librarians' knowledge of resources and expert searching, and the tool has the potential to provide an added value for specific target groups by allowing convenient but powerful access to clinical evidence. Second, the configurability of the tool encourages collaboration between librarians and audiences to allow for assessing user need and creating a tool that meets these needs. Third, adapting the evidence ranking system from the Oxford Centre for Evidence-Based Medicine [17] potentially reduces the time needed to conduct critical appraisal. In addition, isolating Cochrane Reviews in the interface permits users to navigate to the highest-quality results first, if the source is appropriate for the clinical question. Finally, the evidence pyramid layout can function as a teaching aid, allowing users of the system who are unfamiliar with the principles of EBM to make connections between real search results and levels of evidence.

While the EBM Search Tool has demonstrated promise, a number of enhancements could improve its capabilities. First, the usability of the interface for evaluating information sources could be improved. In part, this could be achieved by fetching and displaying result set sizes for more resources. Displaying result set size is a useful heuristic device allowing the selection of a resource that appears to meet a desired level of specificity, while saving time by allowing users to skip empty sets. In other cases, displaying the number of results might be less useful as a point of comparison. For example, knowing the number of results from a textbook search does not allow one to compare it to the results from an article database. In these cases, alternative methods for communicating what a user can expect by following any given link need to be developed. Technical issues, however, limit the number of resources for which results set sizes can be easily retrieved and displayed. Currently, vendor support for web services or other XML gateways for search and retrieval is limited, and Z39.50-based solutions can be complex to implement. In the long run, this issue should be less significant as more vendors support XML-based services such as search/retrieve via URL (SRU) or the National Information Standards Organization Metasearch XML Gateway Standard (MXG) [19, 20].

It is also essential that the tool remain adaptable to evolving clinical workflows. One of the primary reasons for the initial positive response to the EBM Search tool is that its design takes constraints of the clinical workflow into account. More factors, however, might be taken into account to further improve design in future iterations. In addition to the interface enhancements discussed above, it is also important to consider ways in which the search tool can be embedded in other online environments and workflows. For example, mobile versions of the search tool would be a logical future consideration, as would embedding the search tool in portals, hospital intranets, or other information technology resources that clinicians use. Despite the possibility of further enhancements, it would be important to maintain the foundational aspects of the current design that have led to a service that, anecdotally, has provided clinicians with a useful way to quickly access evidence. Continuing to provide a simple search interface and results interface organized around known constructs for grading levels of evidence will be critical.