PrimeAnswers: A Practical Interface for Answering Primary Care Questions

Abstract

This paper describes an institutional approach taken to build a primary care reference portal. The objective for the site is to make access to and use of clinical reference faster and easier and to facilitate the use of evidence-based answers in daily practice. Reference objects were selected and metadata applied to a core set of sources. Metadata were used to search, sort, and filter results and to define deep-linked queries and structure the interface. User feedback resulted in an expansion in the scope of reference objects to meet the broad spectrum of information needs, including patient handouts and interactive risk management tools. Results of a user satisfaction survey suggest that a simple interface to customized content makes it faster and easier for primary care clinicians to find information during the clinic day and to improve care to their patients. The PrimeAnswers portal is a first step in creating a fast search of a customized set of reference objects to match a clinician's patient care questions in the clinic. The next step is developing methods to solve the problem of matching a clinician's question to a specific answer through precise retrieval from reference sources; however, lack of internal structure and Web service standards in most clinical reference sources is an unresolved problem.

Studies have repeatedly shown that clinicians have significant informational needs that are not met in their practices.^{1,2,3,4,5,6,7} Researchers estimate that although one clinical question arises per patient visit, as many as 70% of these questions go unanswered.^1,7,8 The findings are similar for clinicians at different levels of training and practice settings.⁵ Studies have shown that although many questions generated by clinicians can be answered using clinical reference sources,^{8,9,10,11,12,13,14,15} physicians will make use of readily available information only when minimal effort is required.^{5,6,7,8,9,10,16,17,18} The typical primary care visit lasts 10 to 15 minutes, and most clinicians find that it takes too much time and is difficult to articulate their questions in most information systems.^16,19,20 Studies have found that clinicians prefer to take less than two minutes to find the answer to a question.^21,22 Compounding the problem is the fact that multiple reference systems are required to answer the variety of questions that arise during the clinic day.^19,23 In summarizing a group of studies, Smith⁷ observed that the questions that arise in clinical care are often complex and multidimensional. While physicians are encouraged to identify, appraise, and apply the best evidence in making decisions about individual patients, this ideal is seldom realized.^{20,21,22,24,25,26,27}

Background

In 1999, the University of Washington (UW) Health Sciences Libraries (HSL) designed the HealthLinks Web site around a series of role-based reference tool kits. The most popular was the Care Provider Toolkit, which was populated with direct links to the most relevant reference sources for clinicians. The target audience of the tool kit was practitioners at UW Medicine, a large, dispersed system composed of two medical centers, a primary care network, and a medical school. The School of Medicine is a regional medical education program with over 50 affiliated primary care clinics, clerkships, and residency sites in five Pacific Northwest states. The tool kit was linked in MINDscape, the Web-based interface to inpatient data for the two medical centers and as a desktop icon on the computers in primary care network clinics using EpiCare®. In addition to the tool kit, the library collaborated with the MINDscape developers to create InfoLink buttons that preceded each medication, laboratory result, and condition on the problem list.²⁸ Data showed low use of the InfoLinks in MINDscape. As MINDscape did not offer physician order entry, the developers hypothesized that the precise links to drug and laboratory result information were not presented at the time of need. Data showed that the link to the Care Provider Toolkit was used much more frequently; however, use in the outpatient clinics was low.

In 2000, the HSL and primary care clinicians strategized on methods to increase use of clinical reference and proposed a customized portal called PrimeAnswers.^29,30 The content of this portal would focus on reducing the barriers to evidence-based answers and commonly used information objects for general internists and family physicians. The interface would address the constraints of short encounter times, broad range of questions, continuity of care, and the need to improve patient communication in outpatient clinics. The hypothesis was that a simple and customized portal would increase the use of reference lookup in the examination and attending rooms of the primary care clinics. Other institutions were developing institution-based clinical reference portals and search systems: Primary Care Office InSite (PCOI)^31,32 at Massachusetts General Hospital aggregated institutional guidelines and links to clinical reference in a primary care portal, TRIP^33,34 was an Internet search of filtered evidence documents for physicians in the National Health Service in the United Kingdom, Family Physicians' Inquiries Network^35,36,37 sought to create a database of synthesized evidence answers to questions that arose from real patient encounters by family physicians, and the Stanford Health Information Network for Education (SHINE)^38,39,40,41 at Stanford University Medical Center searched multiple clinical reference sources with integrated results. These projects are compared with PrimeAnswers in the Discussion section of this paper.

Needs Assessment

Three UW primary care clinics were identified as the pilot sites for the new PrimeAnswers portal: UW Medical Center General Internal Medicine Clinic (UWMC GIMC), Harborview Medical Center Family Medicine Clinic (HMC FMC), and the UW Physicians Network (UWPN), composed of 14 neighborhood outpatient clinics distributed across King County. Each clinic director selected a primary care physician to join the project development team. These physicians attended monthly meetings and served as the champions for the project in their clinics. Clinic characteristics are outlined in ▶.

Table 1.

Profile of Target Primary Care Clinics, 2000

Clinic	MDs	Location	EMR	High Speed Internet	Computer Availability
HMC FMC	9	County hospital	MINDscape	Yes	Each exam room
UWMC GIMC	25	Outpatient site	MINDscape	Yes	Central attending rooms
UWPN	60	14 county sites	EpiCare®	Yes	Each exam room

A three-part preimplementation analysis was conducted to assess information use in the targeted clinics: a user survey, direct observation, and interviews/focus groups of clinic physicians.⁴² The response rate to the user survey was 40% (38/94): 16% FMC, 26% GIMC, and 58% UWPN. Distribution of respondents by clinic was six of nine FMC, ten of 25 GIMC, and 22 of 60 UWPN. Demographic characteristics of respondents are outlined in ▶. Of the respondents, 14% thought that their favorite resources were available online, 31% used the library's existing clinical portal more than five times per week, and 30% had used MEDLINE for patient care questions in the clinic. The online reference sources that they used most were Micromedex® and the MD Consult® electronic textbooks, although this is an artifact of available systems at that time. Survey respondents felt the most significant limitations to using online clinical reference were amount of time needed to use (68%), difficulty in articulating a question to reference source (55%), and slow and cumbersome searching (53%). They felt that the best time to look for information was during a patient visit (51%) and immediately after the visit (39%). Survey respondents ranked categories of information resource need in this order: medications, patient education, treatment, common conditions, best evidence, uncommon aspects of common conditions, algorithms or guidelines, and diagnosis. Survey results confirmed that behavior and barriers to using online clinical reference of the targeted clinicians closely matched the conclusions of published studies discussed in the introduction of this article. The categories of questions identified in the survey results closely matched the conclusions of the Ely et al.⁴³ question analysis study.

Table 2.

Pre-Implementation Demographic Characteristics of Survey Respondents, 2000

Gender		Specialty			Years in Practice
M	F	Internal Medicine	Family Medicine	Pediatrics	1–5	6–10	11–20	>20
57% (21)	43% (16)	38% (14)	57% (21)	5% (2)	35% (13)	16% (6)	24% (9)	24% (9)

An evaluator trained in ethnographic techniques conducted an observational study of the information-seeking behavior of eight physicians during a half day at the three clinics. Clinicians were divided equally between family medicine and general internal medicine. Observations were made before and after each patient visit using a think-aloud protocol with stimulus questions. The evaluator distributed a survey and conducted a postobservation debriefing. The debriefing interview included questions about whether the day was typical, how many patients were seen, how many questions arose, what the questions were, if the clinicians searched for or found the answer to their questions, and if they intended to pursue the question later in the day or before the next visit. The clinicians observed did not think that access to a computer was a factor in use of online clinical reference. They did feel that it took too much time to find answers and that searching was slow and cumbersome. Observational data identified categories of recurring information questions during a clinic half day: drug information, patient education, immunizations, travel health, dermatological images, therapy, diagnostic rules for injuries, dietary counseling, and uncommon presentations of common conditions. The project staff interviewed three clinic directors and conducted focus groups at each clinic as well. Clinic directors wanted the system to save time and encouraged the inclusion of frequently used calculators, algorithms, tables, and patient handouts to encompass the full spectrum of information needs. During focus groups, physicians indicated that they preferred presentation of information in the context of a typical patient visit: short, quick answer; link to longer contextual summary; and full source or document. The Ely et al.⁹ study in 2005 described the same desire by physicians for quick access to information within a resource in the form of summary charts, tables, and answers.

Methods

Portal Design

The design objectives for the PrimeAnswers portal were to (1) create a filtered and customized set of content that would make best available evidence as accessible as commonly used textbooks; (2) design automated methods to search the most commonly used external clinical reference systems; and (3) integrate information objects frequently used during the clinic day (e.g., calculators, tables, patient handouts). Clinician advisors envisioned a clean, visual portal that relied on a simple search as the primary mode of navigation. They recommended that the portal itself should be only two levels deep with each page equivalent to one average screen in the clinics to avoid scrolling. They wanted searches and links to be labeled with common names and brief descriptions for lesser known reference sources. The desired content would be mined from a small but significant set of digital information objects held in both proprietary and nonproprietary systems. Metadata would be applied at a deep level to evidence summaries and guidelines with a bottom line clinical answer. The metadata would provide the means to run a search against portal objects and with one click expand the search to comprehensive clinical reference systems.

The PrimeAnswers interface (http://primeanswers.org) is shown in ▶,▶, through ▶. ▶ illustrates a search for osteoporosis screening. ▶ illustrates the search results and the secondary option of a query against one of the four most popular external reference systems. ▶ illustrates an example of an evidence answer with a link to the full document source. The location of the PrimeAnswers search box was moved to the top left “hot spot” based on user testing and usability research.⁴⁴ Direct links to reference objects and popular reference search boxes are aggregated on the home page. Organization of the home page is based on the question types and information needs identified during the preliminary needs assessment. Categories are expanded to a second layer through the home page tabs (e.g., patient handouts, calculators). Page layout is database driven, with the content packaged and displayed via a layout engine controlled by a content editor. Specific content items are placed in topics that are arranged on virtual pages. The virtual pages behave like normal Web pages to the user but exist only in the database.

Figure 1.

PrimeAnswers Portal: http://primeanswers.org. Most clinicians use the interface by typing one or two words in the PrimeEvidence search box; in this example, osteoporosis screening.

Figure 2.

Search results from the filtered set of PrimeAnswers reference objects are displayed. A secondary option to run the same term(s) in four external sources was a clinician-requested feature.

Figure 3.

A bottom-line answer for reference objects in PrimeAnswers is provided. The actual bottom line is provided within the system if allowable or alternatively a link to the specific section within an external document if anchor tags are provided.

System Description

Portal content is divided into sources, queries, and documents. The database contained 374 sources, 183 queries, and 821 documents in August 2003. Sources are links to aggregations of content such as the topics in British Medical Journal's (BMJ) Clinical Evidence. Sources provide the mechanism to create parent-child relationships between a source and its documents and queries. Queries are dynamic scripts that store all information (i.e., action, method, parameters) required to create a search against an external source such as UpToDate®. More than one query may be scripted against a target source. Queries are designed as open ended (a search box is displayed for user input) or closed (a topical search is scripted for a specific result). Query results are displayed in a new window at the appropriate page in the target source. Queries eliminate the need for clinicians to learn how to navigate to multiple clinical reference systems to enter a simple search while adhering to licensing restrictions of commercial content providers. For example, four one-click buttons are offered with each PrimeAnswers search result for UpToDate®, Micromedex®, PubMed, and MD Consult®. Clicking one of the buttons invokes a script that searches the external system using the same term(s) and opens the native results page in a new window. Clinicians identified both the sequence and sources that match their typical information lookup workflow.

Documents are informational objects within a source or in some cases independent objects. Metadata for each document are based on Dublin Core⁴⁵ descriptive information (i.e., title, creator, organization, year, keywords) enriched with fields for specific document type, evidence/authority, and an evidence answer or surrogate. Evidence sources were selected on the basis of quality, format, and clinical context. Sources selected were BMJ's Clinical Evidence, Family Practice Inquiries Network (FPIN) Clinical Inquiries, Clinical Practice and Rational Clinical Exam articles in Journal of the American Medical Association, Applied Evidence articles in Journal of Family Practice, practice guidelines of the U.S. Preventive Services Task Force (USPSTF), evidence reports of the Agency for Health Research Quality, and chapters from the Guide to Clinical Preventive Services. The design assumed that an evidence answer (brief clinical bottom line) would be extracted from open-source publishers (e.g., USPSTF) as well as license-friendly publishers or collaborating partners (e.g., FPIN) for fast, one-click retrieval. Licensed sources proved problematic. For publishers that did not allow extraction of a specific answer with a link to their full document, a deep link to a table, figure, or bottom line summary was provided as a surrogate. Most evidence source publishers did not provide metadata for internal structure or a persistent URI (uniform resource identifier). Internal structure mark up relied on HTML anchor tags. As all sources were available through open-source or institutional licensing, a link for the full source was always provided. During the course of the first year, publishers changed their URL (uniform resource link) and HTML tagging, requiring the PrimeAnswers content editors to monitor these systems continually and make global changes for hard-coded links. Critical references tables, predictions rules, and algorithms were duplicated in the PrimeAnswers system if open source to ensure availability and faster retrieval. High-demand calculators were programmed specifically for the PrimeAnswers system. Feedback from clinicians expanded the type and scope of documents to include Centers for Disease Control and Prevention infectious disease summaries, downloadable personal digital assistant tools, and patient education handouts. Clinician advisors selected the patient education content and recommended presentation in the familiar “hanging file” metaphor seen in many examination rooms. Content editors analyzed search terms and failed searches to select additional content coverage (e.g., hepatitis C, West Nile virus, SARS).

The PrimeAnswers system is a Web application built on Microsoft SQL Server® and ColdFusion® version 5, which includes the Verity search engine. ColdFusion® was selected for its rapid prototyping capability, ability to deliver usable interfaces, and use at other biomedical libraries. The portal front-end uses standard HTML to achieve cross-browser compatibility. JavaScript is used primarily for interactive tools and for system responsiveness. The system is deployed on a load-balanced, fault-tolerant cluster of Linux (database) and Windows (application) servers. The system is updated daily and a link checker runs weekly. A separate database is used to log user interactions to track date, time, browser type, internet protocol (IP) address, target URL, referring URL, and search string. All clicks on links are tracked to provide feedback on page design. Use of sources, queries, and documents can be analyzed whether initiated by a search or link. A Web interface to query the usage database provides output in an HTML table or an Excel® file.

Results

▶ shows the use of PrimeAnswers during the first year. An interaction is defined as either a search or selecting a link. Interactions averaged 711 per month (611 low/1,118 high) for 108 potential physician users at the three clinics: 421 per month at UWPN (70 physicians), 292 per month at GIMC (30 physicians), and 66 per month at FMC (eight physicians). In-clinic use was determined by defined IP ranges. The most frequently used content sources are listed in ▶. Prediction rules, calculators, and patient education handouts were the most frequently used document types. The most popular calculators and patient handouts are listed in ▶. Use of PrimeAnswers outside of the three clinics averaged 5,315 per month (3,838 low/7,174 high) during the first year.

Figure 4.

User interactions, March 2002–February 2003.

Table 3.

Most Frequently Used Reference Sources in Rank Order, March 2002–February 2003

UpToDate®

MD Consult® Reference Books

Micromedex®

NHLBI Clinical Guidelines

FPIN Clinical Inquiries

Clinical Evidence

MD Consult® Patient Handouts

Practice of Geriatrics article

Dermatology Online Atlas

PubMed MEDLINE

Table 4.

Most Frequently Used Reference Objects in Rank Order, March 2002–February 2003

Calculators	Patient Handouts
10-Year risk for CHD	Cholesterol
Body mass index	Diabetes
BPH symptom index	Back pain
Osteoporosis risk	Plantar fasciitis
PHQ-9	Knee exercises
Mini-Mental Status Examination	Rotator cuff Exercises

A user satisfaction survey was administered to clinicians at the three sites in January–March 2003 using paper forms with a return rate of 32% (35/108). The return rate by clinic was 14% FMC, 40% GIMC, and 46% UWPN. Distribution of respondents by clinic was five of eight FMC, 14 of 30 GIMC, and 16 of 70 UWPN. The demographic characteristics of those who responded to this survey (▶) indicate a difference in both discipline and years of practice from the preimplementation needs assessment (▶) in 2000. ▶ summarizes the results of the satisfaction survey. Seventy-four percent of respondents used PrimeAnswers. Of these PrimeAnswers users, 73% said they used it one to ten times per week, 87% said they used it while the patient was in clinic, 50% used it frequently to answer a diagnostic or treatment question for a specific patient, and 27% used it to provide patient education for a specific patient; 88% agreed that PrimeAnswers made it easier to find answers during a clinic day, 87% agreed that it made it faster to find answers during a clinic day, 62% agreed that it increased the frequency with which they looked for patient care information, and 88% agreed that it provided information that improved the care that they gave to their patients.

Table 5.

Postimplementation Demographic Characteristics of Survey Respondents, 2003

Gender		Specialty			Years in Practice
M	F	Internal Medicine	Family Medicine	Pediatrics	1–5	6–10	11–20	>20
51% (18)	49% (17)	66% (23)	26% (9)	6% (2)	23% (8)	37% (13)	29% (10)	11% (4)

Table 6.

User Satisfaction Survey Results, 2003

	Physicians at FMC, GIMC, UWPN	Residents at GIMC
Survey Response Rate	32% (35/108)	40% (30/75)
Use PrimeAnswers (PA) (Q1)	74% (26/35)	57% (17/30)
Circle one (Q2)
Use PA 1–10 times per week	73% (19/26)	100% (17/17)
Use PA 11+ times per week	27% (7/26)	0
Circle all that apply (Q3)
Use PA to answer a question before patient arrives	38% (10/26)	35% (6/17)
Use PA to answer a question while patient is in clinic	87% (22/26)	47% (8/17)
Use PA to answer a question after patient leaves	69% (18/26)	24% (4/17)
Use PA to answer a question outside of clinic	46% (12/26)	76% (13/17)
Circle one (Q4)
In clinic use PA mostbefore patient arrives	12% (3/26)	12% (2/17)
In clinic use PA mostwhile patient is in clinic	62% (16/26)	41% (7/17)
In clinic use PA mostafter patient leaves	27% (7/26)	18% (3/17)
Don't use PA in clinic	35% (9/26)	29% (5/17)
Ranking scale: very frequently, frequently, sometimes, rarely, never	Very frequently or frequently	Very frequently or frequently
Use PA to answer diagnostic or treatment questions related to a specific patient (Q5)	50% (13/26)	18% (3/17)
Use PA to provide patient education materials to a specific patient (Q6)	27% (7/26)	0
Use PA for learning outside of patient care (Q7)	15% (4/26)	35% (6/17)
Ranking scale: very strongly agree, agree, neutral, disagree, strongly disagree	Strongly agree or agree	Strongly agree or agree
PA makes it easier to find answers during a clinic day (Q8)	88% (23/26)	76% (13/17)
PA makes it faster to find answers during a clinic day (Q9)	87% (22/26)	76% (13/17)
PA increases the frequency that he/she looks for patient care information (Q10)	62% (16/26)	71% (12/17)
PA changes the way that I look for patient-care information (Q11)	77% (20/26)	71% (12/17)
PA has provided information that improves the care he/she gives to his/her patients (Q12)	88% (23/26)	88% (15/17)

FMC = Family Medicine Clinic; GIMC = General Internal Medicine Clinic; UWPN = University of Washington Physicians Network.

Although not targeted for this intervention, residents rotating through the GIMC became PrimeAnswers users and the same satisfaction survey was administered by paper form during six focus group sessions in March 2003 with a return rate of 40% (30/75). Fifty-seven percent used PrimeAnswers. Of the PrimeAnswers users, 100% used it one to ten times per week, 47% used it while a patient was in the clinic, 18% used it frequently to answer a diagnostic or treatment question for a specific patient, and none used it to provide patient education for a specific patient; 76% agreed that PrimeAnswers made it easier to find answers during a clinic day, 76% agreed it made it faster to find answers during a clinic day, 71% agreed that it increased the frequency with which they looked for patient care information, and 89% agreed that it provided information that improved the care that they gave their patients.

Discussion

Usage data from the targeted primary care clinics show that after initial release, the pattern of use did not rise significantly during the first year. For the clinicians using the system, over 85% of the targeted primary care physicians who used the system agreed that PrimeAnswers made it easier and faster to find answers during the clinic day and improved patient care. Sixty-two percent agreed that PrimeAnswers increased the frequency of providing educational materials to their patients. Over 92% agreed that it improved the care given to patients. Anecdotally, clinicians described how easy access to highly customized information objects improved the quality of their discussions with patients in the examination room. The kind of reference sources information physicians indicated that they would use in the preliminary needs assessment was in fact the information that they used in PrimeAnswers.

Overall usage of the PrimeAnswers system outside the targeted clinics nearly doubled during the first year. Anecdotal evidence indicated that general internists on the wards and physicians in other outpatient clinics chose PrimeAnswers over the Care Provider Toolkit as a more direct method of accessing clinical reference. Clinician feedback led to the development of a set of common conditions pages coauthored by librarians and attending physicians as a teaching tool for residents on the wards. Duke University Medical Center Library implemented a hosted pilot of PrimeAnswers for its Ambulatory Care Clinic in 2003.

Four other institutional projects were in development at the same time as the PrimeAnswers project in 2000–2003. Developed at Massachusetts General Hospital, PCOI^31,32 offered a portal with a Google™ search engine to retrieve guidelines, practice alerts, patient education, drug formulary, insurance coverage, procedures, forms, and calculators. Like PrimeAnswers, PCOI found that aggregating reference objects such as calculators and patient education was critical to meet the full spectrum of workflow needs in primary care. The most popular sources in the PCOI system were the drug formulary, UpToDate®, and patient instructions. Unlike PrimeAnswers, access to external reference sources was limited to direct links to the native search interface. Survey results in 2003 found that of the clinicians who used PCOI, 70% felt that they had saved 10 minutes or more a day as a result of using the portal.

The TRIP database^33,34 indexed evidence documents from heterogeneous sources and provided a simple Web-based search in 2000. While popular with family medicine physicians at UW, many documents were specific to the health care in the United Kingdom, and links to full documents were provided only if freely available on the Internet. PrimeAnswers focused on a leaner set of documents relevant to local practice with richer metadata and a link to the full online document as licensed by UW. The premium version of the TRIP database with broader content and search features became a subscription service in 2003.

The FPIN^35,36,37 developed a methodology to solicit and vote on clinical questions from practicing physicians, organize a clinician-librarian team to answer these questions based on a systematic review of the literature, publish the results as Clinical Inquiries in family physician journals, and create a searchable database of answers. FPIN is an institutional membership organization that relies on clinician faculty and librarians to create evidence answers. PrimeAnswers staff worked closely with FPIN during the early development by hosting the first searchable database and sharing its portal design to create the FPIN electronic library in 2002. The FPIN Clinical Inquiries were one of the most used evidence sources in PrimeAnswers because of specificity of the question and concise best evidence answer for primary care. The only limitation of this system was the low number of available answers in 2003, which grew to over 300 in 2005.

Developed at Stanford University, SHINE^38,39,40 provided a robust clinical reference search across multiple sources. It used full-document tagging for internal content, metadata for external open-access content, and a spell-checker. This approach provided a rapid and homogeneous search experience for clinicians. PrimeAnswers attempted a metasearch early in the project but abandoned the pilot due to slow response time, inconsistency of results, and lack of homogeneous vocabulary. Instead, PrimeAnswers developed the deep-linked query approach that relied on “screen scraping” against many heterogeneous systems as no applications protocol interface (API) or Web services were available. This approach allowed a much wider range of reference content but lacked the seamless results of the SHINE search. SHINE became SKOLAR MD in 2003, a commercial product available by license from Kluwer-Wolters.⁴¹ In 2004, the Lane Medical Library at Stanford created an institution-based search engine called Clinical Core Metasearch.⁴⁶ It builds on the PrimeAnswers deep-search techniques across multiple sources. It searches and integrates a categorized list of “hits” across heterogeneous clinical sources in less than five seconds from open-access, commercial systems and locally indexed documents. Results can be modified for a particular discipline (e.g., pediatrics).

In the future, document structure standards, Web services, XML results, and the proposed HL7 Infobutton API will simplify the integration of clinical reference to find a specific answer from a heterogeneous set of local and commercial information sources both in reference portals and directly in the electronic health record (EHR).^47,48,49 A structured clinical reference document standard similar to the HL7 Clinical Document Architecture⁵⁰ could be processed for automatic retrieval. More sophisticated Web services are now being provided by clinical reference publishers for direct integration into the EHR (e.g., Micromedex®, iConsult®, Clin-eguide®). By reducing the time needed to find the specific paragraph or answer within a clinical reference system, busy clinicians may increase their spontaneous use at the time that the question arises. Information technology should anticipate clinicians' needs and bring the information that they require to the point of care using “just-in-time” context-sensitive information retrieval directly in the EHR.

Conclusion

PrimeAnswers is an institutional approach to meet the information needs of clinicians in primary care clinics by providing a customized reference portal designed to reduce time and effort at the point of care. The portal increased clinician use of reference tools and resulted in a perception of improved patient care. An underlying assumption of the portal is that there is content present to answer clinical questions. The portal addresses the need to bring specific information objects such as patient handouts, risk calculators, and other “at-hand” tools into a simple search that has been very successful. PrimeAnswers is only a first step in solving the root problem of matching a clinician's question to a specific answer embedded in a larger document or set of documents. The next step is to standardize the internal structure of reference documents or synthesized answers using appropriate metadata to retrieve the most likely subtext to answer a specific patient care question. Ultimately, clinical reference applications will benefit from having an HL7 interface that enables direct interoperability with EHRs.