KnowledgeLink: Impact of Context-Sensitive Information Retrieval on Clinicians' Information Needs

Abstract

Objective: Infobuttons are message-based content search and retrieval functions embedded within other applications that dynamically return information relevant to the clinical task at hand. The objective of this study was to determine whether infobuttons effectively answer providers' questions about medications or affect patient care decisions.

Design: The authors implemented and evaluated a medication infobutton application called KnowledgeLink. Health care providers at 18 outpatient clinics were randomized to one of two versions of KnowledgeLink, one that linked to information from Micromedex (Thomson Micromedex, Greenwood Village, Co) and the other to material from SkolarMD (Wolters Kluwer Health, Palo Alto, CA).

Measurements: Data were collected about the frequency of use and demographics of users, patients, and drugs that were queried. Users were periodically surveyed with short questionnaires and then with a more extensive survey at the end of one year.

Results: During the first year, KnowledgeLink was used 7,972 times by 359 users to look up information about 1,723 medications for 4,961 patients. Clinicians used KnowledgeLink twice a month on average, and during an average of 1.2% of patient encounters. KnowledgeLink was used by a wide variety of medical staff, not just physicians and nurse practitioners. The frequency of usage and the questions asked varied with user role (primary care physician, specialist physician, nurse practitioner). Although the median KnowledgeLink session was brief (21 seconds), KnowledgeLink answered users' queries 84% of the time, and altered patient care decisions 15% of the time. Users rated KnowledgeLink favorably on multiple scales, recommended extending KnowledgeLink to other content domains, and suggested enhancing the interface to allow refinement of the query and selection of the target resource.

Conclusion: An infobutton can satisfy information needs about medications. Although used infrequently and for brief sessions, KnowledgeLink was positively received, answered most users' questions, and had a significant impact on medical decision making. The next steps would be to broaden the domains that KnowledgeLink covers to more specifically tailor results to the user type, to provide options when queries are not immediately answered, and to implement KnowledgeLink within other electronic clinical applications.

While providers often have questions during their clinical care activities, these questions frequently go unanswered, even when the answers are available. Studies have estimated that the frequency of questions that are generated during patient care ranges from one question every ten encounters¹ to almost four questions per encounter.² More than half such questions are never answered.^1,3,4,5 The reasons are usually the lack of time, effort, and experience required to formulate one's question explicitly, choose the most appropriate resources, and locate the answer within the resource. Indeed, it has been suggested that any information retrieval task that takes more than 30 seconds is unlikely to be generally adopted by practicing clinicians.⁶

Failure to answer these questions may have serious implications. For example, questions about medications are one of the most frequently generated in the course of patient encounters⁷; yet drug knowledge deficits have been implicated in 29% of adverse drug events (ADEs), and in up to half of all preventable ADEs.⁸ Ironically, there exist numerous excellent sources of drug information for the clinician, increasingly available in electronic formats that are readily accessible, often from the same workstations on which the clinicians are using to look up their patients' medical information in an electronic medical record (EMR).

Infobuttons are a way to bridge the gap between electronic clinical applications (such as EMRs) and electronic sources of reference information. Unlike a simple and static hyperlink to a knowledge resource, an infobutton is a message-based content retrieval function embedded within another application that dynamically returns information that is relevant to the situation/task at hand and that may be specific to the patient situation. One of the first infobuttons was Hepatopix,⁹ which mapped clinical findings, laboratory test results, and critical words or phrases within liver biopsy records to preconstructed MEDLINE searches about certain key topics in hepatology. PsychTopix was an extension of the same program to the psychiatric literature.¹⁰ In particular, Cimino et al.¹¹ have greatly advanced the field, first with MEDLINE button, then Infobutton,¹² and most recently Infobutton Manager.¹³

However, while studies have reported infobutton implementations,¹⁴ user evaluations,¹⁵ and usage statistics,¹⁶ few data are available regarding how effective infobuttons are at answering clinicians' queries or how the use of infobuttons affects medical decision making. Our aims were to describe how often and for what purposes physicians use knowledge links made available to them from within an EMR and to assess providers' perspectives on their impact on care and their satisfaction with links to context-specific knowledge.

Methods

We developed an infobutton called KnowledgeLink¹⁷ to provide patient-specific and context-sensitive links from medications that appear in a patient's EMR directly into the relevant sections of one of two reference resources. Briefly, KnowledgeLink embeds within Partners HealthCare Systems' EMR “look-up” buttons wherever medications appear. The button activates a URL to one of two Web-based information resources, Micromedex® or SkolarMD®, both of which have an application program interface for graphical user interface-independent searching. The queries are context sensitive in that the drug is automatically inferred using simple text-based parsing rules and inserted into the URL. KnowledgeLink displays the query results in a new browser window, which upon termination returns the user to exactly the point left in the medical record. In this way, KnowledgeLink allows a clinician with a question about a patient's medication to access information about that medication with one mouse click and similarly to return to their work with just one more.

The study population included primary care and medical subspecialty providers who access the longitudinal medical record (LMR), the outpatient EMR application used at several clinics throughout Partners Healthcare System. Institutional review board approval was obtained to randomize these clinicians by practice location to one of two groups: LMR with KnowledgeLinks to Micromedex (KL/MDX), or LMR with KnowledgeLinks to SkolarMD (KL/SKL). The randomization was block stratified to ensure that the number of users in the KL/SKL group would be fewer than 250 because only that many licenses for SkolarMD were donated for purposes of the study. Ultimately, a total of eight clinics (about 218 providers) were randomized to KL/SKL and ten clinics (about 372 providers) to KL/MDX. Providers at these clinics consisted of trainees (interns, residents, or fellows), nurse practitioners, and attendings. The providers were notified by an e-mail broadcast about KnowledgeLink, but no formal training or orientation was given.

KnowledgeLink was studied for one year, January 8, 2003, through January 7, 2004, during which time data were collected prospectively about the frequency of use and demographics of the users, patients, and drugs that were queried. Periodically, KnowledgeLink users were e-mailed a short online questionnaire asking them to specify the particular question they had in mind when they used KnowledgeLink, whether the question was answered by using KnowledgeLink, and how the answer affected patient care. These questionnaires were sent within 24 hours of a particular incident of use and contained the date, patient identification, and medication name in order to orient the responder to the particular use of KnowledgeLink being surveyed. So as not to burden them with too many questionnaires, users only were surveyed every fifth use and never more than once weekly.

At the end of the study, providers who had used KnowledgeLink at least five times and on at least two separate dates were asked to complete a more extensive online survey about their overall impressions with KnowledgeLink. Perseus SurveySolutions (Braintree, MA) was used to conduct all the online surveys.

In addition, for those users who gave their explicit consent, SkolarMD search logs were analyzed.

Statistical analyses were performed with SAS 8.0.19 (SAS Institute, Cary, NC). Univariate comparisons of means were analyzed with two-sample t-tests for normally distributed data, and two-tailed Wilcoxon rank sum tests were used for nonnormal data. For multivariate analyses, Poisson regression was used for non-normally distributed outcomes, and logistic regression for binary outcomes. Generalized estimating equations were used to control for correlation in the clustering variable (clinic).

Results

Usage Characteristics

During the one-year study period, KnowledgeLink was used 7,972 times by 359 unique users to look up information about 1,723 different medications for 4,961 distinct patients. There were 6,714 hits registered to Micromedex and 1,258 to SkolarMD, by 218 and 142 users, respectively (one user practiced at two sites). There were on average 22 uses of KnowledgeLink per user (range, 1–825). Providers randomized to KL/MDX averaged 37 uses in the year, while KL/SKL assignees averaged only 11; however, the range was so wide and distribution skewed (▶) that it is more informative to compare the median usage, which was six per user in the KL/MDX group versus five in the SK/SKL group.

Figure 1.

Frequency distribution of KnowledgeLink usage by clinician. The average frequency was 22 times per user per year, while the median frequency was five times per user per year.

Adoption of KnowledgeLink was rapid despite the fact that no specific education was provided about the new service. The rate of new users was highest at the start of the study, and the rate of KnowledgeLink queries remained relatively constant throughout the study period (please see Figure 2, available as a JAMIA online supplement at www.jamia.org). KnowledgeLink was used by a wide variety of staff (▶). In fact, about one-fourth of KnowledgeLink use was by providers who were neither physicians nor nurse practitioners.

Figure 5.

Breakdown of KnowledgeLink users.

The ten most frequently queried medications (▶) closely paralleled the list of most frequently prescribed medications.

Table 1.

Top Ten Medications Queried with KnowledgeLink

Medications	No. of Queries	Rank of Most Prescribed Medications (Range, 1–2,195)
Lisinopril (Zestril)	156	6
Atorvastatin (Lipitor)	94	1
Atenolol (Tenormin)	92	5
Citalopram (Celexa)	82	30
Pantoprazole (Protonix)	71	31
Hydrochlorothiazide	70	9
Paroxetine (Paxil)	63	46
Alendronate (Fosamax)	59	20
Rofecoxib (Vioxx)	57	54
Fluoxetine (Prozac)	39	21

Ranks refer to Partners HealthCare System prescription data.

Univariate Analyses

Clinicians used KnowledgeLink relatively infrequently (average 2.3 times per month; range, 0.1–100; median, 0.5). Usage was statistically significantly higher among those randomized to Micromedex compared to SkolarMD (median 0.56 versus 0.42 uses/month, p = 0.01), but the absolute difference was minuscule and probably not clinically significant. Frequency of usage did not vary significantly by user type or gender in the univariate analyses (please see Table 2, available as a JAMIA online supplement at www.jamia.org).

For a subset of randomized clinics (eight in the KL/MDX group, six in the KL/SKL group), the log of patient encounters was available to compare to the log of KnowledgeLink queries generated from those sites between 1/8/03 and 12/31/03. Analysis of the intersection of these logs (▶) showed that KnowledgeLink was used in a median of 0.7% of patient encounters (0.9% at KL/MDX clinics versus 0.3% at KL/SKL clinics, p = 0.01). Usage per patient encounters was not statistically associated with user type (primary care physician, specialist physician, nurse practitioner, or other) or gender (▶).

Table 3.

Median KnowledgeLink Usage per Patient Encounter by Target Resource, Type of User, Provider Training, and Provider Gender

Variable	No. of Patient Encounters	Median Percentage of Encounters in Which KL Was Used	p-Value
Target resource
SkolarMD	70,178	0.3%
Micromedex	139,792	0.9%	0.01
User type
Primary care physician	157,099	0.8%
Specialist physician	18,371	0.3%	0.17
Nurse practitioner	26,285	1.1%	0.43
Other	8,215	0.3%	0.14
User gender
Male	72,543	1.0%
Female	137,427	0.5%	0.23

KL = KnowledgeLink.

Multivariate Analysis

Unlike the univariate analysis, the multivariate regression model showed some effects of user type and gender on frequency of KnowledgeLink use (▶). Specifically, “Other” providers used KnowledgeLink significantly less often per patient encounter than did primary care physicians (rate ratio, 0.22; p = 0.03), and female providers used KnowledgeLink almost twice as frequently per patient encounter and per month as did male providers (p < 0.05). Once again, Micromedex assignees used KnowledgeLink more frequently than did SkolarMD subjects, about three times as often per month and per patient encounter (p < 0.01).

Table 4.

Multivariate Analyses of Determinants of Frequency Measures Based on Generalized Estimating Equations Fit to a Poisson Distribution to Account for the Nonnormally Distributed Sample

Variable	Use per Month Rate Ratio	p-Value	Use per Encounter Rate Ratio	p-Value
Target resource
SkolarMD	1.00		1.00
Micromedex	3.27	< 0.0001	3.23	0.006
User role
Primary care physician	1.00		1.00
Specialist physician	0.45	0.17	0.48	0.34
Nurse practitioner	0.89	0.78	1.03	0.97
Other	0.65	0.44	0.22	0.03
User gender
Male	1.00		1.00
Female	1.70	0.02	1.81	0.007

Although we could not directly determine whether usage took place while patients were being seen, KnowledgeLink use was most frequent on days of the week and times of the day when patients were typically seen (please see Figures 3 and 4, available as a JAMIA online supplement at www.jamia.org) when patients are typically seen.

Post-use Survey Results

Surveys were sent following 1,005 KnowledgeLink activations to 167 distinct users. Altogether, 289 completed surveys were returned from 89 distinct users (29% response rate). One author (SM) blindly reviewed the questions that prompted the use of KnowledgeLink and assigned them to one or more nonexclusive categories (dosing, indications, contraindications, drug interactions, side effects, pregnancy issues, pharmacology, and patient information requests). The most common type of question was about dosing, but there were some differences according to user role (▶). For example, physicians asked relatively more questions about dosing and drug interactions, nurse practitioners had relatively more requests for patient information, and others had relatively more questions about why a drug was prescribed and its potential side effects.

Figure 6.

Types of questions asked by users of KnowledgeLink. Questions were often multiple and fell into more than one category.

Overall, users reported that 83.8% of their queries were satisfactorily answered (86.0% KL/MDX, 72.5% KL/SKL, p = 0.1) and that 14.9% of the time KnowledgeLink had caused them to change their mind about a medical decision (15.2% KL/MDX, 13.7% KL/SKL, p = 0.7). Interestingly, specialist physicians reported that KnowledgeLink significantly impacted a medical decision almost twice as frequently as did primary care physicians (33.3% versus 17.0%, p = 0.06), while KnowledgeLink rarely affected nurse practitioners' decision making (6.7%, p = 0.07).

Post-study Survey Results

Among the wrap-up surveys, 72 of 389 were returned (19%). Of these, 80% or better of respondents rated KnowledgeLink positively on scales of ease of use, relevance, speed, and improvement in patient care, and 70% or more had positive impressions of the target reference, either Micromedex or SkolarMD (please see Tables 5 and 6 available as JAMIA online supplements at www.jamia.org). Overall, KL/MDX respondents tended to be more satisfied than their KL/SKL counterparts (87% versus 54%, p = 0.05), not so much in how often users reported that they could find answers to their questions, but more related to how quickly and easily the answers could be found. A majority of respondents from both groups recommended that KnowledgeLinks be extended to other domains (problems, diseases, laboratory test results, and reports) and that the ability to refine searches and search a variety of target references be added.

Search Log Analyses

Search path logs were available only for SkolarMD queries. Fifteen individuals consented to have their SkolarMD search path logs from 363 separate KnowledgeLink sessions analyzed. The median duration of a KL/SKL session (minus the time spent viewing the last page^*) was 25 seconds, during which time a median of four pages was examined, only one of which usually contained actual content, the other three being navigation pages. The median time spent viewing each content page was 21 seconds, while only a median 2 seconds was spent on each navigation page. The most frequently used resources in SkolarMD for KnowledgeLink queries about medications were A to Z Drug Facts and Drug Facts and Comparisons (for the full listing in rank order please see Table 7, available as a JAMIA online supplement at www.jamia.org). The former resource was available starting the second half of the study period, replacing “Lexi-Comp's Clinical Reference Library,” which ranked third.

Discussion

The findings presented here advance our understanding of how often infobuttons are used during patient encounters, how effective they are at answering clinicians' questions, and how often they can affect medical decision making. For example, KnowledgeLink adoption was rapid even though formal instruction about the new functionality was not provided. Many staff members, not just direct care providers, used KnowledgeLink to answer a variety of medication-related questions. The usage of KnowledgeLink per month that we observed was similar to the findings of Cimino et al.¹⁶ of online resources used while using a clinical information system. Users in that setting used infobuttons between one and 22 times over six months. Cimino et al. found that laboratory test results review was the most frequent context, supporting our users' requests to extend KnowledgeLink functionality to domains other than medications. They also noted that there are many different types of users, with corresponding differences in the resources they tend to use.

Overall, usage per patient encounter was low (about 1%). One reason may be that clinicians do not access the EMR during patient encounters, when their queries are generated, but before or after the patient visit because of time constraints. Another may be that clinicians may have been unaware or unfamiliar with KnowledgeLink, since we provided no education about it other than a simple announcement of the new functionality. Of course, it is also possible that clinicians had fewer self-recognized medication-related information deficits than would have been predicted based on previous studies^1,2 or that these deficits were about drugs not yet prescribed rather than about ones already on a patient's medication list, the main trigger for activating KnowledgeLink.

Despite the low overall usage, survey respondents reported KnowledgeLink to be very useful, convenient, and effective, answering over 80% of questions posed. This compares favorably with studies of other non-infobutton online drug information sources. For example, one group reported that complete answers to queries were obtained in less than a third of searches, that lack of time was a major factor limiting use, and that infrequent users felt the resource did not fit well with their daily work routines.¹⁸ Another study reported complete answers to queries in only 40% of cases and that the biggest problems were navigating software, translating clinical problems into questions, selecting appropriate resources, inadequate training, and excessive time required to access the information.¹⁹ Such problems were rarely cited by KnowledgeLink users.

In 15% of cases, use of KnowledgeLink significantly altered a medical decision. Interestingly, KnowledgeLink more frequently affected the decisions of specialist physicians than those of generalists or nurse practitioners. One study that examined how often answered queries result in altered decisions reported a rate of 20%. However, this pilot program relied on general practitioners mailing their questions to a local academic general medicine department; the median time to research and compose answers was 2.5 hours, and the median interval between receipt of questions and dispatch of answers was 3 days.²⁰ Taken together, these data suggest that an infobutton such as KnowledgeLink can help answer questions that occur during routine practice, with substantially less investment effort and time.

Although users did not directly compare one target resource with the other, the target resource correlated significantly with how often KnowledgeLink was used and with users' self-reported impressions. There are a number of possible explanations. First, although SkolarMD has a rich set of medication information resources, it does not focus on that domain as much as Micromedex, and therefore users may have preferred Micromedex over SkolarMD for medication information. Indeed, A to Z Drug Facts, the most frequently used resource in SkolarMD, only became available halfway through the study period. However, when each half study period was analyzed separately, there was no significant qualitative change in the comparison between the KL/SKL and KL/MDX groups with respect to the median utilization per month, the median utilization rate per encounter, or the rate at which users' queries were answered.

Another factor that may account for the discrepant usage rates between SkolarMD and Micromedex may be users' familiarity with each of the resources. Micromedex had been available to users at Partners for over 10 years prior to this study, while SkolarMD was only first made available to KnowledgeLink users for this study, and no training or orientation was provided. Therefore, users randomized to KL/SkolarMD may have been negatively affected by having to learn how to navigate a new knowledge resource.

Third, differences in user interface between SkolarMD and Micromedex, rather than information content, may also have played a role. This is suggested by the fact that users of both groups reported similar rates of obtaining answers, but diverged on questions about how easy, intuitive, and quick each resource was to navigate (please see Table 6, available as a JAMIA online supplement at www.jamia.org). For example, that only a median of one in four pages displayed by SkolarMD was content versus navigation may have discouraged use in the long term. Unfortunately, comparable search log data were not available from Micromedex in order to evaluate this hypothesis further.

Finally, although we did not find evidence of such, bias may have been introduced by the block-stratified randomization scheme employed to limit the size of the SkolarMD group. For all these reasons, firm conclusions about the relative merits of the target resources should not be made.

Despite the overall low usage, the service was praised, both in terms of end-of-year survey responses as well as in general comments received in the periodic surveys. It was also reassuring that clinicians in just seconds received answers to their questions over 80% of the time and that KnowledgeLink use had a significant impact on medical decision making 15% of the time. We had originally intended to measure whether KnowledgeLink use translated into improved patient outcomes, but because KnowledgeLink was used so infrequently, the sample size required to detect a statistically significant effect would have been enormous. Instead, we only are able to present self-reported rates of impact. However, as is most clearly summarized by one user's comment: “This info actually saved me from making a medical error…I rarely use this feature of the LMR, but this time it really helped me.”

These results suggest that users find the delivery of context-specific knowledge to be useful.

This study has several limitations. It was carried out at only one institution, so that the results may not generalize to sites using a different electronic health record. The survey response rates were low, and it is possible that non-responders may have been less enthusiastic about KnowledgeLink than were responders. We did not aggressively encourage use of the knowledge resources nor provide training for KnowledgeLink, and it is possible that usage might have been greater had we done so. In addition, the application might have been used more had we covered more domains than just medications.

Overall, these data suggest that infobuttons such as KnowledgeLink can satisfy information needs of busy clinicians by incorporating automated search and retrieval functionality within the same electronic clinical applications they use to complete their work. To promote increased use of such tools, we plan to expand the number of domains that KnowledgeLink covers (such as problems, diseases, and test results) and to attempt to better anticipate clinicians' knowledge needs based on the specific context (such as the user's role and the particular application functionality being used).