Abstract
Background
We sought to understand how clinical information systems users access online information resources through an Infobutton Manager (IM) that provides direct, context-specific queries.
Methods
We used 5.7 years of system log files, one year of pop-up surveys and user feedback, and an e-mail survey to collect information about our users' experience with online resources and the IM.
Results
4,577 users used the IM 30,374 times to access information resources 20,249 times. We obtained 195 pop-up surveys, 108 instances of online feedback, and 70 e-mail surveys. User satisfaction with the IM ranged from 69% to 92% for various aspects and user groups; 74% felt that the IM had a positive effect on patient care decisions and 20% reported specific positive impact on patient care.
Conclusions
Context-specific access to health knowledge resources has been successfully accomplished, although the success varies with context and user type. Impact on patient care decisions has been positive. Our findings suggest ways to further increase use of the IM.
INTRODUCTION
The information needs of clinicians have been widely studied over the past 20 years, with consistent findings: needs occur more often than realized, with a large proportion remaining unresolved. Studies of clinicians using clinical information systems have produced similar findings.1 One approach to addressing such needs is context-specific links between clinical data and information resources, which we (and others) refer to as “infobuttons”.2
Recent studies have examined the effects of such links on resolving information needs. For example, Rosenbloom and colleagues at Vanderbilt University studied their Patient Care Provider Order Entry with Integrated Tactical Support (PC-POETS) application, which provided topic-specific links to on-line resources relevant to specific orders as the user was entering the orders.3 PC-POETS was used by 105 house officers 278 times during the 11.5 month study period, compared to 18 uses of resources by control subjects.
In a second study, Maviglia and colleagues at Partners Healthcare Systems studied not only the use of their infobutton application (Knowledgelink), but its impact on satisfying clinicians' information needs.4 They randomized Knowledgelink to provide access to two different drug knowledge resources and used log files, e-mail surveys and on-line surveys to determine how well the system performed and whether there was any impact on patient care. They observed 7,972 uses by 359 users (mostly physicians and nurses) over a one-year period. Their users reported obtaining answers to 84% of their questions, with an impact on patient care 15% of the time.
We have modified our original infobutton approach to incorporate an Infobutton Manager (IM), which offers the user a set of context-specific questions from which to chose. Each question, in turn, acts like an individual infobutton, by serving as a context-specific link to a resource that provides information about the concept of interest, as it relates to the user's specific information need.5 To date, our IM has been integrated into one “home-grown” and one commercial clinical information systems at New York Presbyterian Hospital (NYPH), as well as two systems at other institutions (screen shots of our IM can be found in Appendix Figures 1–7).
We sought to study the use of our IM at our institution, in order to learn who, what, where, how, and why it is being used, as compared to traditional access to on-line resources. We used of log file analyses, pop-up surveys, and an e-mail survey to obtain quantitative and qualitative information on the use, usability, and usefulness of the IM, as well as its impact on patient care decisions.
METHODS
System Log Files
The clinical information system (WebCIS) at the Columbia University Medical Center campus of NYPH provides users with a link to a “Health Resources” (HR) page that, in turn, includes links to a variety of popular health knowledge resources. A previous study has shown that clinicians in our institution use the HR page as their preferred method for accessing health information resources.1
WebCIS records in a system log file all actions taken by its users, including all accesses to the HR page and its links, and the IM. The IM, in turn, writes a record in the log file each time a user selects a question from the IM page. Log file records include user ID, patient ID, IP address of the user, time, date, and other activity-specific details. If a user accesses the HR page without logging into WebCIS, the log file records are considered “anonymous”.
In September 2005, we integrated the IM with NYPH's installation of the Eclipsys XA system (Eclipsys Corporation, Boca Raton, FL). System logs from XA are not available for analysis. While IM records are available, user information was not included in the parameters passed to the IM, due to limitations of the integration method. Therefore, IM usage by XA users is recorded as “anonymous”.
We reviewed log files from August 2002 through February 2006 to identify all access to the HR page, the links on the HR page, the IM, and the questions on the IM page. Where possible, we identified the user and the WebCIS or XA activity that immediately preceded the HR or IM use. All records for users in the Department of Biomedical Informatics were excluded from the analysis.
Pop-Up Survey
In January 2005, we instituted an on-line survey containing questions about ease of use, success in finding information, and helpfulness. The survey is presented to IM users as a “pop-up” Web browser window that appears randomly, four out of every ten times that a user selects a question from the IM page. Appendix Figure 8 shows the pop-up survey. When a user answers the questions and submits the form, the responses and contextual information about the user, task, and selected IM question are e-mailed to the research team. When a user expresses dissatisfaction, we simulate the user's experience and attempt to contact the user, via e-mail, to learn more about their reasons for dissatisfaction.
Infobutton Manager Feedback
The IM page includes a suggestion box, labeled “What question did you wish you could ask but wasn't listed?” (see Appendix Figure 9). When a user submits a suggested question, it and contextual information about the user, task, and selected question are sent by e-mail to the research team.
E-Mail Survey
An e-mail survey was sent in March 2006 to all people who used the IM at least once during the preceding two months. The survey included five Lickert-scale questions about the usability and usefulness of the IM and the links it provided. Two open-ended questions were also included to elicit comments about ways in which the IM might have helped or hindered a patient care decision. User roles were determined from the WebCIS user directory. The survey can be found in Appendix Figure 10.
RESULTS
System Log Files
In March 2004, when the IM was integrated with WebCIS, 1,224 users accessed it 2,407 times (78/day). Usage rapidly fell to about half that rate, but began to rise gradually, to reach 2,329 uses per month (83/day) by 1,024 users in February 2006 (including 181 uses by Eclipsys XA users, since the IM was integrated with that system in September 2005). 615 WebCIS users accessed the HR page 3,504 times in March 2004 (113/day) and by February 2006, 758 users accessed it 5,528 times (197/day). Over the entire two-year period, 4,577 users used the IM 30,374 times, while 3,889 users used the HR 116,214 times. Appendix Figures 11–12 show this usage over the entire period of the study.
Most users could be categorized as one of four types: nurse, attending physician, housestaff, and student. The roles of the remaining users either could not be determined or represented several small user groups, such as pharmacists, social workers, etc. These users were treated as a fifth “other” group.
By February 2006, the IM was integrated into six WebCIS contexts (laboratory results review, microbiology culture results review, microbiology sensitivity results review, inpatient drug order review, outpatient drug order review, and diagnosis list) and two Eclipsys XA contexts (inpatient drug orders and inpatient lab orders). Relative use of the IM and HR in each of these contexts, by each user type, is shown for some contexts in Figure 1 and all contexts in Appendix Figure 13. In general, we found that similar numbers of nurses (928), attending physicians (977) and housestaff (988) used the IM. Attendings and housestaff tended to use the IM less than the HR, while nurses and students tended to use the IM more than the HR. In most application contexts, HR was used far more than IM; the exception was inpatient medications, where the IM was used three times as often.
Figure 1.
Selection of IM and HR links, based on user type, the three contexts (described in the text). Data are from March 2004 to February 2006, inclusive. During that time 1,032 nurses, 1,063 attending physicians, 1,102 housestaff, 469 students and 1,374 other users used the IM and/or the HR. Note that housestaff and attendings generally use the HR more than the IM, with a reverse pattern in the InPatientDrugs context.
Once a user evoked the IM, they only selected a question link 48.7% of the time over all. However, the rate of question selection was substantially higher in some contexts, and for some user roles, than others. For example, when the application context was inpatient drug order review, questions were selected between 63.3–78.7% of the time, depending on user type, while the rate ranged between 23.0–63.3% for outpatient drug order review. On the other hand, in many cases, users chose more than one question after evoking the IM. By contrast, users chose at least one resource link from the HR page 90.6% of the time, with a range of 82.6–93.7%. Appendix Figure 14 shows the rate at which users chose zero, one, or more links in the IM or HR.
The introduction of the IM into WebCIS and Eclipsys XA produced dramatic increases in access to health resources in some contexts. Figure 2 shows, for six selected contexts, access to health resources via the HR links over time from August 2002 to February 2006 and the additional access to health resources that occurred, beginning in March 2004, via the IM. Appendix Figures 15 and 16 show the total access, over time, for all contexts. All in all, since its introduction two years ago, the IM has helped users access resources 20,249 times and currently is doing so at the rate of 1,226 per month.
Figure 2.
Monthly use of Health Resource (HR) and Infobutton Manager (IM) links for several application contexts. The bottom (white) regions represent access via the HR page and the top (colored) regions represent access via the IM. Deployment of the IM in Eclipsys XA began in September 2005 for inpatient lab order entry and inpatient drug order entry, January 2006 for Diagnosis List (not shown), and March 2004 for the remaining contexts. Note that Eclipsys XA does not provide users with access to the HR page.
Pop-Up Survey
Users answered the survey 195 times out of approximately 3,642 times it appeared, for a response rate of 5.4%. 83% of respondents agreed or strongly agreed that the sites were easy to navigate, 69% agreed or strongly agreed that they got an answer to their question, and 77% agreed or strongly agreed that the sites were useful. Nurses and attendings tended to have more positive responses than students and housestaff. Appendix Figures 17 and 18 show details of the responses.
Of the 82 negative responses to the three questions, 33 (40%) were associated with the hospital's laboratory manual. When we investigated the users' experiences, we found that the manual was unavailable during 7 instances, accounting for 21 (26%) of all negative responses.
Infobutton Manager Feedback
The users used the feedback feature on the IM page 135 times. Of these, 108 comments included questions or topics. The most common contexts in which questions were provided were Laboratory Results Review, where questions were about interpretation of the test result (30 of 54 questions) and inpatient drug order review, where questions were about the side effects, dosage, cost, trade names, and interactions of the drug (15 of 22 questions). Appendix Table 1 lists the questions, the contexts, and the types of users.
E-Mail Survey
Of the 1,410 users who accessed the IM during the months of January and February 2006, e-mail addresses were available for 1,228 (224 nurses, 301 attendings, 301 housestaff, 76 students, and 316 others). Seventy-one users (5.8%) responded to the e-mail, including eight that declined to complete the survey. Survey respondents included 11 nurses, 13 housestaff, 25 attendings, 5 students, and 9 others.
Overall responses to all questions were very positive. When questions were normalized to a 1–5 scale, with 1 being strongly positive, 3 being neutral and 5 being strongly negative, 86 of 306 responses were strongly positive (28.1%), 127 were positive (41.5%), 73 were neutral (23.9%), 18 were negative (5.9%) and 2 were strongly negative (0.7%). Attendings and housestaff were more positive about all aspects of the IM (73.3% and 78.6%), while nurses and students were less so (60.4% and 60.0%).
Overall, 92% of respondents felt that the IM was easy to use, 89% felt that the desired question was on the IM list at least half the time, 62% felt that the IM was faster than their usual method for answering questions, 90% felt that they received a helpful answer at least half the time, and 74% felt that the IM had a positive effect on their patient care decisions. Appendix Figures 19–22 provide details of the survey responses. Fourteen of the respondents reported specific instances of improved patient care decisions, including faster decision making, correction of a medication dose, and decision to perform a biopsy. No respondent reported any negative impact on patient care decisions, and fifteen reported specifically that the IM had not done so.
DISCUSSION
The current study used a multi-modal evaluation approach, including two years of log files, 195 pop-up surveys, 108 instances of online feedback, and 70 e-mail surveys, to study the use of the IM by 4,577 users over a two year period. We were able to tell who our users were, the contexts in which they used the IM, the questions that they were attempting to answer, and their success with answering them, as well as the cases where they did not find an appropriate question and what questions they had in those cases.
This study, the largest reported to date, reinforces the findings of previous studies of the impact of context-specific access to health knowledge on clinical information system users and their decisions. For example, our various measures of satisfaction ranged from 69–92%, similar to the 84% rate found by Maviglia and colleagues. The IM was reported to have a positive impact at least half the time by 74% of respondents (compared with Maviglia's 15% rate).
While the results of the study are encouraging, showing increasing and enthusiastic use of the system with positive results, we also note areas where improvement is needed. Despite the ease of use, speed, anticipation of questions, and provision of appropriate answers to those questions, the increase in use of the IM has been more gradual than expected (or desired). Although the IM has dramatically increased accesses to online resources in some contexts (such as inpatient drug order review), use has been paradoxically low in other contexts (such as in outpatient drug order review).
Part of the problem is clearly that on over half of the occasions that a user evokes the IM, he or she fails to select a proffered question. This suggests that either the IM is failing to anticipate the user's need or that the user is unable to find the question on the IM page in a timely way. Closer inspection of the log file shows which contexts, and for which user types, this is occurring most often, but it does not tell us why. For example, 57.8% of the time that housestaff evoked the IM while reviewing laboratory results, they did not select a question. Four of the eleven questions that were suggested by housestaff in this context dealt with laboratory test reference ranges, suggesting that this may be a major area to be addressed. However, when the IM is invoked in this situation, it almost always provides a test-specific link to the hospital laboratory manual, which contains the reference ranges for the tests. So, if this is the question of interest one third of the time, why aren't users selecting the question?
The present study provides useful information about the contexts and user types for which the IM requires additional development (e.g., to improve the relevance of questions for nurses and students in all contexts and for all users in the laboratory detail context). Additional methods, such as observational studies5 and analysis of the timestamp data in the log files, will help identify whether the problem is the question selection (in which case, questions can be added to the IM's database) or question presentation (in which case, redesign of the IM user interface will be in order).
This study has also shown that attention must be paid to the resources invoked by the IM questions. Users blame the IM when a resource (such as the lab manual) is not working. In response, we have developed a system that automatically tests each IM resource and notifies system developers when a failure is detected. This system has consequently provided several timely warnings of resources that have gone off-line (including two instances involving the lab manual), allowing us to fix them quickly.
The IM compares favorably to other NYPH systems, which typically rate between 40–60% on internal NYPH user satisfaction surveys (except WebCIS itself, which consistently rates 80%; unpublished NYPH data). The delivery of over 20,000 accesses to online resources is a substantial accomplishment and represents 9% of the overall use of health resources by clinical information system users at NYPH. A large percent of respondents to our survey felt that the IM had had a positive effect and 14 respondents identified one or more specific situations in which patient care improved. Overall, we conclude that the IM has produced an absolute increase in access to online resources and is viewed in a generally positive light.
The interpretation of our results is subject to some limitations. First, we cannot provide denominators for the log file statistics, since we don't know, at any given time, how many active WebCIS users there were, nor can we tell how many opportunities users had to evoke the IM. However, we know that the number of IM users in each user group was similar to the numbers of HR users. We also know that the IM provided a significant proportion of all access to online resources (20–90%, depending on context).
Another limitation is related to the response rates of our evaluation interventions. Certainly, response rates of 5.4% and 5.8% cannot be presumed to be representative. However, we can at least state that we have evidence of 210 satisfied users (out of 266 (195+71) respondents) and that we have improved the care of at least 14 patients.
Our findings support the hypotheses that the IM improves access to online information and that improved access to online information is a factor in improved patient care decisions. We can therefore conclude that the IM is improving patient care, although the magnitude has yet to be determined. Our findings to date should encourage and inform other institutions,4,5 vendors,6 and standards development organizations7 that seek to implement their own Infobutton Managers.
CONCLUSIONS
Context-specific access to health knowledge resources has been successfully accomplished for clinical information system users at NYPH, although the success varies with context and user type. Impact on patient care decisions has been positive. The main area for improvement appears to be the rate at which the IM anticipates the user's information need. Addressing this issue should lead to increased IM use which, in turn, will lead to increased information resource use and result in clinicians making better informed patient care decisions.
Acknowledgments
This work is supported in part by NLM grant R01LM07593. The author thanks Sue Bakken, Leanne Currie, Beth Friedman, and Vimla Patel for their input on the evaluation process, Jianhua Li for programming support, and Rick Gallagher for assistance with the log files.
Appendix
Due to space considerations, illustrative figures and tables that are not critical to understanding the paper or the study results have been archived at: www.dbmi.columbia.edu/cimino/amia06-app.pdf
References
- 1.Cimino JJ, Li J, Graham M, Currie LM, Allen M, Bakken S, Patel V. Musen MA, editor. Use of Online Resources While Using a Clinical Information System. AMIA Fall Symp. 2003:175–9. [PMC free article] [PubMed] [Google Scholar]
- 2.Cimino JJ, Elhanan G, Zeng Q. Supporting Infobuttons with Terminological Knowledge. JAMIA. 1997;4 (Suppl):528–532. [PMC free article] [PubMed] [Google Scholar]
- 3.Rosenbloom ST, Geissbuhler AJ, Dupont WD, et al. Effect of CPOE user interface design on user-initiated access to educational and patient information during clinical care. JAMIA. 2005;12(4):458–73. doi: 10.1197/jamia.M1627. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Maviglia SM, Yoon CS, Bates DW, Kuperman G. KnowledgeLink: impact of context-sensitive information retrieval on clinicians' information needs. JAMIA. 2006;13(1):67–73. doi: 10.1197/jamia.M1861. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Cimino JJ, Li J, Bakken S, Patel VL. Theoretical, empirical and practical approaches to resolving the unmet information needs of clinical information system users. JAMIA. 2002;9 (suppl):170–174. [PMC free article] [PubMed] [Google Scholar]
- 6.www.thomson.com/infobuttonaccess
- 7.www.hl7.org/Library/Committees/dss/infobutton-proposal-2006-01-04.zip