An Integrated Approach to Computer-Based Decision Support at the Point of Care

Abstract

Information needs that arise when clinicians use clinical information systems often go unresolved, forcing clinicians to defer decisions or make them with incomplete knowledge. My research characterizes these needs in order to build information systems that can help clinicians get timely answers to their questions. My colleagues and I have developed “Infobuttons”, which are links between clinical information systems and on-line knowledge resources, and have developed an “Infobutton Manager” (IM) that attempts to determine the information need based on the context of what the user is doing. The IM presents users with a set of questions, each of which is a link to an online information resource that will answer the question. The Infobutton Manager has been successfully deployed in five systems at four institutions and provides users with over 1,000 accesses to on-line health information each month, with a positive impact on patient care.

Introduction

Attention to the issue of medical errors increased dramatically with the publication of the Institute of Medicine's report examining fatalities related to adverse events (AEs) in health care (1). Subsequently, several studies have examined the types of errors that result in AEs (2–5) A recurring theme in these studies has been that clinicians lack adequate access to information.

My colleagues and I (6) and others (7,8) have begun to address this problem by first developing frameworks for understanding information needs, with particular attention to understanding how various information needs might be resolved through evidence-based practice (9). We have subsequently applied our framework to the analysis of observational data, collected while clinicians were practicing in a variety of patient care settings (10–14).

We have previously experimented with this approach by creating a link, called the Medline Button (15) between one of our clinical information systems and a Medline search engine. With the advent of the World Wide Web, and its attendant resources, we expanded our approach and renamed it the “infobutton” (16). Based on the results of our observational studies, we concluded that while many of the information needs arising during typical patient care could be addressed with direct links from the electronic medical record system, infobuttons were not flexible enough to accommodate the variety of information needs that might arise in any given context. We therefore developed a system, called the Infobutton Manager (IM), that attempts to match the clinician's setting (i.e., the clinician/user's characteristics, the patient's characteristics, the task being performed and the actual patient data available) with the likely information needs and then provides links to resources that can automatically address the need (Figure 1) (17). The purpose of this paper is to present the IM technology, describe experience with its development and usage to date, and review the lessons learned.

Fig. 1.

Steps needed to anticipate and address clinician information needs during use of clinical information systems. Some initial understanding of the likely needs of the user must be made, based on information extracted from the electronic medical record (EMR). An information resource must then be identified that can address the need. Once the resource is known, the clinical information can be translated from its original form into one that will be recognized by the resource. With the information properly translated, the resource can then be automatically queried and the results can be presented back to the user.

Identifying Clinician Information Needs

One approach to developing the IM might have been to establish a set of links for the users based on the resources available or the questions that we believed would be most appropriate for a given context. We chose instead to let the users' information needs guide the development of the system. Previously, we used a variety of modalities to learn about user information needs, including surveys, focus groups, and analysis of system log files showing which on-line information resource clinicians were using (11,13). However, discordance among the results led us to believe that these methods were not providing accurate information and that direct observational studies would better reflect the true information needs of clinicians as they used clinical information systems.

In order to observe clinicians in actual practice, we developed a “portable usability lab” (Figure 2) that used a video cassette recorder to capture the users' interactions with their computers. Subjects were asked to think aloud as they used their systems in the normal manner. Video was captured directly from the computer screen using a digital video converter (Scan Do Ultra, Communications Specialties Inc., Hauppauge, New York), obviating the need to use a more intrusive video camera, while audio was captured with a small microphone.

Fig. 2.

The Portable Usability Lab. The “lab” consists of a video converter and microphone, placed at the user's workstation (normally hidden from sight) and cart containing a video cassette recorder (VCR), a laptop computer to control the video converter and a video monitor. The converter and microphone are connected to the cart equipment via a 75-foot cable, allowing the researcher to observe and record from a distance the user's activity on the workstation.

Once the videotape was recorded, the subjects' comments were transcribed and analyzed to detect any explicit (spoken) or implicit (based on use of information resources) information needs. The needs were than coded using a modified classification system that characterizes the type of information need, the likely resource for resolving the need, and the subject's success at resolving the need.

Design of the Infobutton Manager

Each information need discovered through our observational studies was represented with a question (e.g., “What are the guidelines for heparin use?”) and a Web link known as a Uniform Resource Locator or URL (e.g., “http://infonet.nyp.org/Pharmacy/Pharmacy-M/H—I/HeparinGuidelinesWeb-9.05.pdf”). In most cases, the URLs required some customization to meet specific needs. For example, the query “What is the differential diagnosis of 〈X〉 in (PubMed)?”, where “〈X〉” is some finding that the user might be reviewing (such as a lab test result) would be paired with a URL that looks something like: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?term=〈X〉+AND+diagnosis[MeSH+Subheading]

When evoked, the IM will automatically replace the “〈X〉” with the user's term of interest. The questions and URLs are stored in a database, along with information indicating the situations in which they might be appropriate (e.g., ordering heparin or reviewing laboratory findings) and for which users (nurses, physicians, etc.) and which patients (based on gender and age). For example, the heparin guideline question would be appropriate when a physician is ordering heparin for an adult patient or a nurse is reviewing a heparin order on an adult patient*

Deployment and Usage

The IM is integrated with clinical systems as a link that a user may select when he or she has an information need. When the user clicks on a link, the link evokes the IM, passing it contextual information (user type, patient age and gender, the clinician's task, and the term of interest (e.g., a microbiology culture result or an order for heparin). The IM matches the contextual information against its database of questions, customizes them as appropriate (e.g., replacing the “〈X〉”'s with terms of interest), and presents the questions to the user as a list of World Wide Web links. Some example screens are shown in Figure 3.

Fig. 3.

Screen Shots of the Infobutton Manager. A clinical information system is shown in the upper half of the figure (in this case, WebCIS, from New York Presbyterian Hospital), showing laboratory results for a fictitious patient. Each test result has an Infobutton (small round icons with the letter “i”); if a user clicks on an Infobutton, such as the one to the right of “PROTHROMBIN”, the Infobutton Manager is evoked and it produces a list of questions (as shown in the lower left part of the figure). If the user clicks on one of the questions, such as “What is the NYPH guideline for managing adult patients with elevated INR due to warfarin?”, a page is produced from an on-line resource (in this case, a guideline from the NYPH guideline library).

Links to the IM have been integrated into WebCIS, New York Presbyterian Hospital's (NYPH) proprietary clinical information system (18), and into the Eclipsys XA order entry system, recently installed at NYPH. Links have also been integrated into the New York State Psychiatric System's PSYCKES system, the Regenstrief Medical Records System at University of Indianapolis, and the NextGen system (NextGen Healthcare Information Systems, Inc., Horsham, PA) installed at Crystal Run Healthcare, a large physicians' practice in Monroe, New York.

Evaluation

The primary method for evaluating the use of the IM was through review of the log files. These log files record every instance in which a user clicks on the IM link and every instance in which a user clicks on a question in the list provided by the IM. In each case, the log file shows the activity that was being carried out immediately prior to clicking on the IM link. The log file also indicates instances in which users searched other on-line health information systems directly. The log file analysis was supplemented with on-line (“pop-up”) surveys and e-mail surveys to determine user impressions of usability, usefulness and impact of the IM. The specifics of these methods, and some preliminary results, were recently presented (19).

Observations of Clinician Information Needs

Over a two-year period, we have carried our observations in a variety of settings, including in-patient nursing units, intensive care units, and out-patient clinics. A total of 251 subjects have been observed, for a total of 36.25 hours of recorded observations. Analysis of these recordings is on-going but has yielded over 250 information needs thus far.

The information needs have fallen into roughly three equal groups: those that require specific clinical information from the patient's record, those that require institution-specific information and those that can be resolved through use of on-line information resources. For the latter group, we have assembled a list of 101 questions thus far, each of which corresponds to some information need from the observations, and each of which provides an answer through an automated link to an on-line resource. The resources can range from simple documents to on-line search engines (such as PubMed) to expert systems. A sample of the questions, the contexts that evoke them and the resources they use to resolve them is shown in Table 1.

TABLE 1.

Sample questions from the Infobutton Manager knowledge base, showing the contexts in which the questions are evoked and the resources used to answer them.

Question	Context	Concept of Interest	User Roles	Patient Gender	Patient Age	Resource
What are the NYPH Guidelines for potassium replacement in adults?	Lab Order	POTASSIUM TEST	All	Both	Adult	NYPH Guidelines
What is the NYPH ICU Hyperglycemic Protocol?	Drug Order	INSULIN	All	Both	All	NYPH Guidelines
What is the differential diagnosis of 〈X〉?	Lab Order	FINDING	All	Both	All	Dxplain
What is the adult dose?	Drug Order	DRUG	All	Both	Adult	Micromedex
What is the pediatric dose?	Drug Order	DRUG	All	Both	Children	Micromedex
What are its dosages and forms?	Drug Order	DRUG	All	Both	All	Micromedex
What are the adverse effects?	Lab Order, Drug Order	DRUG	All	Both	All	Micromedex
What are the drug interactions?	Lab Order, Drug Order	DRUG	All	Both	All	Micromedex
What is the pregnancy category?	Drug Order	DRUG	All	Female	Child-Bearing	Micromedex
What diagnostic information does FirstCONSULT have for 〈X〉?	Diagnoses	DISEASE	All	Both	All	First Consult
What is the drug summary from Drug Consult?	Drug Order	DRUG	All	Both	All	Drug Consult
What is the toxicity of 〈X〉?	Lab Order	DRUG	All	Both	All	PubMed
What does the NYPH Lab Manual say about this test?	Lab Order	LAB TEST	All	Both	All	NYPH Lab Manual
What is the Spanish patient information from Lexi-Comp?	Drug Order	DRUG	All	Both	All	Lexi-Comp
What does UpToDate say about 〈X〉?	Drug Order	DRUG	All	Both	All	Up to Date
What does Harrison's say about 〈X〉?	Drug Order	DRUG	All	Both	All	Harrison's Princ. Int. Med.
What are NYPH guidelines for managing adult patients with elevated INR due to warfarin?	Drug Order, Lab Order	WARFARIN, COAG. STUDIES	All	Both	Adult	NYPH Guidelines
What are the NYPH guidelines for managing pediatric patients with elevated INR due to warfarin?	Drug Order, Lab Order	WARFARIN, COAG. STUDIES	All	Both	Child	NYPH Guidelines

Log File Analyses

The log files were reviewed from March 2004 through September 2006. During that period, 2,728 unique users accessed information resources 28,519 times with the IM. By comparison, 2,901 unique users accessed other health information resources 149,974 times. The relative rate at which the IM was used, as compared to other sources, varied based on the clinical task the user was carrying out, as shown in Figure 4. Recently, the IM has been used an average of 1,200 times a month by approximately 300 unique users. Figure 5 shows the usage of the IM, relative to other resource use, for four common clinical tasks for the month of September 2006.

Fig. 4.

Use of Infobuttons as Compared to Traditional Health Information Resources. Each graph shows one context from the clinical information system, and the number of accesses each month (y-axis) to information resources (light regions are via traditional methods; dark regions are via Infobuttons) from March 2004 to September, 2006 (x-axis). In some contexts, such as Laboratory Results Review, Infobuttons contribute a small percentage of accesses, compared to traditional health information resources. In the other contexts, the Infobutton Manager is the predominant method of access. Note that the bottom two contexts reflect access from the order entry system, which does not generally provide access to on-line resources.

Fig. 5.

Use of Infobuttons as Compared to Traditional Health Information Resources during September 2006. The graph shows the number of accesses each month (y-axis) to information resources for users in each of three contexts from the clinical information system for the month of September, 2006. Note that Infobutton use (light regions) is the predominate access method when users are reviewing in-patient medications but, for some unknown reason, traditional health resource use (dark regions) predominates when users are reviewing outpatient medications.

The numbers above reflect access to on-line information resources. The actual use of the IM was much higher, but did not always result in the user selecting a question. The rate at which question selection occurred varied with the clinical task and the user type. When users were reviewing in-patient drug orders, question selection occurred 63.3%–78.7% of the time (depending on the user type) but was as low as 10.5%–24.3% when reviewing patients' diagnosis lists. By comparison, when users selected the “Health Resource” page (a page of non-context-specific links to on-line resources), they chose resource links 82.6%–93.7% of the time.

Usability, Usefulness, and Impact

As reported previously, users provided feedback via 195 responses to pop-up surveys (a 5.4% response rate) and 73 e-mail surveys (a 5.8% response rate). The IM was judged to be easy to use by 89% (pop-up) and 93% (e-mail) of respondents, with 89% of the e-mail respondents finding that the question they needed was appearing more than half the time and was helpful 90% of the time. Pop-up respondents found the IM helpful 77% of the time, even though it only found the answer to a question 69% of the time.

Attending physicians and nurses were the most positive in their comments, with resident physicians being less so, and students less than residents. There were negative responses to all the questions in each survey, but the most frequent were related to the NYPH Laboratory Manual, which was often off-line when needed.

When asked about impact on patient care, 74% responded that the IM has a positive impact on care. Respondents cited 14 specific cases in which patient care was improved through the use of the IM. For example, one respondent reported:

“A patient with an elevated lab level-I didn't understand what might make it abnormal. The findings suggested to me I should do a biopsy and he is scheduled for that.”

Another reported:

“Providing medication administration information. I was unfamiliar with the route of administration ordered (inhalent versus the typical IV route) and was not comfortable giving it without documented evidence that it was a safe route of administration.”

There were no reports of negative impact on care.

Discussion

Our research on the information needs of clinicians using clinical information systems has found that, like other settings that have been studied, information needs are common and, all too often (∼50% of the time), they are left unresolved. Our analysis of the information needs we observed shows that a large proportion of them can be represented through simple questions and resolved with links to specific on-line information resources.

The IM has been successfully deployed and is providing a substantial number of accesses to on-line health information each month to clinicians in the actual practice of patient care. Users in general find the IM easy to use and helpful. In many settings, they find it to be faster than the traditional methods of information retrieval and use it preferentially over those other methods. Impact on patient care has been positive, with no reported adverse impact or events.

There is still room for improvement. Not all users are reporting that their questions of interest are on the lists produced by the IM. This suggests that further research is needed to study clinicians' information needs. However, when asked to specify questions that they would like, users often suggest ones that are, in fact, on the list, albeit in slightly different forms.

For example, when resident physicians were reviewing laboratory results and selected the IM, they did not select a question 57.8% of the time, suggesting that they had a question (hence, their evocation of the IM) but did not find it on the list. Four of the eleven questions they suggested in surveys dealt with laboratory test normal ranges. This was despite the fact that every time the IM is invoked in the context of laboratory test results review, it includes a link to the NYPH Laboratory Manual; selecting that link produces a page of information about the test, including its normal ranges.

Clearly, more work needs to be done beyond providing the right questions. Education about the use of the IM, including its questions and resources, may help with understanding how to best exploit the capabilities of the IM. It may also help with the rate of adoption, which as shown in Figure 4, is rapid at first and then levels off fairly quickly without being used by a majority of clinical information systems users. We are currently exploring methods to educate, via e-mail reminders, those users who use health resources, but not the IM, while reviewing out-patient medication orders where (as shown in Figure 5), usage is much lower than review of in-patient medication orders.

Revision of the IM user interface may also be needed. This latter suggestion is supported by the fact that while users take only about three seconds to chose a health information resource from the list of resources, they take on average 27 seconds to chose a question from the IM page. We are currently attempting to address the user interface problem by deploying a new, simpler interface (see Figure 6).

Fig. 6.

The Next Version of the Infobutton Manager. See Figure 3 for comparison to the previous version.

While our evaluation methods provide quantitative results reflecting usage, they do not tell us if the IM is improving patient care. The survey results suggest a positive impact, but the low response rate prevents us from generalizing to the entire population of users (and patients) and, in any case, are susceptible to a variety of biases. Other researchers have similarly attempted to address the issue of benefit from context-specific access to on-line information resources (20,21). The number of subjects studied and the accesses provided were fewer than in our study, but their evaluation methods and results were similar: modest usage and anecdotal reports of benefit. While access to on-line health information at the point of care is deemed to be a desirable capability, like the use of other evidence-based materials in patient care, a more penetrating study will be needed to prove that it provides true benefit.

Conclusions

We have successfully provided context-specific access to health knowledge resources at the point of care, based on empirical studies of clinician information needs. The Infobutton Manager has increased the use of these resources by clinicians in actual patient care; in some contexts, this increase has been dramatic, when compared to baseline use of traditional resource access. Further work is needed to improve the ability of the Infobutton Manager to provide the right questions, to teach the users to find and pursue the questions and to determine if answering their questions in a timely fashion leads to better-informed clinicians and better patient care decisions.

DISCUSSION

Mitch: Houston: I was thinking back, and I had my own infobutton, Marshall Wolf, when I was an intern, I wondered if you had looked at who is using this? You had said you had been to wards and so forth.

Cimino: New York: It's about one-quarter attending physicians, about one-quarter nurses, one-quarter housestaff and one-quarter other. They could be technologists, research assistants and so on, and we actually have different statistics broken down. For instance, the nurses use the infobuttons more than the health resources, and the attendings do as well. The housestaff use the health resources more than the infobuttons, that sort of thing.

Mitch: I guess the second question I had is how quickly do you find the interns are able to get into your system and start using it?

Cimino: That's a good question there. You know, we actually did an orientation this year, where I forced them to actually use them as part of an exercise, and there was a big bump in July and August, and it's dropped off again. When I show this to them in clinic, I say, “Well why don't you click on this button?”, and they say, “Wow, this is great.” And then they use it for a week or two, and then they forget. Maybe the navigation is just too tedious for them, but it's a slow adoption process. Definitely, we can do better there.

Kamholz: Manhasset: Have you seen a difference in the information source choices by level of training? Specifically, we are concerned at our place that medical students and junior residents automatically default to predigested sources, like Up To Date, rather than tracking something down through primary sources, like primary journals or things like that.

Cimino: We have the data. I haven't analyzed it that carefully, and the system keeps track of your job title but it doesn't remove the old one when you get a new one. So we have people who are listed as students, residents, attendings, and we don't know necessarily their year. So, it will require more manual analysis to get it. We do have the data, and we can look at that.

Kamholz: Could you change the system so it could have a default list of choices that would force people to use primary sources?

Cimino: Well, it would be interesting to get that through the IRB, I think, because then they would suddenly be research subjects and would have to sign consent forms to allow me to do this. I could certainly do it technically. I also control the health resources list. So, I can put things at the top if I want people to notice them, and they notice them right away when things show up at the top of that list, but I don't restrict their access to particular resources.

Gallin: Bethesda: I would imagine that updating this and keeping it current is going to be a challenge. Have you ever considered using something like Wikipedia where a list of physicians on your team could update this information constantly?

Cimino: I always get this question. I should probably just put a slide in and address it. In terms of updating the information, there are three aspects that I have to depend on. First of all, I have to determine whether the questions change, and I think the questions are sort of timeless: “How do I treat this disease?”, “What is the dose of this drug?”, etc. We do monitor and allow users to tell us what question wasn't on the list. We have to update our question database, but that happens slowly. We have to update the links to the resources, because sometimes the owners of the resources move them around or change them so that the links no longer work. I actually have a program that runs every morning at 4 a.m. and tests all the links to make sure they are still working. But finally, the content is the thing that I don't have any control over and don't have the resources to maintain. So I go to Harrison's and Up To Date and Micromedex, and I have to assume that they are keeping their data up-to-date. In terms of a Wikipedia approach, we actually tried that at Columbia. We had an electronic textbook back in the late 80s, and we had a great chapter on gastroenterology. It had all these different levels and citations and cross links. It was a demo; we never got any of the other chapters done, and it was impossible to maintain. It takes a lot of resources; you have to pay people, to do that. Wikipedia, just letting people volunteer, may not be something you want to depend on.

Gallin: If you got the right people, it might work.

Cimino: Maybe this group would be the right kind of group to do something like that, but, you know, it would be tough. We would have to pay people and say, “You do this section and keep it up-to-date.” So we use Harrison's. We pay Harrison's and Micromedex and all these other publishers to provide us that information, and they keep it up-to-date.

Blantz: La Jolla: Wandering around the wards at University of California, San Diego, I have two concerns, and it isn't directly related to your science, but two issues. Have you defined in younger physicians addictive computer behavior, which sort of makes them sit in front of a computer all day rather than talk to their patients. I guess that's one thing. Are there particular young physicians who seem to be spending more than necessary amounts of time with this type of resource, and what are the consequences? The second, which is a related question, have you ever done a time-in-motion study about how much time our housestaff spends in front of computers already? I don't know if it's just cut and paste or whether it's really a learning process, but it concerns me because even twenty years ago, I'm not sure what the hell we did as residents, you know, because we didn't sit in front of the computer all that time, and I really wonder whether communication with patients is being compromised. I'm not saying it's your fault, but I'm just saying these types of things require validation in terms of whether the real outcome is beneficial.

Cimino: I have not done a global study in my own institution. Other people have studied this issue. I can tell you that our observational data show that clinicians rush to the computer, get what they need and rush back out to do whatever it is they do. Twenty, twenty-five years ago when I was going through my residency, I know what I spent my time doing. I spent it writing, and I am a very slow, horrible writer, and so now I can document much faster on the computer. I spend a lot less time messing around with the paper chart. I spend no time looking for the paper chart, which is what I used to do, or trying to find x-rays or all these other things that I had to track down manually because there was only one physical copy. So the time that I spend in front of computers now is time that I used to spend in the hallways, the elevators, the stairwells, looking for the x-rays that I hid so I would have them for morning report, and things like that. So you know, there is a trade off and I think that my observation is that the housestaff are not spending inordinate amounts of time sitting in front of the computer. They want to get that work done too. People have studied order entry specifically and have shown that housestaff spend a lot more time in the order entry process when it's computerized, at least when it's in start up, but there is a trade off. The trade off is, hopefully, that the orders are better written, more legible, that they are cross-checked, and so on.

Collier: Newark: We have had some real success with wireless notebook tablets, and so our resident teams now carry wireless computers throughout the hospital so that they can actually access at the point of care, which has been something that has been, I think, an advance. Are you all using wireless?

Cimino: We have wireless carts that are used by the nurses, and the housestaff will use them if they need one. First of all, we don't have the resources to provide them all with these tablets. The hospital has chosen to go with desktop.

Collier: We just have one tablet that goes per team.

Cimino: We could. There is no reason we couldn't.