The Contribution of Observational Studies and Clinical Context Information for Guiding the Integration of Infobuttons into Clinical Information Systems

Abstract

The integration of information resources into clinical information systems may be based on a variety of approaches, including what is practical to provide, what is hypothetically useful, and (based on user studies) what is empirically useful. Resource selection may also be based on the user’s context (role, task, patient demographics, concept of interest, etc.).

Methods:

Two years of log files showing the context-specific use of information resources, especially via use of an infobutton manager (IM), were examined for patterns of usage.

Results:

3483 known unique users (and an unknown number of anonymous users) accessed information resources 65,128 times, including the use of the IM 21,907 times by at least 2367 unique users. Links provided in response to empirical information needs accounted for 16% of the total links and 32% of the accesses to information resources. Links that were provided based on patient demographics accounted for 29% of the total links and 13% of the accesses to information resources. Links that were provided based on the users concept of interest accounted for 21% of the total links and 13% of the accesses to information resources.

Conclusion:

Knowledge of observed information needs and of context attributes is useful for guiding automated selection of resource links by an Infobutton Manager.

Introduction

Clinical information system (CIS) developers use a variety of strategies for integrating health knowledge resources (or simply, “resources”) into their systems to address their users’ information needs. One common approach is to provide a static list of links to whatever resources are available. We refer to this as the practical approach.

Recently, CIS developers have begun to provide links to resources deemed relevant, given the user’s context, which can be characterized by attributes such as the user’s role and task, the patient’s age and gender, and the clinical concepts related to the task such as a particular test result or medication.¹ Such dynamic selection of relevant information resources goes by a variety of names, including infobuttons and infobutton managers (IMs).²

The mechanism by which information resources are selected by a CIS for a given context may be through “hardcoding” links to specific resources directly into the CIS or employing an IM to dynamically match context parameters to resources. Ultimately, however, the CIS developers (or their collaborators, such as informaticians and librarians) must decide which resources are to be provided by links and IMs.

A common approach to resource selection is for a developer to hypothesize about the needs that would be appropriate given a particular situation and then select resources from among the available, practical choices. This approach may include input from CIS users, as was done by Del Fiol and colleagues who used focus groups to guide the inclusion of context-resource knowledge in the IM at LDS Hospital.³ Because this method involves hypothesizing about the resources that will be useful in an actual situation, we refer to this as the hypothetical approach.

Many researchers have observed clinicians to identify information needs that arise in actual clinical practice, with the idea that the observations will lead to a better understanding of the resources that should be provided in the patient care setting.⁴ Several information needs studies have conducted at Columbia University in a number of settings⁵ and have been used to inform the addition of links to an IM.⁶ We refer to this as the empirical approach.

While a great deal of work has been carried out to study clinician information needs, to construct solutions (such as infobuttons and IMs), and study the benefits of such solutions, little or no work has been published describing how the use of observational evidence and context knowledge impacts the use of CIS-integrated resources. The present study analyzes log files to better understand such impact.

Background

Clinicians at the Columbia University Medical Center campus, including those at New York Presbyterian Hospital, use two major CISs: WebCIS (developed at Columbia⁷) and Sunrise Clinical Manager (Eclipsys Corporation, Boca Raton, FL). Users of WebCIS (but not of the Eclipsys system) can access a Web page called “Health Resources” (HR) that provides a list of links to a variety of national, local, and commercial resources.

WebCIS users can access the Columbia IM by clicking on “i” icons next to clinical information such as medications and test results. Eclipsys users can access the Columbia IM by clicking on a standard “Item Info” button that is associated with orderable items (laboratory tests, medications or nursing activities). When a user selects an IM link, context information (the medical concept and task and, in the case of WebCIS, the user ID, and the patient age and gender) is passed to the IM.

The HR page contains practical links, independent of the user’s context, based on advice from the Columbia Health Sciences librarians about available resources. Many of the links provided by the IM are based on hypothetical use, predicted by the Columbia infobutton researchers and others at Columbia. For example, the chair of the Department of Medicine believed that links to guidelines for use of drugs such as warfarin and heparin should be provided whenever clinicians were ordering those drugs or reviewing relevant laboratory results. Table 1 shows examples each of these types of links.

Table 1:

Examples of links from various application contexts, including practical (p) links available through the HR page in WebCIS, and hypothetical (h) and empirical (e) links available through the IM in WebCIS and Eclipsys. Some links are selected by the IM based on patient demographics (d) or concept of interest (c).

Application Context	Resource - Topic	Link Type	Comments
Lab Results	CPMC Lab Manual - General Information	e	Observed need for normal ranges and collection instructions
Lab Results	Lab Tests Online - General Information	h	Basic information resource on lab tests
Lab Results	UpToDate - General Information	p,h	Basic information resource on lab tests
Lab Results	Dxplain - Differential Diagnosis	p,h,c	Offered if lab test is related to a finding (e.g., glucose test related to hyperglycemia
Lab Results	Calculator - Anion Gap	h,c	Offered for sodium, pottassium, chloride, bicarbonate tests
Lab Results	PubMed – Toxicity	p,h,c	Offered if test measures a drug
Lab Results	NYPH Guideline - Pediatric Warfarin	h,d,c	Offered if test measures effect of warfarin (prothrombin time or INR) and patient is child
Lab Results	Micormedex - Breast Feeding Restrictions	p,h,d,c	Offered if test measures a drug and patient is child-bearing-age female
Inpatient Drug List	NYPH Guidelines – Antibiotics	h,c	Offered if drug is an antibiotic
Inpatient Drug List	Micromedex - Pregnancy Restrictions	p,h,d	Offered if patient is a child-bearing age female
Inpatient Drug List	Beers Criteria - Drug Dosing	h,d,c	Offered if patient is elderly and drug is in Beers Criteria class
Diagnosis List	Merriam-Webster - Definition	e	Observed need for definitions of diagnosis terms
Outpatient Drug List	Lexi-Comp - Spanish Patient Information	e	Observed need for patient information
Outpatient Drug List	Lexi-Comp - Pill Appearance	e	Observed need to know what pills look like
Outpatient Drug List	Lexi-Comp - Pediatric Information	e,d	Observed need for patient information, offered when patient is a child
Nursing Orders	Eclipsys Documentation - Training	e	Observed need for access to Eclipsys training materials

We have previously carried out observations of clinicians using WebCIS and Eclipsys in a variety of clinical settings.^5,6 Analysis of videotape allowed us to identify information needs that arose and to characterize the contextual factors related to those needs. Using these findings, we added empirical links to the IM that would provide, given the appropriate context, links to resources to address the needs. For example, we found that residents in the medical clinic wondered aloud about the appearance of medications, so we found a resource that contained pill images and instructed the IM to provide a link to the resource whenever a user was ordering or reviewing patient medications.⁵ Table 1 shows examples of such empirical links provided by the IM.

We further noted that specific aspects of the context, besides the general task, could be used to limit the dynamic selection of links. For example, when a user is ordering medications, patient demographic information (age and gender) could be used by the IM to determine whether or not links to resources related to risks of breast feeding and pregnancy should be selected. Similarly, when a user is reviewing laboratory test results, the concept of interest (that is, the specific test) could be used by the IM to determine whether links to resources related to medication dosing and adverse effects (when the test is a drug level) should be provided. The IM uses such additional context information to selecti both theoretical and empirical links. Table 1 shows examples of links that are selected based on demographic and concept information.

Methods

We have previously described the log files available from WebCIS and the IM.⁸ Briefly, WebCIS, the HR page, and the IM record each action carried out by their users. By looking for instances where WebCIS users evoked the HR or IM and then locating the immediately preceding log record for the same user, one can infer the context in which the user evoked the HR or IM. It is therefore possible to reconstruct user activity immediately preceding access to resources, whether through practical, hypothetical or empirical links. Log files were processed to identify all instances of HR, IM, and resource use. Contextual information was retained, where available, including user, application context, and concept of interest.

User job titles are available through the Columbia University Medical Center’s Lightweight Directory Access Protocol (LDAP) Directory. Job titles were characterized as “attending” (physician), “nurse”, “housestaff”, “student” (any type), and “other”. When users with multiple job titles (largely due to the accumulation of previous titles) were mapped to multiple types, they were characterized at the most advanced academic level, so that attending/housestaff were considered attendings, nurse/students were considered nurses, etc. All roles took precedence over “other”, while “attending” and housestaff took precedence over “nurse”.^* IM log file records associated with use of the Eclipsys system do not include user ID. Such records were therefore attributed to an “unknown” user type.

Instances of HR, IM and resource use were compiled for each of the application contexts in which the IM can be evoked. Resource use was characterized with respect to practical (via the HR), empirical (via the IM, based on observed information needs) and hypothetical (via the IM, but not based on observed information needs). IM resource uses were further characterized as to whether they were related to particular aspects of the user’s context (provided by the IM based on patient age, patient gender or concept of interest) or not (always provided by the IM for a given application, regardless of other context parameters). Additional analyses were conducted to identify any interesting patterns of use based on user type, patient demographics, or concept of interest.

Results

Log files from 2007 and 2008 were analyzed. During this period, the HR page was accessed 134,837 times by 4092 users, 2962 of whom used the HR page to access resources 127,276 times. When only the contexts in which the IM was available were considered, 2107 users accessed the HR page 44,541 times, with 1921 of these users (418 attendings, 747 housestaff, 259 nurses, 275 students and 22 others) accessing resources 43,221 times. During the same period, the IM was accessed 40,140 times by 5365 known users (plus additional Eclipsys users), 2367 of whom (433 attendings, 420 housestaff, 436 nurses, 432 students and 645 others) used the IM to access resources 16,771 times, plus an additional 5136 times by unknown users through Eclipsys. A total of 805 known users (178 attendings, 266 housestaff, 123 nurses, 150 students and 88 others) accessed resources by both methods. Thus, at least 3483 unique users made use of resource links 65,128 times, with at least 68% of the users choosing IM-supplied links 34% of the time in applications where both HR and IM were available. Table 2 shows the number of uses of practical, hypothetical and empirical resource links for each of the application contexts in which the IM was available, with breakdown by user type.

Table 2:

Users’ selection of links, by type (p=practical, h=hypothetical, e=empirical), within each application context, by user type. The full table can be found at www.dbmi.columbia.edu/cimino/amia09

Application Context	User Type									Other		Unknown		Total	# Links Avail.	% Links
	Link Type	Attending		Housestaff		Nurses		Student
		#	%	#	%	#	%	#	%	#	%	#	%	#
Lab Results	p h e	10888 1146 389	30.7 17.2 12.8	17290 1000 381	48.7 15.0 12.5	2422 1280 723	6.8 19.2 23.8	2248 1783 672	6.3 26.7 22.1	2644 1458 876	7.4 21.9 28.8	- - -	- - -	35492 6667 3041	26 34 5	40.0 52.3 7.7
Inpatient Drug List	p h e	472 610 172	36.7 22.7 22.1	412 336 33	32.0 12.5 4.2	176 621 139	13.7 23.1 17.9	82 430 196	6.4 16.0 25.2	145 690 237	11.3 25.7 30.5	- - -	- - -	1287 2687 777	26 25 4	47.3 45.5 7.3
Diagnosis List	p h e	430 141 140	28.7 15.4 10.7	170 64 86	11.3 7.0 6.5	445 182 267	29.7 19.9 20.3	25 111 209	1.7 12.2 15.9	428 415 611	28.6 45.5 46.5	- - -	- - -	1498 913 1313	26 6 4	72.2 16.7 11.1
Lab Orders	p h e	- - -	- - -	- - -	- - -	- - -	- - -	- - -	- - -	- - -	- - -	- 318 708	- 100.0 100.0	- 318 708	26 20 2	54.2 41.7 4.2
Total	p h e	13398 2452 713	31.0 16.4 10.3	20316 1685 506	47.0 11.3 7.3	3491 2575 1564	8.1 17.2 22.6	2518 2430 1078	5.8 16.2 15.6	3499 2710 1729	8.1 18.1 24.9	4460 3118 1342	10.3 20.8 19.4	43222 14970 6932	260 169 32	56.4 36.7 6.9

The most common application context was Laboratory Results Review. In this context, 65 links were available: 26 (40%) practical links provided by the HR, 34 (52%) hypothetical links provided by the IM, and 5 (8%) empirical links provided by the IM.

The 34 hypothetical links accounted for 15% of the resources selected, while the 5 empirical links accounted for 7%. Thus, in this context, empirical links were more popular on a per-link basis.

The application context most notable for use of IM links over HR links was the Inpatient Drug List. In this context, 55 links were available: 26 (47%) practical links provided by the HR, 25 (46%) hypothetical links provided by the IM, and 4 (7%) empirical links provided by the IM. The 26 practical links accounted for 27% of the resource selected, while the 25 hypothetical links accounted for 57% and the 4 empirical links accounted for 16%. Thus, in this context, hypothetical and empirical links were more popular, on a per-link basis, than the practical links, with empirical links again having the edge.

One context with a notable use of empirical links was the Lab Orders. In this context, 22 links were available: 20 (91%) hypothetical links and 2 (9%) empirical links (no practical links were available because the HR is not available in this Eclipsys context). The 20 hypothetical links accounted for 31% of the resources selected, while the 2 empirical links accounted for 69%.

Another context with notable use of empirical links was the Diagnosis List. In this context, 36 links were available: the usual 26 (72%) practical links provided by the HR, 6 (17%) hypothetical links provided by the IM, and 4 (11%) empirical links provided by the IM. The 26 practical links accounted for only 40% of the resource selected, while the 6 hypothetical links accounted for 26%, and the 4 empirical links accounted for 35%. Thus, in this context, both hypothetical and empirical links were more popular, on a per-link basis, than the practical links, with empirical links again enjoying the greatest popularity.

Overall, for the thirteen contexts where the IM was available, there were 201 nonunique links, of which 169 (84%) were hypothetical and 32 (16%) were empirical. The hypothetical links accounted for 68% of the IM resource usage while the empirical links accounted for 32%.

As shown in Table 2, patterns of resource selection varied by user type, with attendings and housestaff using practical links most often (81% and 90%, respectively), followed by hypothetical links (15% and 8%, respectively) and relatively little use of empirical links (4% and 2%, respectively), although this trend was markedly reversed in the Inpatient and Outpatient Drug Lists. Nurses and students, meanwhile, made use of practical links much less often (46% and 42%, respectively), with much higher use of hypothetical links (34% and 40, respectively) and empirical links (20% and 18%, respectively), when compared to physicians.

Table 3 shows the number of uses of resource links with respect to the use of patient demographic information (age and gender) by the IM to select resources for the user. Overall, 43 of 201 (21%) of the links were based on patient demographics, accounting for 2785 of 21,902 (13%) of the IM link usage. In the outpatient setting (Outpatient Drug List and Outpatient Drug Orders), the demographics were used to select 10 of 50 (20%) of the IM links but accounted for 519 or 1655 (31%) of links used. There were no usage patterns related to user type.

Table 3:

User of source links based on whether the IM uses patient demographics (d) or not (n) to select them. Full table at www.dbmi.columbia.edu/cimino/amia09.

Application Context	Link Type	Total Srcs Used	% Srcs Used	# Links Avail.	% Links Avail.
Lab Results	n d	9583 128	98.7 1.3	29 10	74.4 25.6
Inpatient Drug List	n d	2506 958	72.3 27.7	21 8	72.4 27.6
Inpatient Drug Orders	n d	2537 849	74.9 25.1	20 7	74.1 25.9
Outpatient Drug List	n d	483 489	49.7 50.3	21 5	80.8 19.2
Outpatient Drug Orders	n d	253 230	52.4 47.6	19 5	79.2 20.8
Total	n d	19122 2785	87.3 12.7	158 43	78.6 21.4

Table 4 shows the number of uses of resource links with respect to the use the concept of interest by the IM to select resources for the user. Overall, 58 of 201 (29%) of the links were based on concept of interest, accounting for 2771 of 21,902 (13%) of the IM link usage. Most usage was related to access to drug resources when the concept of interest was a drug test. There were no patterns related to user type.

Table 4:

Use of source links based on whether the IM uses the concept of interest (c) or not (n) to select them. Full table at www.dbmi.columbia.edu/cimino/amia09.

Application Context	Link Type	Total Srcs Used	% Srcs Used	# Links Avail.	% Links Avail.
Lab Results	n c	7122 2589	73.3 26.7	8 31	20.5 79.5
Inpatient Drug List	n c	3448 16	99.5 0.5	24 5	82.8 17.2
Inpatient Drug Orders	n c	3302 84	97.5 2.5	23 4	85.2 14.8
Total	n c	19136 2771	87.4 12.6	143 58	71.1 28.9

Conclusions

In this study of information resource use by clinicians, knowledge of observed information needs and of relevant contextual attributes appears to have been useful for guiding automated selection of resource links by an Infobutton Manager. This is the first study that correlates observed needs with actual usage and demonstrates that contextual aspects are related to resource selection.