A comparison of clinicians' access to online knowledge resources using two types of information retrieval applications in an academic hospital setting

Abstract

Objective:

The research studied whether a clinician's preference for online health knowledge resources varied with the use of two applications that were designed for information retrieval in an academic hospital setting.

Methods:

The researchers analyzed a year's worth of computer log files to study differences in the ways that four clinician groups (attending physicians, housestaff physicians, nurse practitioners, and nurses) sought information using two types of information retrieval applications (health resource links or Infobutton icons) across nine resources while they reviewed patients' laboratory results.

Results:

From a set of 14,979 observations, the authors found statistically significant differences among the 4 clinician groups for accessing resources using the health resources application (P<0.001) but not for the Infobuttons application (P = 0.31). For the health resources application, the preferences of the 4 clinical groups varied according to the specific resources examined (all P≤0.02).

Conclusion:

The information-seeking behavior of clinicians may vary in relation to their role and the way in which the information is presented. Studying these behaviors can provide valuable insights to those tasked with maintaining information retrieval systems' links to appropriate online knowledge resources.

Highlights.

• An in-depth analysis of computer log files provides an opportunity to better understand clinicians' information-seeking behaviors and the clinical setting where information-seeking activities occur.

• The researchers' analysis of one set of such log files shows that clinicians pursue answers to their information needs differently depending on their tasks and clinical roles.

Implications.

• Information retrieval applications that ease clinicians' access to appropriate online health knowledge resources lower barriers to information seeking.

• Information retrieval application activity log files capture rich data about clinicians' information-seeking events.

• An understanding of clinicians' preferences for certain online knowledge vehicles at the point of care is important to medical librarians in their role supporting medical teams in clinical decision making.

• Increased knowledge about clinicians' information-seeking activities is valuable to those who are tasked with decreasing clinicians' barriers to effective information seeking.

INTRODUCTION

The high frequency of clinicians' information needs in everyday practice has been well documented 1. While computer-based knowledge resources are useful for addressing these needs, ineffective search skills and perceived lack of time are common barriers to information seeking 2. These barriers can cause clinicians to defer decisions or to make them with incomplete information 3–5. One increasingly popular approach to lowering these barriers is implementing information retrieval applications to ease clinicians' access to appropriate online health knowledge resources 6.

Currently, clinicians at Columbia University and the New York Presbyterian Hospital have access to two types of information retrieval applications that are designed to aid information-seeking activities at the point of care 7. One, called Health Resources (HR), is always available on the clinical information system main menu and provides users with a list of static links to more than twenty-five different resources 8. The other information retrieval application, the Infobuttons (IBs), is available through icons adjacent to specific clinical concepts (e.g., laboratory test results and medications) 9. The IBs were initially “hard-wired” into systems to provide novel, although limited, direct connections to an online bibliographic database 10. However, during the course of refining the functionality of IBs, a need for a more sophisticated application that can manage complex clinical questions was identified from observational studies 11. Therefore, the function of the IBs was changed from a direct link to a single resource to an “infobutton manger” that provided a set of links to different resources. Unlike the HR, with its long list of links to the home pages of resources, the IB manager provides a limited set of links that are customized to retrieve information relevant to the particular concept of interest from the electronic health record. This reduces the workload of the clinician by presenting a list of appropriate links, in the right place, at the right time 7,9. The updated IB automatically gathers information about the clinician, patient, and clinical task on hand when a clinician initiates an information-seeking activity. Then, the application matches those parameters against a knowledge database that contains information about resources and criteria to determine which resource links to present to the user 7. Figure 1 demonstrates the typical workflow of an end user when interacting with these applications.

Figure 1.

Information retrieval approaches offered at New York Presbyterian Hospital

The knowledgebase used by the IB manager is created and maintained by those persons responsible for providing knowledge resources to clinicians. This typically involves medical librarians and system developers who work together to decide which resources are best for a given situation. End-user surveys and computer log file analysis with descriptive statistics are often used to evaluate the usability of information retrieval applications. However, those approaches may provide a piecemeal understanding about the usefulness of the application 9. For example, one evaluation of an IB manager with similar approaches yielded mixed findings with nearly 3 out of 4 users reporting being satisfied but then accessing HR almost 3 times more than IB (116,214 HR observations vs. 30,374 IB observations, respectively) 11. Also, evaluation and maintenance activities are important to update the preferred online health knowledge resources and directives found in the IB knowledge database. If the knowledge is not kept up to date, then it will eventually fall behind clinical practice trends, leading to end users abandoning the application altogether. In response to these challenges, a call for medical librarians' involvement in developing clinical systems to meet well-documented information needs is on the rise 12. In fact, scientists at Columbia University and University of Utah have formed partnerships with the National Library of Medicine to support the ongoing work of maintaining information retrieval applications 13.

As information technology–based initiatives in health care continue to evolve to match the needs of the end users, so should evaluation approaches that are used to determine whether the application meets the needs of the end users and what can be done to improve them. The authors' intention for this paper was to find a cost-effective and rigorous way to better understand the determinants of clinical information-seeking behaviors at the point of care. Therefore, in this paper, we present a preliminary study for an advanced understanding of whether clinicians' roles influence their choices for certain online health knowledge resources in a given clinical context. We performed a statistical comparison of observed frequencies resulting from two applications that the clinicians used to access online health resources. We think that our findings will benefit those who are tasked with facilitating clinicians with information-seeking tasks. To the best of our knowledge, a statistical comparison of clinicians' preferences for online health knowledge resources through information retrieval applications has not been implemented.

METHODS

Data source

We studied the information-seeking activity of clinicians at the Columbia University Medical Center, where an electronic medical record system called WebCIS has been in use since 1994 14. All user actions in WebCIS, including the selection of the HR links and IB icons, are recorded in a log file. The log file can therefore show not only who evoked the HR links or IB icons, but what action they performed immediately prior to the evocation and what resource (if any) they selected from the lists of preferred links.*

Inclusion criteria

As we aimed to apply a statistical approach to study computer log file records, we searched for a log file dataset that contained a large number of HR and IB uses for a generic clinical task. This approach helped us to normalize differences in clinicians' roles and responsibilities, such as the authority to prescribe medications, that otherwise would have resulted in large differences in usage. We used a dataset of WebCIS log file records that was made available for research in 2008 with approval of the Columbia University Institutional Review Board. The data set excludes patient identifiers; we further removed user identifiers after characterizing the user type. We restricted the data set to those records related to reviewing laboratory test results, as previous studies have shown this to be the context in which information resources are most often accessed. From this log file, we selected those records that indicated that users had reviewed laboratory results and then selected either HR or IB. We further limited log file records to the five most frequently used IB resources and five most frequently used HR resources in which the user type was one of four clinical roles of interest: attending physicians (“attendings”), housestaff physicians (“housestaff”), nurse practitioners, and nurses. We considered other user groups to be either too heterogeneous (e.g., “student” and “other”) or too small (e.g., physician assistants) to provide sufficient data for analysis. Altogether, nine resources were identified, one of which was accessible from both applications (Table 1).

Table 1.

The most often selected resources through Infobutton (IB) and Health Resource (HR) links

Statistical analysis

Because one uniquely identified user could access multiple resources in single or multiple sessions, we addressed issues of repeated measures by translating raw data into percentages. For both HR and IB contexts, we separately calculated, for each user, the percent of use for each of the 5 most popular resources for that context. Using the percentages allowed each user to contribute equally to the data. For example, if a user used Resource A 2 times, used Resource B 3 times, and did not use Resources C, D, or E, then this user's percentages of use would be: A, 40%; B, 60%; C, 0%; D, 0%, and E, 0%.

For both HR and IB, we separately performed a one-way multivariate analysis of variance (ANOVA) using the 5 resource percentages as dependent variables and the clinician roles as a categorical independent variable. Wilks' lambda test statistic was used to determine if the four clinician roles differed in their distribution of the five resources. We then conducted an ANOVA for each separate resource to see if the percent of resource used differed among the four clinician roles. If statistically significant, a Tukey-Kramer post hoc test was employed for pairwise comparisons among the clinician roles. All analyses used 2-sided P-values at an alpha of 0.05. We used SAS software version 9.2, SAS Institute, Cary, North Carolina.

RESULTS

Out of 73,472 log file observations that were recorded in 2008, 14,979 observations from 1,430 unique users matched the desired criteria for analysis. From those, HR had 13,034 observations and IB had 1,945 observations. Of those, 181 of the observations were produced by overlapping users (83 housestaff, 66 attendings, 18 nurse practitioners, and 14 nurses, respectively). The top IB and HR resources used were the Columbia Presbyterian Medical Center (CPMC) Lab Manual (IB), National Guidelines Clearinghouse (IB), Lab Tests Online (IB), Harrison's (IB), UpToDate (IB and HR), MEDLINE (HR), Micromedex (HR), Lexicomp (HR), and Infectious Disease References (HR).

For IB analysis, there were 513 unique users, with 192 attendings having the highest number of observations (695/1,945, 35.73%), followed by 108 nurses (541/1,945, 27.81%), 150 housestaff (498/1,945, 25.60%), and 63 nurse practitioners (211/1,945, 10.85%). The percentage use of the 5 resources did not vary significantly among clinician user types (P = 0.31), and, within each resource, the percentages of use by clinician user groups were not significantly different (all P>0.10) (Figure 2).

Figure 2.

Infobutton laboratory results review context resource means by user type

For HR analysis, there were 917 unique users, with 510 housestaff having the highest number of observations (7,453/13,034, 57.18%) followed by 276 attendings (4,520/13,034, 34.68%), 67 nurse practitioners (830/13,034, 6.37%), and 64 nurses (231/13,034, 1.77%). The use of each of the top 5 resources differed significantly among the clinician user types (P<0.001) (Figure 3). For the Infectious Disease Reference, housestaff use was significantly higher than that of nurse practitioners, attendings, and nurses (P<0.001). For Lexicomp, use by nurses was significantly higher than that of attendings (P = 0.0225). For MEDLINE, use by nurses was significantly higher that of nurse practitioners, attendings, and housestaff (all P<0.0001). For Micromedex, use by attendings was significantly higher than that of housestaff (P  = 0.0014). Finally, for UpToDate, use by nurse practitioners and attendings was significantly higher than that of housestaff (P = 0.0008 and P<0.0001, respectively); use by nurse practitioners was higher than that of nurses (P = 0.0469); and use by attendings was also significantly higher than that of nurses (P = 0.0256) (Figure 3).

Figure 3.

Health Resources laboratory results review context resource means by user type

DISCUSSION

We have presented a reproducible, statistical approach to determine methods for identifying role-specific preferences for information seeking across 1,430 clinicians, a number much greater than would be possible with an observational study. In applying these methods to users at 1 institution, we determined that users are not inherently always different and that they tend to seek information in ways that are similar to their peers in certain contexts. For the HR application, housestaff behavior sometimes differs from that of attendings (for example, housestaff used the Infectious Disease Reference nearly 5 times more often, while attendings preferred Micromedex and UpToDate), and nurse practitioner behavior is sometimes more like their physician colleagues' (preferring MEDLINE less and UpToDate more) than their fellow nurses.

One finding from this analysis is that the clinicians' role is an important attribute in information-seeking activity. Another finding is that the task context does not directly predict the type of resource that users require. Indeed, in the laboratory context, only three out of nine resources (CPMC Lab Manual, Infectious Disease References, and Lab Tests Online) were specific to laboratory testing. Taken together, these findings indicate that simple predictions of information needs may be inaccurate. While a complex cognitive model might improve predictions, our empiric approach (that is, to simply observe which users select which resources in which contexts) provides an alternative, practical, and immediate solution.

We did not examine the relationship between ordering applications on the HR and IB screens. While such ordering does have an impact on application selection 7, it could not account for the differences seen between user roles, because ordering was not based on role. However, the expertise of medical librarians combined with rigorous analysis methods could lead to more informed decision making on ordering the resources accessible by information retrieval applications, which would ultimately lead to more stable application usage activity. For example, medical librarians' knowledge about clinicians' information needs could be correlated with findings from rigorous log file analysis when updating application links to health resources. Also, if an institution is equipped with multimodal information retrieval applications, any changes to accessing online health resources can be piloted in simpler applications to determine whether they produce a viable increase in usage frequencies before they are implemented into more sophisticated ones. Because medical librarians are well known for their work in assisting clinicians with forming questions and seeking answers 12, their role in designing and developing information technology–mediated tools, as explained above, would help to reduce clinicians' barriers to information seeking. Altogether, stronger evaluation methods would support the US Office of the National Coordinator on Health Information Technology's recommendations to add decision support aids such as infobuttons 15.

Limitations

Our study was not an attempt to determine the rate at which information needs occur, because this would require calculating a “denominator” that depended on knowing the total number of users who were clinically active at any given time and how active they were (number of patients, severity of patients' conditions, primary care or consulting role, etc.). Moreover, generalizability of our findings is low because we studied recorded observations from one clinical context and had only one overlapping resource (UpToDate), which limited our ability to drill down further and perform additional comparisons by resource types and different clinical contexts. However, we propose that empirical analyses like the one reported here would complement conventional evaluation approaches. Even though we used a data set collected in 2008 that had been made available specifically for this type of research, it is possible that the types of resources chosen by clinicians have changed over the intervening years. In fact, the possibility of year-to-year changes supports our contention that stronger evaluation methods are needed on an ongoing basis. Because the differences in clinicians' information-seeking activities may change over time, these differences will continue to require scrutiny.

CONCLUSION

Context-specific information retrieval systems occupy a unique position in bridging the knowledge gap in health care. Our methods for analyzing infobutton and health resource log files are applicable to other settings where developers and librarians may seek to understand how personal and environmental factors influence information seeking at the point of care. We found that clinicians' roles and the context in which information is offered sometimes influences their preferences for accessing online health knowledge resources in a given task and should be taken into account when matching information needs to information resources.