Abstract
Introduction
The development of tools to meet the information needs of clinicians requires an understanding of the clinician and the context in which clinical decisions are being made.
Methods
We conducted an observational study of clinicians’ information needs via think-aloud protocols during which we observed physicians and nurses as they used the clinical information system. Protocol analysis was then used to identify the information needs events, the types of questions that were asked, the method of meeting that need, the success or failure of meeting the specific information need, and the context in which it arose.
Results
Results indicated that a) unmet information needs occur frequently and that b) the predominant feature of these unmet needs is that they are patient-related or domain-specific. Two categories of context: laboratory and medical communication accounted for more than half of the events.
INTRODUCTION
Research seeking to identify the information needs of clinicians has been carried out in a variety of settings including private medical practices, acute care, ambulatory care, and health sciences libraries.1,2,3 This research consistently demonstrates that during encounters with patients, clinicians have frequent unmet information needs. Thus, even though the physicians may state that they have no information needs, observational studies demonstrate clinicians exhibiting information seeking behavior.1
No prior research has examined the information needs of clinicians while they are using a clinical information system (CIS). It is our contention that research seeking to empirically identify the clinical information needs of clinicians at the point of care, while the clinician is using a CIS, will provide important information about the contexts in which clinical questions arise. Further, once an information need and its context are identified, context-specific information resources can be created and tailored to meet the needs of the clinician during interactions with the CIS.
The following observational study was conducted to examine the unmet clinical information needs of attending physicians, housestaff, nurses and medical and nursing students while they used the CIS in a variety of settings in a large academic medical center. We sought to characterize the types of needs, the contexts in which the needs arose, how the users attempted to address their needs, and how successful those attempts were.
METHODS
Physicians, nurses, and medical students and nursing students from three clinical sites (a general medical/surgical nursing station, the coronary care unit (CCU), and a general medical clinic) at the Columbia-Presbyterian campus of New York Presbyterian Hospital were chosen to participate in the study. Users of the CIS at these sites were observed via using a portable usability lab (PUL)4. The PUL uses a videoconverter, a videocassette recorder and an audiocassette recorder to capture audiotape recordings of the clinicians’ voice, as well as videotape of the CIS screen. Informed consent was first obtained and the clinicians were subsequently observed during CIS encounters within the normal course of the day. The PUL was stored away from the clinicians, providing a very unobtrusive method of observation so as to not disrupt the clinicians’ workflow.
Work-sampling methodology was used to sample the clinicians as they carried out their normal practices. This methodology is based on the premise that randomly spaced observations of workers will provide a normally distributed range of events.5,6 As such, the data were collected over a period of four months at random times of the day and during random days of the week including nights and weekends, capturing a representative sample of individuals and experiences. The observation sites were selected based on this method as well as from system logs that identified clinical areas with high utilization of the CIS. Preliminary analysis of the data collected in the medical/surgical area revealed that nurses were not well represented in the initial sample. This was largely due to nurses’ limited use of the CIS in their daily tasks in this area. Examination of the log files indicated a high volume of nurses used the CIS in the CCU, which prompted subsequent data collection in this area.
Once the data were collected, the audiotapes were transcribed and the videotapes were reviewed to identify ‘Information Need Events’. The ‘Information Need Event’ was the unit of analysis for this study because, as has been demonstrated previously, the individual will usually express more than one information need for any given patient encounter.1,7 The Information Need Event was also characterized as the unit of analysis because the context-specific solution will address the Information Need Event not only in the context of the individual, but more importantly, in the context of CIS environment.8
A coding schema that accurately characterized the Information Need Events was developed based on theories related to evidence-based medicine, information seeking behavior, clinical decision making, and medical error literature.2,3,9,10 The classification schema included: 1) Type of Event; 2) Type of Resource Used; 3) Type of Outcome; and 4) Context. The development of the coding schema is discussed in an accompanying paper (Graham M, et al. In press).11 See Table 1 for additional descriptions of these codes.
Table 1.
Information Need Event Coding Categories
| Information Need Event Type | |
| Subject (Patient) | Related to subject (patient) |
| Institution | Related to particular place of practice |
| Domain | Related to medical or nursing domain |
| Subject-Institution | Intersection of subject and institution |
| Subject-Domain | Intersection of subject and domain |
| Domain-Institution | Intersection of domain and institution |
| Subject-Domain-Institution | Intersection of subject, domain and institution |
| Foreground | Subject (patient) management question |
| Background | Root question - (e.g. who? what?) |
| Explicit | Clinician verbally expresses an information need |
| Implicit | Clinician uses a resource other than CIS |
| Resource Type | |
| Computer | Computer based resource used |
| Human | Other clinician or human resource used |
| Paper | Paper resource e.g. paper chart or book |
| Outcome Type | |
| Success | Clinician sought & found information |
| Deferred | Clinician did not seek an answer |
| Failure | The clinician sought, but did not find desired information |
| Context | |
| Laboratory | All laboratory reports available on the CIS |
| Patient Report | Procedure report or other reports available on the CIS |
| Communication | Medical communication via the CIS e.g. Progress notes, medication list |
| Non CIS Computer | Computer based resource not accessible via the CIS e.g. Standards of Care |
| Generic CIS | General CIS areas such as physician’s patient list or department list |
| Non Computer Resource | Paper chart, conversation (telephone or in person) |
| Miscellaneous | Computer applications, not directly related to domain or patient information |
The context in which the events arose was defined as the part of the CIS in which the clinician was located when the information need was identified. As such, there were 38 categories such as a specific laboratory result (e.g. calcium level, chemistry) or a specific information resource that was used during the information need events (e.g. oncology reference book). These 38 parameters were sorted into the seven contextual dimensions also listed in Table 1.
RESULTS
Fifteen hours and twenty-nine minutes of audiotape and videotape were recorded while clinicians were using the CIS. One hundred and fifty four information needs events were identified, an average of eleven events per hour of tape. The recording time represented only the time when the users were actually using the CIS. The PUL was turned off when the CIS was not in use or when asked to do so by the participant.
The coding schema was tested for reliability by three of the authors (LC, MG, JC) who independently coded the events. A subset of 30 events was initially coded and disagreement was resolved through discussion. Inter-rater reliability for the initial subset was 93.3%. When the remaining events were coded, reliability increased to 97%.
Information Needs Events
Thirty-five clinicians participated in the study; 58% (19) were female. Nurses represented 22% (8) of the clinicians. The number of events per clinician varied, from one to twenty-five events per clinician depending on the task and immediacy of the information need.
Seventeen (49%) of the 35 clinicians demonstrated only one information need event, 14 (40%) of the clinicians demonstrated between two and seven events, and 4 (11%) of the clinicians experienced a ‘high volume’ of events - between 17 and 25 information need events. This meant that four clinicians experienced 73 of the 154 (48%) information needs events. None of these clinicians was a nurse. The 14 clinicians who experienced between two and seven events represented 31% of the events. The largest group, those with a single information need event, experienced only 11% of the 154 information needs events.
Two of the ‘high volume’ individuals were housestaff using the CIS in the clinic where they were observed for a two-hour period each. Twenty-four information needs events were identified for each of these clinicians during the two-hour observation period. One of these housestaff physicians saw no patients during the observation period but he was actively engaged in using the CIS to update his patients’ pharmacy records. The other clinician saw six patients in the two hours and used the CIS while engaging in conversation with his patients.
The two medical/surgical unit ‘high volume’ clinicians were one housestaff physician and one attending physician. Each individual was in the process of patient data collection and analysis and in which their CIS interaction had the objective of accurately diagnosing and planning care for several patients.
More specifically, these physicians were observed during dynamic interchanges with other clinicians throughout which time several information needs were verbalized. Both of these ‘high volume’ clinicians were engaged in using the CIS for relatively long periods of time. The clinician that elicited 25 events used the system for 35 minutes while engaged in a dynamic discussion with two or three other housestaff and the attending that elicited 17 events used the system intermittently over one hour and 45 minutes.
Those clinicians who expressed between one and seven information need events were also using the CIS primarily to diagnose and plan patient care. The varied intensities of data collected from the clinicians is consistent with work-sampling methodology.
Information Needs Event Types
The types of information need events, the types of resources used, and the types of outcomes are displayed in Table 3 at the bottom of this page.
Table 3.
Information Needs Events by User Type
| Subject | Institution | Domain | Subject Domain | Subject Institution | Institution Domain | Institution- Domain-Subject | |
|---|---|---|---|---|---|---|---|
| Housestaff | 48 | 19 | 29 | 2 | 7 | 4 | 1 |
| Nurse | 5 | 8 | 2 | -- | 4 | -- | -- |
| Attending | 8 | 3 | 5 | 2 | -- | -- | -- |
| Student | 1 | 5 | 1 | -- | -- | -- | -- |
| Total | 62 | 35 | 37 | 4 | 11 | 4 | 1 |
| Background Question | Foreground Question | Explicit | Implicit | Success | Deferred or Failed | Computer | Human or Paper | |
|---|---|---|---|---|---|---|---|---|
| Housestaff | 59 | 51 | 92 | 18 | 58 | 52 | 61 | 49 |
| Nurse | 13 | 6 | 16 | 3 | 7 | 12 | 9 | 10 |
| Attending | 12 | 6 | 16 | 2 | 7 | 11 | 4 | 13 |
| Student | 5 | 2 | 7 | 0 | 3 | 4 | 3 | 4 |
| Total | 89 | 65 | 131 | 23 | 75 | 79 | 77 | 77 |
Types of Events by Group
Information needs related to the Subject (patient) were almost twice as likely than Domain or Institutional information needs. Domain-specific and Institutional information needs were almost equally represented in this sample. The ratios of Domain and Institutional needs to Subject related needs were consistent between housestaff and attendings (1.6:1.66 and 2.5:2.66), however, the nurse group had a proportionately higher number of institutional events.
Foreground and Background questions were equally distributed amongst the housestaff, but Background questions were twice as likely to occur in the other groups (nurses, attendings and students).
All 12 of the Background question events were derived from the ‘high volume’ attending physicians’ exchanges. The majority of the events were explicit; only 15% of the questions for the groups were implicit.
Types of Outcomes and Resources by Group
In general, the Information Needs Events were likely to be successful (49%). However, when the number of successful events was examined by grouping, the nurses, attendings and students were only about half as likely to succeed than the housestaff. One example of a paper information resource that was used was a textbook on clinical oncology that was used by a housestaff physician to prepare for the presentation of patient data to his professors. Examples of computer resources are patient clinical information on the CIS and health information resources external to the CIS.
Events by Question Type, Resource & Outcome
Cross-tabulations of the event codes were constructed to compare the numbers of events by question type, resource used and outcome identified. The important grouping results are displayed above in Table 4. The Subject related events in which the information resource was a Human were more successful than those in which the resource was a Computer. Many of the unsuccessful information needs events sought out using a computer were deferred. Three of the four Subject-Domain intersection events were successful, seven of the eleven Subject-Institution events were successful, two of the four domain-Institutional intersections were successful and the Subject-Domain-Institution (SDI) intersection item was also successful. This SDI item was a foreground question that used a computer resource.
Table 4.
Events by Type, Resource & Outcome
| Type/Resource | Outcome | |||
|---|---|---|---|---|
| Success | Deferred | Failed | ||
| Subject/Human | Back | 10 | 3 | 1 |
| Fore | 3 | 3 | 2 | |
| Total | 13 | 6 | 3 | |
| Subject/Computer | Back | 7 | 7 | 3 |
| Fore | 5 | 6 | 6 | |
| Total | 12 | 13 | 9 | |
| Subject/Paper | Back | 1 | 1 | 0 |
| Fore | 2 | 2 | 0 | |
| Total | 3 | 3 | 0 | |
| Domain/Human | Back | 3 | 5 | 4 |
| Fore | 1 | 0 | 0 | |
| Total | 4 | 5 | 9 | |
| Domain/Computer | Back | 5 | 3 | 1 |
| Fore | 5 | 3 | 1 | |
| Total | 10 | 6 | 2 | |
| Domain/Paper | Back | 0 | 0 | 0 |
| Fore | 2 | 1 | 3 | |
| Total | 2 | 1 | 3 | |
| Institution/Human | Back | 10 | 1 | 3 |
| Fore | 6 | 0 | 0 | |
| Total | 16 | 1 | 3 | |
| Institution/Computer | Back | 2 | 0 | 1 |
| Fore | 0 | 2 | 4 | |
| Total | 2 | 4 | 5 | |
| Institution/Paper | Back | 1 | 1 | 1 |
| Fore | 1 | 0 | 0 | |
| Total | 2 | 1 | 1 | |
Back = Background Question; Fore = Foreground Question
Contextual Categorization
The results of the contextual categorization were as follows: Lab 43 (27%); Medical Communication 42 (25%); Non-Computer Resource 31 (21%); Reports 11 (9%); Computer but not CIS 10 (7%); CIS General (6%); and Miscellaneous 8 (5%).
Context by User Type
Medical Communication was the most frequently observed context in which an information need was identified for the sample with all events occurring with the housestaff group. The second most frequently occurring context by user type was Non-Computer Resource, which included the paper chart, telephone conversation, and books at a rate of five events for each category. The Miscellaneous context in which the nurses’ Information Needs Events occurred was in the use of a new administrative software application that allowed the users to view physicians’ institutional privileges.
Context by Event Type
Of the three most frequently occurring contexts (Lab, Medical Communication and Non-Computer Resource represented 73% of events), event types were primarily Subject related and Explicit. Foreground and Background questions were equally likely for these three groups as was the likelihood of succeeding or failing (Success: Lab 50%; Medical Communication 49%; Non-Computer Resource 44%).
DISCUSSION
The use of observational methods and think-aloud protocol analysis has proven to be a valid and useful method to identify clinical information needs while clinicians were using the CIS. Our research has demonstrated that unmet clinical information needs persist across care areas at a rate between eight and thirteen events per hour. When the number of interactions with a CIS in the hospital environment is taken into consideration, the volume of information needs events in any given hospital is staggering to imagine.
We demonstrated that Subject related information needs are the most commonly occurring types of information needs and that 51% of these information needs went unmet. The number of Background questions in which a Human resource was utilized was 76% versus 54% in which a Computer resource was used. This suggests that human interaction was used to meet background information needs more frequently than a computer resource.
Although relatively small, the sample of 35 clinicians was a rich source of Information Need Events. Observation time for any individual ranged from one minute to two hours. One attending physician was responsible for all 12 of the Attending physician Background questions and less than half of these Background questions were successful, but if they were successful, the resource was most likely a Human. This clinician was observed using the CIS while having a number of patient related telephone conversations. Dynamic inter-collegial interchanges such as this via the telephone or in person provided the richest expressions of information needs. Given individual variation in ability to think aloud, others may also find dialogues such as these useful in information needs studies.
Despite efforts to capture the information needs of nurses, this group was underrepresented when compared to physicians in this sample. Efforts to increase nurse representation in the sample improved the numbers, but not sufficiently to capture nurse information needs in the manner that physician information needs were identified. Our results demonstrated that nurses were more likely to have Institutional types of information needs as compared to housestaff. This may in part be related to the limited number of nurse-focused functions in the existing CIS or to the type of nurse user events that our sample captured. On the other hand, this data may be the explained by the types of information needs that are representative of these particular groups in this institution. Nurses information needs may arise in a non-CIS context.
Our coding strategies facilitated capture of the context specific nature of individual interactions with the CIS and inter-collegial interchanges in which the CIS was used, thus providing guidance for the development of information resources tailored to a variety of contexts (i.e., infobuttons).
Frequency of Events by Context and by User Type
Table 2.
Relationship Between Audio-Video Time and Information Needs Events
| Area | Minutes | # Events | Events/Hr |
|---|---|---|---|
| Medical/Surgical | 535 (58%) | 71 (46%) | 8 |
| Medical Clinic | 219 (24%) | 48 (31%) | 13 |
| Coronary Care | 175 (19%) | 35 (23%) | 12 |
| Total | 929 | 154 | Avg 11 |
Acknowledgments
This project was supported in part by National Library of Medicine Training Grant LM07079-11 and by National Library of Medicine Grant 1R01LM07593.
REFERENCES
- 1.Covell DG, Uman GC, Manning PR. Information needs in office practice: Are they being met? Annals of Internal Medicine. 1985;103:596–599. doi: 10.7326/0003-4819-103-4-596. [DOI] [PubMed] [Google Scholar]
- 2.Corcoran-Perry S, Graves J. Supplemental information seeking behavior of cardiovascular nursing. Research in Nursing and Health. 1990;13:119–21. doi: 10.1002/nur.4770130208. [DOI] [PubMed] [Google Scholar]
- 3.Krikelas J. Information Seeking Behavior: Patterns and Concepts. Drexel Library Quarterly. 1983;19:5–20. [Google Scholar]
- 4.Cimino JJ, Patel VL, Kushniruk AW. Studying the human-computer-terminology interface. Journal of the American Medical Informatics Association. 2001;8(2):163–73. doi: 10.1136/jamia.2001.0080163. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Sittig DF. Work-Sampling: A statistical approach to evaluation of the effect of computers of work patterns in healthcare. Methods of Information in Medicine. 1993;32:167–174. [PubMed] [Google Scholar]
- 6.Pelletier D, Duffield C. Work sampling: Valuable methodology to define nursing practice patterns. Nursing and Health Sciences. 2003;5:31–38. doi: 10.1046/j.1442-2018.2003.00132.x. [DOI] [PubMed] [Google Scholar]
- 7.Gorman P. Information needs in primary care: a survey of rural and nonrural primary care physicians. Medinfo. 2001;10(Pt 1):338–42. [PubMed] [Google Scholar]
- 8.Cimino JJ, Li J, Bakken S, Patel VL. Theoretical, empirical and practical approaches to resolving the unmet information needs of clinical information system users. Proceedings of the American Medical Informatics Association Symposium. 2002:170–4. [PMC free article] [PubMed] [Google Scholar]
- 9.Sackett DL, Richardson WS, Rosenberg W. and Haynes R.B. (1997). Evidence-based medicine: how to practice and teach EBM. New York: Churchill Livingstone Inc.
- 10.Leape LL. Error in Medicine. Journal of the American Medical Association. 1995;272(23):1981–1857. [Google Scholar]
- 11.Graham M, Currie LM, Allen M, Bakken S, Patel VL, Cimino JJ. Characterizing information needs and cognitive processes during CIS use. (2003) AMIA Fall Symposium (In press). [PMC free article] [PubMed]