Abstract
Purpose
To understand how much time residents spend using computers as compared with other activities, and what residents use computers for.
Method
This time and motion study was conducted in June and July 2010 at NewYork-Presbyterian/Columbia University Medical Center with seven residents (first-, second-, and third-year) on the general medicine service. An experienced observer shadowed residents during a single day shift, captured all their activities using an iPad application, and took field notes. The activities were captured using a validated taxonomy of clinical activities, expanded to describe computer-based activities with a greater level of detail.
Results
Residents spent 364.5 minutes (50.6%) of their shift time using computers, compared with 67.8 minutes (9.4%) interacting with patients. In addition, they spent 292.3 minutes (40.6%) talking with others in person, 186.0 minutes (25.8%) handling paper notes, 79.7 minutes (11.1%) in rounds, 80.0 minutes (11.1%) walking or waiting, and 54.0 minutes (7.5%) talking on the phone. Residents spent 685 minutes (59.6%) multitasking. Computer-based documentation activities amounted to 189.9 minutes (52.1%) of all computer-based activities time, with 128.7 minutes (35.3%) spent writing notes and 27.3 minutes (7.5%) reading notes composed by others.
Conclusions
The study showed residents spent considerably more time interacting with computers (over 50% of their shift time), than in direct contact with patients (less than 10% of their shift time). Some of this may be due to an increasing reliance on computing systems for access to patient data, further exacerbated by inefficiencies in the design of the electronic health record.
With the increasing focus on the dissemination of health information technology (HIT) and the electronic health record (EHR), questions regarding the impact of these technologies on clinical practice become of paramount importance. Studies conducted thus far outline both the benefits of HIT and EHRs, such as reduction in medication errors and improved patient outcomes,1,2 and their unintended consequences, such as reduced efficiency, lower quality of care, increased possibility of medical errors,3,4 and disruptions in clinicians' workflows, particularly if the design of the HIT or EHR does not match clinical work practices.5
Despite the growing importance of HIT and EHRs, the actual patterns of clinicians' use of these technologies remain poorly understood. Many studies do not discriminate between different types of tasks clinicians perform using electronic systems or only focus on particular types of HIT, such as computerized provider order entry3,6,7 and electronic documentation.8,9 However, the rich functionality of modern EHRs suggests that a wide variety of tasks related to patient care may now be performed using computing systems. Moreover, new generations of clinicians increasingly rely on computing and information technologies in different aspects of their lives. The question then becomes how these increasing expectations and experiences with technologies change the way clinicians practice medicine, engage with patients, and carry out their professional duties.
Previous studies of resident workflows suggested physicians in training spend little time on direct patient care and the majority of their time on educational and administrative tasks.10,11 These observations remained consistent for several decades,12,13 leading to a standing concern about residents' time allocation. The introduction of HIT and EHRs inspired an ongoing debate in the medical community not only about their benefits and limitations,14,15 but also about their impact on how residents spend their time, particularly with regard to the proportion of time residents spend on direct patient care.10 The study reported here contributes to this discussion by investigating the use of computing and information technologies by residents in inpatient care settings. The goal of the study was to understand not only how much time residents spend using computers as compared with other activities, but also what residents use computers for, thus providing a snapshot of the role of HIT in modern patient care. To accomplish this, we conducted a time and motion study of medical residents at a large urban teaching hospital. To add to the previous research in this area, the study expanded on a validated taxonomy of clinical activities by introducing a set of fine-grained categories describing computer use that allowed us to examine these activities with an unprecedented level of detail.
Method
Empirical setting
We conducted this time and motion study in June and July 2010 at NewYork-Presbyterian/Columbia University Medical Center (NYP/CUMC), a large urban teaching hospital in New York, New York. NYP/CUMC had over 2,300 beds and discharged over 110,000 patients in 2009, with an average length of stay of 6.4 days.16
We conducted this study with residents (first-, second-, and third-year) on the general medicine service. Patient care on the general medicine service was practiced in teams; the physicians on each team included an attending physician, a fellow, a second- or third-year resident, a first-year resident (intern), and a medical student.
In 2004, NYP/CUMC deployed a commercial EHR system (Allscripts Sunrise, Allscripts, Chicago, Illinois). Before that, NYP/CUMC used WebCIS, which was developed in-house. The Allscripts Sunrise EHR system included a number of modules, separated into tabs (e.g., results, flowsheets, and orders). Most licensed independent practitioners entered their notes directly into the EHR via a keyboard and mouse as opposed to using dictation.
In addition to the EHR, all residents had two-way pagers, which were used as the main means of communication on the general medicine service. No work-related mobile or handheld devices were actively used during the study.
Subjects
At the time of the study, 16 residents rotated on the general medicine service (8 second- or third-year residents and 8 interns). Four (25.0%) of these residents participated in a pilot study, conducted immediately prior to the study described here. Another 7 (43.8%) participated in the final data collection study—3 (42.9%) of these participants were interns, who began their first rotations 3–4 weeks prior to the study, and 4 (57.1%) were second- and third-year residents. The 7 participants were randomly selected from the pool of rotating residents with the assistance of chief residents. All residents who were approached agreed to participate in the study, and there were no withdrawals. Each participant was observed for the entire duration of one of his or her shifts. The participants were introduced to the purpose of the study (time and motion study of their activities); there was no compensation for participating.
Study design
This study used a time and motion design to allow us to capture data on clinical activities with a high level of detail.10 During the course of the study, an experienced observer (L.M.) shadowed each participant for the 7–14 hours of their day shift (typically starting between 7:00 a.m. and 8:00 a.m. and ending between 3:00 p.m. and 9:00 p.m.). We did not include night shifts in the study. While shadowing participants, the observer captured all their activities using a custom-developed iPad application. Notably, the application allowed us to capture multiple activities simultaneously to account for multitasking (e.g., when participants were viewing a patient's record while talking on the phone). The observer had extensive expertise with qualitative research methods, including observations of clinicians' use of electronic documentation systems17 and clinical work practices.18
During the data collection sessions (i.e., the residents' shifts), the observer kept detailed field notes describing the captured activities and the context of these activities. We conducted informal interviews with study participants as member checks to confirm our findings and interpretations of the findings. The interviews were conducted during the days following the observations as one-on-one conversations and lasted between 15 and 45 minutes.
The study was approved by the institutional review board of Columbia University Medical Center; all participants consented to participate in the study prior to observation.
Taxonomy of clinical activities
For this study, we used a modified taxonomy of clinical activities developed by Overhage et al19 and refined by Pizziferri et al,20 which we expanded to allow for a fine-grained examination of computer-based activities. Specifically, we added six new subitems to the looking up data category—viewing the patient list, viewing flowsheets, viewing results, viewing imaging data, viewing general patient data, and viewing a visualization of patient data. In addition, we introduced several new subitems describing different activities related to managing patient care (e.g., managing handoff to-do lists and clearing to-do flags). Because dictation was not used, we did not include the category of reviewing dictation in our taxonomy. The final breakdown of the computer-based activities category is given in Table 2 [NOTE TO LWW: This is not the Table 2 callout].
Table 2. Time Spent on Different Computer-Based Activities by Residents Per Shift, Time and Motion Study, General Medicine Service, NewYork-Presbyterian/Columbia University Medical Center, June and July 2010.
Computer read/write category | Average total time spent, min | % of average total computer-based activities time |
---|---|---|
Documentation | ||
Writing notes (or documenting) | 128.7 | 35.3 |
Viewing the list of available notes | 33.9 | 9.3 |
Reading notes | 27.3 | 7.5 |
Total | 189.9 | 52.1 |
Looking up patient dataa | ||
Viewing the patient list | 25.5 | 7.0 |
Viewing flowsheets | 25.1 | 6.9 |
Viewing results | 15.9 | 4.4 |
Viewing imaging data | 3.3 | 0.9 |
Viewing general patient data | 0.7 | 0.2 |
Viewing a visualization of patient data | 0.3 | 0.1 |
Total | 70.8 | 19.4 |
Managing orders | ||
Writing orders | 24.2 | 6.6 |
Viewing orders | 23.9 | 6.6 |
Total | 48.1 | 13.2 |
Other | 24.0 | 6.6 |
Communicating | ||
Paging | 9.2 | 2.5 |
0.4 | 0.1 | |
Total | 9.6 | 2.6 |
Logistics | ||
Logging in | 6.5 | 1.8 |
Printing | 0.3 | 0.1 |
Managing schedules | 0.1 | 0.0 |
Managing operating room | 0.1 | 0.0 |
Total | 7.0 | 1.9 |
Managing to-do lists | ||
Clearing to-do flags | 5.4 | 1.5 |
Managing handoff to-do lists | 1.2 | 0.3 |
Total | 6.6 | 1.8 |
Reference look up | ||
UpToDate | 3.8 | 0.5 |
1.6 | 0.1 | |
Wikipedia | 0.1 | 0.1 |
Total | 5.5 | 1.5 |
Managing medications | ||
Looking up doses | 0.3 | 0.0 |
Writing prescriptions | 2.7 | 0.2 |
Total | 3.0 | 0.8 |
Total computer-based activities time | 364.5 |
See the main text for more information on the six items in this category.
Data analysis
The analytic approach used in this study was inspired by the work of Zheng et al5 that described multiple analytical tools to visualize and uncover hidden regularities embedded in the sequential execution of patient care tasks in a clinical workflow. The frequencies and durations of activities were calculated using Excel (version 14.5.8 for Mac, Microsoft, Redmond, Washington). The activity visualization tool used in Figure 1 [NOTE TO LWW: This is not the Figure 1 callout.] was custom-developed using the D3 visualization package.21 The aggregated activity view (heat map visualization) used in Figure 2 [NOTE TO LWW: This is not the Figure 2 callout.] was generated using an Excel stacked bar chart feature. Finally, we reviewed all field notes taken during observations and grouped them in accordance with the taxonomy based on the category of activities they referred to. We also conducted a thematic analysis of notes and transcripts from interviews. These notes, transcripts, and themes were used to provide context and explanations for data captured with the time and motion study.
Figure 1.
Aggregated times (in minutes) spent on different activity categories for each participant in the time and motion study, general medicine service, NewYork-Presbyterian/Columbia University Medical Center, June and July 2010. Each row represents a single participant; different shades of gray indicate different activity categories. Notably, in this figure the Computer read/write category does not include the subcategory Documenting (or writing notes), which is presented as a separate category here.
Figure 2.
Heat map visualization of the activities captured for each participant in the time and motion study, general medicine service, NewYork-Presbyterian/Columbia University Medical Center, June and July 2010. The x axis shows the timeline of observations, starting from between 7:00 a.m. and 8:00 a.m. and ending between 3:00 p.m. and 9:00 p.m. In the y axis, each row represents a single participant; the rows are grouped by the participants' postgraduate year (interns on top and second- and third-year residents on bottom). Each shaded block represents a captured activity. The length of the blocks on the horizontal timeline represents the duration of the activities. White spaces (see participants 2 and 6) indicate times when participants were called off the floor and reflect a pause in data capture activities. Notably, in this figure the Computer read/write category does not include the subcategory Documenting (or writing notes), which is presented as a separate category here.
Results
General patterns of activities
The total times spent by residents on different clinical activities per shift are presented in Table 1. On average, out of 720.2 minutes shift time, the residents spent 364.5 minutes (50.6%) using computers (Computer read/write); in comparison, they spent 67.8 minutes (9.4%) interacting with patients (Patient). After computer-based activities, the second most time-consuming activity residents did during their shift time was talking with others in person (Talking, 292.3 minutes [40.6%]); this included conversations about both patient care and general social topics. In addition to reading and writing notes on the computer, residents spent a quarter of their shift time (186.0 minutes [25.8%]) handling paper (Paper read/write), primarily printouts of the electronic signout note; residents in the study often printed this note in the morning and used it throughout the day to capture updates and handwritten to-do lists. The residents spent just over an hour of their shift time in rounds (Rounds, 79.7 minutes [11.1%]). Because the general medicine service covered several floors of the hospital building, the residents spent a considerable amount of their shift time walking or waiting (Moving/waiting, 80.0 minutes [11.1%]). The residents spent 54.0 minutes (7.5%) of their shift time talking on the phone (Phone); most of this time was spent in consultations and on managing patients' discharges. The residents spent 22.8 minutes (3.2%) of their shift time engaged in nonwork or personal activities (Personal). The remaining activity category, Looking for (used for activities related to searching for documents or people), was insignificant and took 2.4 minutes (0.3%) of residents' shift time. Notably, because the study allowed for the capturing of different activities that happened at the same time, the total time of captured activities (1,149.4 minutes) exceeded the total observation time per shift (720.2 minutes on average) by 429.2 minutes (59.6% of shift time), suggesting 59.6% of residents' time was spent multitasking. Figure 1 shows aggregated times spent on different activity categories for each participant.
Table 1. Time Spent on Different Clinical Activities by Residents Per Shift, Time and Motion Study, General Medicine Service, NewYork-Presbyterian/Columbia University Medical Center, June and July 2010.
Activity category | Average total time spent, min | % of average shift timea |
---|---|---|
Computer read/write | 364.5 | 50.6 |
Documentingb | 128.7 | 17.9 |
Talkingc | 292.3 | 40.6 |
Paper read/write | 186.0 | 25.8 |
Moving/waiting | 80.0 | 11.1 |
Rounds | 79.7 | 11.1 |
Patient | 67.8 | 9.4 |
Phoned | 54.0 | 7.5 |
Personal | 22.8 | 3.2 |
Looking fore | 2.4 | 0.3 |
Total | 1,149.4 | 159.6 |
Due to frequent multitasking, the average total time of captured activities exceeds the average total time of observations (720.2 minutes) by 59.6%.
Data for Documenting (or writing notes) are included in the Computer read/write category above, but are provided here for reference.
Included conversations about both patient care and general social topics.
Most of this time was spent in consultations and on managing patients' discharges.
Used for activities related to searching for documents or people.
Computer-based activities
Table 2 presents a more detailed breakdown of the residents' computer-based activities per shift. On average, over half of this time, 189.9 minutes (52.1%) was spent on documentation (writing, viewing, and reading notes). The residents spent considerably more time writing their own notes (128.7 minutes [35.3%]) than readings notes composed by others (27.3 minutes [7.5%]). Notably, the residents spent another 33.9 minutes (9.3%) viewing the list of available notes. This most commonly occurred in situations where residents were expecting a note (from an attending physician or a consultant) and were checking to see whether the note had been made available.
On average, the residents spent over an hour (70.8 minutes [19.4%]) looking up patient data. This time included viewing the patient list, a necessary step when switching between patients (25.5 minutes [7.0%]); viewing flowsheets, which covered patient vital signs, intakes and outputs, respiratory parameters, and nursing assessments (25.1 minutes [6.9%]); viewing results, which included laboratory test results and radiology, cardiology, and pathology reports in either the new or legacy EHR system (15.9 minutes [4.4%]); viewing imaging data (3.3 minutes [0.9%]); viewing general patient data (0.7 minutes [0.2%]); and viewing a visualization of patient data, a custom-developed feature, which allows patient data to be viewed on a timeline (0.3 minutes [0.1%]). The residents also spent 48.1 minutes (13.2%) managing orders, including viewing (23.9 minutes [6.6%]) and writing (24.2 minutes [6.6%]) orders, and just under 10 minutes using computer-mediated communication, such as paging (9.2 minutes [2.5%]) and e-mail (0.4 minutes [0.1%]). Other activities, related to logistics, managing to-do lists, reference look up, and managing medications, each took less than 10 minutes on average. Finally, the residents spent 24.0 minutes (6.6%) on computer-based activities not included in the taxonomy; these were classified in the other category.
Visualizing activities
Figure 2 shows a heat map visualization of the activities captured for each participant during the observed clinical shifts. As is evident from the visualization, patient interactions (black) happened mainly in three different ways: as part of the morning pre-rounding activities, after rounds (usually to carry out the established care plans), and sometimes before the end of the shift. The figure also shows that there was a visible difference in time spent with patients between interns and residents; the informal interviews confirmed the participants perceived interns as being primarily responsible for direct patient contact, unless it involved procedures that required a higher level of skill or expertise. Both computers and paper were most commonly used in the morning while preparing for rounds, after rounds to update care plans, and later in the day to document changes in patients' conditions and to check off completed to-do list items. In the morning hours, the use of computers was tightly coupled with the use of paper. During this time, the residents copied important bits of data from the EHR onto their paper notes to have this information available for rounds. In the afternoon, however, the use of computers was tightly interspersed with communication, as residents often engaged in discussions with others in person and on the phone while using computers.
Discussion
In this study, we examined how residents (first-, second-, and third-year) on the general medicine service in a large urban teaching hospital spent their shift time, with a particular focus on their use of computers. Consistent with previous reports, our study showed residents spend considerably more time interacting with computers (over 50% of their shift time), than in direct contact with patients (less than 10% of their shift time). The difference we found, however, was even more pronounced than what had been reported in earlier studies (e.g., 40% computer use and 12% patient care10). The study may have identified two factors that might contribute to the high level of computer use: inefficiencies in the design of the EHR system and an increasing reliance on computing systems for access to patient data.
Inefficiencies in the design of the EHR system
First, the study indicated a significant portion of the computer-based activities was dedicated to documentation (128.7 minutes, 35.3%). This finding is higher than similar findings in previous reports (e.g., 21% of time spent on documentation22,23), including a previous report from our institution.24 This observation raises a question about whether electronic documentation is inevitably time-consuming and burdensome or whether there are limitations in the design of the current electronic documentation systems that inflate documentation time. This study may have highlighted several aspects of electronic documentation that contributed to inefficient use of time spent documenting. For example, residents spent 33.9 minutes (9.3%) of their shift time viewing the list of available notes, rather than reading them. Our field notes suggested that these situations often occurred when residents were waiting for an attending physician to “drop” their progress note, which would make the discussed care plan official, before proceeding with the planned activities. Since the EHR did not alert the residents to newly posted notes, they had to periodically look for updates. In addition, the high degree of fragmentation in the organization of the patient record, reflected in the six distinct items within the looking up patient data category (which corresponded to six different areas of the EHR containing patient data), may have required residents to spend a considerable amount of time consolidating data from these different areas. Finally, the reliance on desktop computers and their positioning away from patients made it impossible for clinicians to integrate computer-based activities with more direct patient care activities.
Increasing reliance on computing systems for access to patient data
Second, the study suggested half of residents' computer-based activities time was spent on activities not related to documentation, but rather to reviewing patient data (looking up patient data, 70.8 minutes [19.4%]), managing and coordinating patient care (managing orders, 48.1 minutes [13.2%], and communicating, 9.6 minutes [2.6%]), and other activities (other, 24 minutes, [6.6%]). These findings suggest that delivery of patient care necessitates frequent updates to orders and to-do lists to allow members of patient care teams to carry out their respective responsibilities in an efficient and effective manner. Moreover, during the informal interviews, residents who previously completed specialty rotations, such as nephrology, reported decreased reliance on patient contact and increased reliance on information stored within the EHRs (e.g., the available laboratory test results). With the growing amount and richness of patient data available only through computing systems, these trends are likely to amplify. Questions remain, however, as to the role of patients in helping clinicians to interpret these data and ways computing systems can help to facilitate engagement between patients and clinicians, rather than diminish it.
These findings further reinforce the importance of continuous focus on improvements to the design and usability of EHRs.25,26 They also suggest the need to re-envision the EHR as a dynamic tool for facilitating and coordinating complex multidisciplinary patient care and for enhancing communication within patient care teams and between clinicians and patients, rather than as a static record of patient care.
Study limitations
This study has a number of limitations. First of all, it was conducted with a limited number of participants on one general medicine service within one large urban teaching hospital. As such, it has limited generalizability beyond these settings. However, the scale of the study is comparable with other recent time and motion studies of resident workflows. Moreover, the distribution of activities captured in the study is likely to be different in specialty units, such as critical care units, or for different types of residents, such as surgical residents. In addition, the study was conducted in June and July, at the time when new interns begin their residency and do not yet have established work patterns. Further research can show whether these patterns change over time as interns gain more experience. Complementing time and motion studies of clinical workflows with analysis of EHR usage logs (similar to Hripcsak et al24) could allow for an expansion of the number of participants and examinations of differences in workflows between physicians in different subspecialties and of different parameters (time of day, severity of patients' conditions, etc.) on clinical work.
Conclusions
This time and motion study investigated how residents on the general internal medicine service of a large urban teaching hospital spent their shift time, with a particular focus on their use of computers. The study may have uncovered a number of inefficiencies in the design of the EHR system that led to inefficient use of time for documentation of and reviewing patient data, suggesting improvements in the design and usability of EHRs may help to streamline computer-based activities. Arguably, the practice of medicine may have reached the “point of no return” in regard to its reliance on computing systems. The question then becomes not whether and how much clinicians should use computers, but what they should use them for and to what degree the use of computing systems can support clinical care activities.
Acknowledgments
Funding/Support: This work was funded by the National Library of Medicine (National Institutes of Health) grants R01 LM006910 “Discovering and applying knowledge in clinical databases” and T15 LM007079 “Training in Biomedical Informatics at Columbia University.”
Footnotes
Other disclosures: The authors have no competing interests for this publication. The authors have no competing interests for this publication.
Ethical approval: This study was approved by the institutional review board of Columbia University Medical Center.
Contributor Information
Lena Mamykina, Assistant professor of biomedical informatics, Department of Biomedical Informatics, Columbia University, New York, New York.
David K. Vawdrey, Assistant professor of clinical biomedical informatics, Department of Biomedical Informatics, Columbia University, and vice president, Value Institute, NewYork-Presbyterian Hospital, New York, New York.
George Hripcsak, Chair, Department of Biomedical Informatics, Vivian Beaumont Allen Professor of Biomedical Informatics, Columbia University, and director, Medical Informatics Services, NewYork-Presbyterian/Columbia University Medical Center, New York, New York.
References
- 1.Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, Powe NR. Clinical information technologies and inpatient outcomes: A multiple hospital study. Arch Intern Med. 2009;169:108–114. doi: 10.1001/archinternmed.2008.520. [DOI] [PubMed] [Google Scholar]
- 2.Wong DH, Gallegos Y, Weinger MB, Clack S, Slagle J, Anderson CT. Changes in intensive care unit nurse task activity after installation of a third-generation intensive care unit information system. Crit Care Med. 2003;31:2488–2494. doi: 10.1097/01.CCM.0000089637.53301.EF. [DOI] [PubMed] [Google Scholar]
- 3.Campbell EM, Sittig DF, Ash JS, Guappone KP, Dykstra RH. Types of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc. 2006;13:547–556. doi: 10.1197/jamia.M2042. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Han YY, Carcillo JA, Venkataraman ST, et al. Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system. Pediatrics. 2005;116:1506–1512. doi: 10.1542/peds.2005-1287. [DOI] [PubMed] [Google Scholar]
- 5.Zheng K, Haftel HM, Hirschl RB, O'Reilly M, Hanauer DA. Quantifying the impact of health IT implementations on clinical workflow: A new methodological perspective. J Am Med Inform Assoc. 2010;17:454–461. doi: 10.1136/jamia.2010.004440. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: The nature of patient care information system-related errors. J Am Med Inform Assoc. 2004;11:104–112. doi: 10.1197/jamia.M1471. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Ash JS, Sittig DF, Dykstra R, Campbell E, Guappone K. The unintended consequences of computerized provider order entry: Findings from a mixed methods exploration. Int J Med Inform. 2009;78:S69–S76. doi: 10.1016/j.ijmedinf.2008.07.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Ammenwerth E, Spötl HP. The time needed for clinical documentation versus direct patient care. A work-sampling analysis of physicians' activities. Methods Inf Med. 2009;48:84–91. [PubMed] [Google Scholar]
- 9.Poissant L, Pereira J, Tamblyn R, Kawasumi Y. The impact of electronic health records on time efficiency of physicians and nurses: A systematic review. J Am Med Inform Assoc. 2005;12:505–516. doi: 10.1197/jamia.M1700. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Block L, Habicht R, Wu AW, et al. In the wake of the 2003 and 2011 duty hours regulations, how do internal medicine interns spend their time? J Gen Intern Med. 2013;28:1042–1047. doi: 10.1007/s11606-013-2376-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Moore SS, Nettleman MD, Beyer S, et al. How residents spend their nights on call. Acad Med. 2000;75:1021–1024. doi: 10.1097/00001888-200010000-00020. [DOI] [PubMed] [Google Scholar]
- 12.Parenti C, Lurie N. Are things different in the light of day? A time study of internal medicine house staff days. Am J Med. 1993;94:654–658. doi: 10.1016/0002-9343(93)90220-j. [DOI] [PubMed] [Google Scholar]
- 13.Finkler SA, Knickman JR, Hendrickson G, Lipkin M, Thompson WG. A comparison of work-sampling and time-and-motion techniques for studies in health services research. Health Serv Res. 1993;28:577–597. [PMC free article] [PubMed] [Google Scholar]
- 14.Cimino JJ. Improving the electronic health record—Are clinicians getting what they wished for? JAMA. 2013;309:991–992. doi: 10.1001/jama.2013.890. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Wang C, Huang AT. Integrating technology into health care: What will it take? JAMA. 2012;307:569–570. doi: 10.1001/jama.2012.102. [DOI] [PubMed] [Google Scholar]
- 16.NewYork-Presbyterian. 2010 annual report. [Accessed November 12, 2015]; http://nyp.org/pdf/annual_report_2010.pdf.
- 17.Mamykina L, Vawdrey DK, Stetson PD, Zheng K, Hripcsak G. Clinical documentation: Composition or synthesis? J Am Med Inform Assoc. 2012;19:1025–1031. doi: 10.1136/amiajnl-2012-000901. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Mamykina L, Wolf CG. Evolution of contact point: A case study of a help desk and its users. In: Kellog W, ACM Digital Library; ACM Special Interest Group on Supporting Group Work; ACM Special Interest Group on Computer-Human Interaction, editor. Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work; New York, NY. pp. 41–48. [Google Scholar]
- 19.Overhage JM, Perkins S, Tierney WM, McDonald CJ. Controlled trial of direct physician order entry. J Am Med Inform Assoc. 2001;8:361–371. doi: 10.1136/jamia.2001.0080361. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Pizziferri L, Kittler AF, Volk LA, et al. Primary care physician time utilization before and after implementation of an electronic health record: A time-motion study. J Biomed Inform. 2005;38:176–188. doi: 10.1016/j.jbi.2004.11.009. [DOI] [PubMed] [Google Scholar]
- 21.Bostock M, Ogievetsky V, Heer J. D3: Data-Driven Documents. IEEE Trans Vis Comput Graph. 2011;17:2301–2309. doi: 10.1109/TVCG.2011.185. [DOI] [PubMed] [Google Scholar]
- 22.Hollingsworth JC, Chisholm CD, Giles BK, Cordell WH, Nelson DR. How do physicians and nurses spend their time in the emergency department? Ann Emerg Med. 1998;31:87–91. doi: 10.1016/s0196-0644(98)70287-2. [DOI] [PubMed] [Google Scholar]
- 23.Gabow PA, Karkhanis A, Knight A, Dixon P, Eisert S, Albert RK. Observations of residents' work activities for 24 consecutive hours: Implications for workflow redesign. Acad Med. 2006;81:766–775. doi: 10.1097/00001888-200608000-00016. [DOI] [PubMed] [Google Scholar]
- 24.Hripcsak G, Vawdrey DK, Fred MR, Bostwick SB. Use of electronic clinical documentation: Time spent and team interactions. J Am Med Inform Assoc. 2011;18:112–117. doi: 10.1136/jamia.2010.008441. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Thyvalikakath TP, Dziabiak MP, Johnson R, et al. Advancing cognitive engineering methods to support user interface design for electronic health records. Int J Med Inform. 2014;83:292–302. doi: 10.1016/j.ijmedinf.2014.01.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Walji MF, Kalenderian E, Tran D, et al. Detection and characterization of usability problems in structured data entry interfaces in dentistry. Int J Med Inform. 2013;82:128–138. doi: 10.1016/j.ijmedinf.2012.05.018. [DOI] [PMC free article] [PubMed] [Google Scholar]