Skip to main content
Journal of the American Medical Informatics Association : JAMIA logoLink to Journal of the American Medical Informatics Association : JAMIA
. 2020 Mar 6;27(5):798–807. doi: 10.1093/jamia/ocz231

A systematic review of the impact of health information technology on nurses’ time

Esther C Moore 1, Clare L Tolley 1,2,, David W Bates 3,4,5,6, Sarah P Slight 1,2,3
PMCID: PMC7309250  PMID: 32159770

Abstract

Objective

Nursing time represents one of the highest costs for most health services. We conducted a systematic review of the literature on the impact of health information technology on nurses’ time.

Materials and Methods

We followed PRISMA guidelines and searched 6 large databases for relevant articles published between Jan 2004 and December 2019. Two authors reviewed the titles, abstracts, and full texts. We included articles that included a comparison group in the design, measured the time taken to carry out documentation or medication administration, documented the quantitative estimates of time differences between the 2, had nurses as subjects, and was conducted in either a care home, hospital, or community clinic.

Results

We identified a total of 1647 articles, of which 33 met our inclusion criteria. Twenty-one studies reported the impact of 12 different health information technology (HIT) implementations on nurses’ documentation time. Weighted averages were calculated for studies that implemented barcode medication administration (BCMA) and 2 weighted averages for those that implemented EHRs, as these studies used different sampling units; both showed an increase in the time spent in documentation (+22% and +46%). However, the time spent carrying out medication administration following BCMA implementation fell by 33% (P < .05). HIT also caused a redistribution of nurses’ time which, in some cases, was spent in more “value-adding” activities, such as delivering direct patient care as well as inter-professional communication.

Discussion and Conclusions

Most of the HIT systems increased nursing documentation time, although time fell for medication administration following BCMA. Many HIT systems also resulted in nurses spending more time in direct care and “value-adding” activities.

Keywords: medical records systems, computerized, nurses, time management, health information technology

INTRODUCTION

Nursing time represents the single highest cost for most health services. Healthcare services around the world are also experiencing a shortage of nursing staff,1 with the National Health Service vacancies for registered nursing and midwife staff increasing by approximately 19% since October 2015 in the UK.2 In the US, it is projected that the number of nurses in employment will increase by 15% over the next 15 years, with 7 states expected to have nursing shortages by 2030.3 A report published by the Australian government predicted that the demand for nurses would exceed supply by approximately 85 000 in 2025 and 123 000 in 2030.4 This highlights the need for healthcare organizations internationally to increase nursing efficiency to continue to deliver high quality patient care with limited resources.

Nurses use a wide range of health information technology (HIT), including electronic health records (EHRs), computerized provider order entry (CPOE), and barcode medication administration (BCMA) as part of their daily activities.5,6 These systems have the potential to increase nurses’ productivity, for example, by reducing the amount of time that they spend carrying out documentation.7 Productivity is defined as the output per unit of input; in nursing, this translates to how efficient the nurse is at carrying out their required activities and whether they meet the needs of their patients.8 Recently, nursing productivity has been considered in terms of the amount of time nurses spend delivering “value-adding” care; this includes the time spent delivering direct patient care as well as inter-professional communication and other activities which have a direct benefit to the patient.9 Techniques such as work sampling and time and motion studies have been used to measure how much time nurses spend performing various nursing activities.10–14 The usability of HIT can have an impact on nurses’ workflow, and this can vary significantly between different systems.15,16 Cho et al17 compared the usability of 6 different electronic nursing records and found significant variation (of between 226.3 seconds to 457.2 seconds) in the time taken to complete certain tasks. It is therefore important to consider this when comparing across different HIT implementation studies. The number of empiric evaluations of the impact of HIT on nurses’ time has grown in recent years. For example, Bosman conducted an observational study using work sampling to evaluate the impact of an intensive care information system on the time that nurses spent on documentation.18 They found that it reduced documentation thus allowing nurses to spend more time on patient care activities and, in particular, tasks which are often omitted, delayed, or unfinished—otherwise known as “Missed Care.”18,19 Several studies have shown that missed care has an impact on both patient safety and satisfaction.20–23

However, HIT may also bring about an increase in the time taken to perform certain tasks with changes in workflow. For example, Yeung et al24 conducted a time and motion study to evaluate the impact of an electronic vital sign recording system on documentation workflow and found that nurses would often informally document results on paper before inputting the same information on the computer. The net result was that more time was spent on documentation due to duplication.24 Similarly, HIT encouraged multitasking and interruptions to a nurse’s workflow.25 Studies have shown that high volumes of alerts generated by clinical alarm systems were actually false alarms, with no clinical relevance, and resulted in users becoming desensitized to such systems.26 Some nurses also found technology particularly stressful, especially when it malfunctions.27 Burnout in physicians has been associated with spending large amounts of time documenting on EHRs and has been referred to as a potential reason for the increased number of physicians leaving the profession in the US.28 One solution to this has been to improve the efficiency of documentation and work flows such that physicians have more time with patients.28,29

OBJECTIVE

Poissant et al7 conducted an important systematic review in 2005 that measured the impact of EHRs on the time efficiency of physicians and nurses and reported that EHRs did not reduce the amount of time that physicians spent on documentation but did for nurses.7 This search was conducted over 14 years ago and focused specifically on EHRs with only 10 of the 20 included studies reporting on nursing activity.7 Since this review was published, there have been considerable advances in technology, and more published studies have explored the many new technologies used by nurses (eg, BCMA). To address this knowledge gap, we conducted a systematic review of the literature on the impact of health information technology on nurses’ time.

METHODS AND MATERIALS

Information sources and search

We searched 6 databases: Embase, Medline, Cumulative Index to Nursing and Allied Health Literature, Scopus, PsycInfo, and Web of Science. The key words were based on those used by Poissant et al7 (see Figure 1).

Figure 1.

Figure 1.

Sets of key words used in the literature search.

The search was conducted in February 2018, updated in December 2019, and limited to papers published after 2004, as in the previous review.7 We were particularly interested in the last 15 years, with $19.2 billion in funds committed to the incentivization of CPOE adoption in 2009 and the rapid advancement in technology that has taken place since then.30

Inclusion and exclusion criteria

We included studies that examined the effects of HIT on nurses. Studies set in a care home, hospital, or community clinic were considered. Studies that had a comparison group, recorded the time taken to carry out documentation or medication administration, and documented quantitative estimates of the time differences were also included. We excluded papers that were not published in English or were opinion pieces.

Study selection

All studies were included regardless of the study design. Titles and abstracts were screened by 2 independent reviewers (EM and CT). In the case that the abstract was not available, the paper was put through to full text review. The full texts were then screened by the same 2 reviewers (EM and CT) against the inclusion and exclusion criteria. Disagreements were solved by discussion, and the reasons for exclusion documented. In accordance with the previous systematic review, terms such as “time charting” were interpreted as time documenting.7 In the case that studies referred to “paper charting” and “electronic charting” separately, these were combined and interpreted as time documenting to allow for comparisons to be made across studies. We also collected data related to time in medication-related tasks, such as BCMA, where time administering medication was measured. Studies that did not specifically report on time documenting or time spent on medication-related tasks were reported separately. In the case that studies were carried out over more than 2 time points, the first and last time point were selected for inclusion in the analysis.

Evaluation process

Two methods were used to evaluate the studies in this review. We used the ROBIN-1 tool to assess bias in the included studies (Supplementary Material Tables 1 and 2).31 The answers were combined to give an overall risk of bias assessment as either: low, moderate, serious, critical, or no information. The second method used a similar approach to that used by Poissant et al.7 The study designs were ranked, with posttest control studies ranking most highly, followed by the multi-group pretest-posttest studies, and finally one-group pretest-posttest designs.7 The data collection methodologies were also ranked, with time and motion observer methodology most highly, followed by work sampling techniques, and then self-reporting surveys.7

Relative time differences were calculated for each study to facilitate comparison across the studies. This was calculated by taking the time to complete a task with a computer minus the time taken to complete the task on paper, divided by the time to complete task on paper. This was then multiplied by 100 to get a percentage, producing a negative value when the intervention was time efficient. In order to compare studies with different sized sampling units (observations, working shifts, number of patients), weighted averages were calculated. The articles were grouped according to the type of HIT investigated, the time measure used (documentation time or time in medication administration), or the type of sampling unit used. Weighted averages were calculated using the following formula:

Weighted average=i+1n[SWi*RTDi]i+1nSW(i)

Where sampling weight (SW) = (ngroup1+ngroup2) and relative time difference = ((documentation timegroup2 – documentation timegroup1)/documentation timegroup1) * 100. P values were used to assess the significance; if this was not available, then the author was contacted and the required information requested.

RESULTS

We identified 1647 articles in the initial search (see Figure 2). After removal of duplicates (n = 254) 1393 titles followed by 484 abstracts were screened against our inclusion and exclusion criteria. Finally, 113 full-text articles remained, which were then read and assessed. Thirty full texts met our inclusion criteria; we reviewed their reference lists for relevant articles, finding another 3 articles that met our inclusion criteria. In total, 33 articles were included in this review. Table 1 provides an overview of the included studies’ design and data collection methodology.

Figure 2.

Figure 2.

Flow diagram showing the article search and inclusion process.

Table 1: Study designs and data collection methods of selected papers.

Data collection methodology
Study design Time and motion observed/video recording Work sampling Survey/self-report
Posttest control
  • Qian et al (2015)32

  • Hollingworth et al (2007)33

  • Westbrook et al (2013)34

  • Elganzouri et al (2009)35

  • Wilbanks et al (2018)36

Yee et al (2012)37 Tsai et al (2010)38
Multiple group pretest-posttest
  • Cornell et al (2010)39

  • Poon et al (2008)40

  • Poon et al (2006)41

  • Devine et al (2010)42

  • Pereira et al (2013)43

  • Schenk et al (2018)44

  • Leung et al (2018)45

  • Walker et al (2019)46

  • Schachner et al (2016)47

  • Schachner et al (2016)48

  • Gomes et al (2016)49

  • Huang and Lee (2011)50

one-group pretest-posttest
  • Boeckx et al (2010)51

  • Banner and Olney (2009)52

  • Asaro and Boxerman (2008)53

  • Banet et al (2006)54

  • McComas et al (2014)55

  • Dwibedi et al (2012)56

  • Dwibedi et al (2011)57

  • Weiss and Weiss (2011)58

  • Munyisia et al (2011)59

  • Munyisia et al (2014)60

  • Franklin et al (2007)61

  • Hakes and Whittington (200 8)62

  • Burkle et al (2010)63

  • Saarinen and Aho (2005)64

Study characteristics

Twenty-one papers reported the impact of 12 different HIT systems on the total time that nurses spent on documentation tasks (see Supplementary Material Table 1).32,33,35–37,39,47–49,51–54,58–60,62–64 These included CPOE and electronic nursing documentation (n = 3),35,53,54 EHRs (n = 7),44,46,49,52,59,60,62 electronic medication administration records (eMARs) (n = 4),32,37,39,55 electronic clinical information systems (n = 2),58,64 a renewed nursing e-chart (n = 2),47,48 an electronic intensive care information system (n = 1),51 a patient data management system (n = 1),63 and an electronic documentation system (n = 1).36 Fourteen of these studies reported results in terms of the amount of time spent on medication administration or medication related activities, 2 of these studies reported time in medication administration, and 1 study on the time per prescription event (see Supplementary Material Table 2).32,34,35,38,40–44,50,55–57,61 The remaining 6 studies each implemented a different intervention including barcoded vaccinations,43 CPOE plus desktops in examination rooms,42 an electronic medication management system,34 a closed loop system (which incorporated electronic prescribing), ward-based automated dispensing, barcode patient identification and eMARs,61 a system that incorporated an eMAR, CPOE and EMR,35 and a CPOE system which is known as an Inpatient Medication Order Entry system (IPMOE) in Hong Kong.45 Eighteen of the studies were conducted in the USA, Australia (n = 5), Taiwan (n = 2), Argentina (n = 2), and 1 study in Belgium, Germany, UK, Canada, Finland, and Hong Kong (see Supplementary Material Tables 1 and 2 for more information). The time from implementation to evaluation was reported in all but 6 studies (see Supplementary Material Tables 1 and 2).35–37,43,45,47 We did not find a relationship between the time from implementation to evaluation and the impact on nurses’ time.

Quality appraisal

Risk of bias for each study is reported in Supplementary Material Tables 1 and 2. Overall, 11 studies accounted for some confounders, such as the nurse’s level of education, level of experience, patient characteristics, the number of patients under the care of a nurse, and the number of nurses on the unit.34,36–38,40,45,49,56,57,62,64 However, no study controlled for all confounders. For example, many studies did not collect data during a night shift, and therefore none were classified as having a low or moderate risk of bias. Table 1 summarizes the results of the second method of evaluation. Time and motion was the most common data collection method, accounting for 64% of included studies compared to work sampling (n = 27%) and survey or self-reporting (n = 9%). Fourteen studies used one-group pretest-posttest, 12 used multiple group pretest-posttest, and 7 used posttest control.

We found a large range in the time between the implementation and when evaluation measurements were taken, from 2 to 24 months (see Supplementary Material Tables 1 and 2). Increased familiarity with a system may affect the time taken to complete certain tasks, which should also be considered when interpreting these data.

The following sections present the findings grouped according to studies that reported the time spent on (1) documentation, (2) medication administration, and (3) other tasks. Although there are a range of nursing task lists available in the literature, the activities included may differ according to division of work performed depending on the country or setting.65 There is therefore a lack of consensus about which task list to use and therefore we opted to group our findings according to 3 key areas of interest, where HIT may have impacted on nursing time.

Time spent on documentation

Twenty-one studies investigated the impact of various HIT interventions on the time that nurses spent on documentation. Seven of these reported a decrease in time spent documenting, including 2 studies that implemented a nursing e-chart and analyzed the same dataset,47,48 2 studies that implemented an EHR,49,59 1 study that implemented a patient data management system,63 another an eMAR,32 and a study that evaluated 2 electronic clinical documentation systems: auto-filling (ie, insertion of complete blocks of text to be edited) and computer assisted (ie, data selection with semistructured documentation).36

A further study that evaluated the impact of an electronic medication charting system at 2 different sites found a reduction in time at 1 and an increase in time at the other.39 When the average relative time difference of the 2 sites was calculated, there was a −1% change in the time spent documenting.39 Of note, the authors reported that CPOE had been implemented at 1 of these sites but did not specify which one.39 A further study involving the implementation of a patient data management system, found an overall decrease of −34%, which was primarily due to reductions in both the time spent on documentation in daily activities and discharging a patient.63 The relative time difference for studies that reported time saving ranged from −16% to −37%.

Eleven studies reported an increase in the time spent on documentation (see Supplementary Material Table 1).33,35,44,46,51–54,60,62,64 One study conducted over 3 hospital sites investigated the implementation of eMAR, CPOE, electronic medical records, and electronic medication storage/cart storage and reported the largest increase in the time taken to complete medication documentation with a relative difference of +180% in the time taken to complete medication documentation.35 The authors also found an increase in the time taken to retrieve and administer medication.35 Hollingworth et al33 investigated the impact of prescribing electronically on both desktop and laptop computers compared to prescribing on paper in a primary care setting. This intervention resulted in an increase in the total time that nurses spent on computer and writing tasks, although this was statistically insignificant.33 Asaro and Boxerman and Banet et al both evaluated CPOE and electronic nursing documentation in an emergency department53,54 and reported an increase of 7.9% (significant) and 3% (significance not reported) in the time spent documenting, respectively.53,54 Banner and Olney implemented automated clinical documentation as part of an EHR and found a statistically significant increase of 52% in the percentage of time spent documenting postimplementation.52 Munysia et al60 implemented an EHR in a nursing home and reported a significant increase of 62% in the documenting time. Walker et al46 conducted 2 studies: 1 on a surgical ward, which implemented a fully integrated EHR system; and 1 on a medical ward, which only implemented the medication support component of the EHR. A statistically significant increase in the time spent documenting during ward rounds was found in both wards, a 1.7% increase on surgical ward and a 72% increase in the median time spent documenting during medication rounds on the medical ward.46 Schenk et al44 also implemented an EHR and reported an increase in the time spent documenting of 3% (calculated by adding together the time spent documenting notes, documenting medicines, and transcribing/managing orders). Boeckx et al51 were the only authors to implement an HIT system in a pediatric intensive care unit and found a statistically insignificant increase of 41% in the percentage of time that nurses spent documenting information. Saarinen and Aho implemented a clinical information system in an adult intensive care unit and reported a 49% increase in the time spent documenting nursing care.64 Hakes and Whittington reported a 17% increase in the percentage of time spent documenting following implementation of an electronic medical record without CPOE in medical/surgical units.62 Two studies did not report any change in the time spent documenting postimplementation of a new HIT system.37,58

Time spent on documentation: Weighted averages

Weighted averages were calculated for studies that investigated the same HIT intervention and used the same sampling unit; these can be seen in Figure 3. CPOE and electronic nursing documentation resulted in an increase of +5.1% time spent on documentation.53,54 EHRs were associated with an increase of +22% time spent on documentation, which was calculated from 3 studies (only the data from the surgical ward was used from Walker et al due to this being the most comprehensive).46,49,52 These data are represented by “Electronic health record (1)” in Figure 3. A further 2 studies that focused on the implementation of EHRs—but used a different sampling unit—had a weighted average of +46%, represented by “Electronic health record (2)” in Figure 3.59,60 The 2 studies that implemented eMAR systems used different sampling units and measured the time in different activities, and so a weighted average could not be calculated.32,55 Two studies examined the impact of a renewed nursing e-chart and analyzed the same dataset, so a weighted average was not calculated.47,48

Figure 3.

Figure 3.

Weighted averages calculated for each health information technology from studies using comparable sampling units. Electronic health record (1) calculated from studies which used the number of shifts as their sampling unit. Electronic health record (2) calculated from studies which used the number of observations as their sampling unit.

Time spent on medication administration related tasks

Seven studies reported on the use of barcode technology, 6 of which were BCMA,38,40,41,50,56,57 and 1 referred to using barcoding to document vaccine administration (Supplementary Material Table 2).43 Poon et al and Dwibedi et al each published 2 studies that reported on the use of a BCMA system based on the same data;40,41,56,57 we included the most recent study in our analysis.40,56 Three studies reported a decrease in the time spent in medication administration postimplementation of BCMA.38,40,50 Huang and Lee found a statistically significant decrease,50 and Poon et al40 reported a reduction in medication administration time, although this was statistically insignificant. Tsai et al38 reported the greatest time saving (−50%) resulting from the implementation of BCMA but did not report the significance of their findings. Dwibedi et al56 reported a significant increase in the time spent on medication administration, although the authors categorized tasks differently than other studies. A study that reported on barcoded vaccinations found a significant decrease in the time spent recording information related to the vaccination postimplementation; this was the only barcode technology study to be conducted in a community setting.43

Two studies reported results following the implementation of an eMAR,32,55 and 2 further studies implemented a system which incorporated an eMAR alongside other HIT.34,61 Qian et al32 reported both time in documentation and the percentage of time spent on medication administration in a 3-hour medication round after the implementation of an eMAR and found no statistically significant change in the amount of time spent in either activity. They did, however, report a significant reduction in the time spent locating medication-related records. McComas et al55 also implemented an eMAR and found this resulted in a statistically significant increase of +27% in the amount of time taken to complete the medication administration process. Franklin et al61 implemented a closed-loop system incorporating electronic prescribing, ward-based automated dispensing, barcode patient identification and eMARs and reported how the amount of time spent on non-IV drug rounds decreased with a relative difference of −20%, However, a relative difference of +36% was found for the amount of time spent on medication activities between drug rounds, and both results were statistically significant.61 Leung et al45 implemented an IPMOE system and found a statistically insignificant increase (+14%) in the time spent on medication administration. The authors proposed that this increase was due to scanning medications, which was associated with increased patient safety. This study also reported a decrease in medication-related tasks (−25%), which may have offset the increased time spent in medication administration, overall resulting in time saved.45 Westbrook et al34 implemented an electronic medication management system, which included electronic prescribing and medication administration records. A statistically insignificant increase of +8.1% in time spent on medication-related tasks was reported.34 Schenk et al44 implemented an EHR and also found an increase of 3.2% in the time spent in medication administration tasks (calculated from the combined time spent preparing medication, administering medication, managing IVs, and explaining medication action to patients/family).

Time spent on medication administration related tasks: weighted averages

The weighted average for the time difference using BCMA calculated from 4 studies that used the same sampling units and HIT was −33% see Figure 2.38,40,50,56 The weighted averages for other interventions could not be calculated.32,34,43,44,55,61

Studies reporting in terms of “other tasks”

We identified 2 studies that reported on “other tasks” and were therefore not included in the weighted average calculations. Westbrook et al34 and Devine et al42 both investigated the impact of a CPOE and electronic medication management system on efficiency. Westbrook et al34 reported a statistically insignificant increase of +29% on time in direct care. Devine et al42 conducted a study which examined the impact of CPOE and desktop computers on the time for prescribers to complete prescriptions and found a relative time difference of −8.4% in the time taken for nurses to complete a prescription event.

DISCUSSION

We performed a systematic review assessing the impact of HIT on nurses’ time. We found that most HIT systems increased nursing documentation time, although time did fall for medication administration following BCMA implementation. Our results also showed that HIT often caused a redistribution of nurses’ time, which in some cases was spent in more “value-adding” activities, such as direct care and communicating with other members of staff and patients’ relatives.

Previous work has shown that HIT can be used to enhance preventative healthcare delivery, decrease the frequency of adverse drug events, and contribute to a reduction in medication errors.66–68 Despite these benefits, however, if HIT is not compatible with the existing workflow, it is unlikely to be successful.69,70 One systematic review that investigated the barriers to and facilitators of the successful implementation of EHRs found that a lack of time and heavy workload reduced the probability of a successful implementation.71 Many approaches are already available (eg, building interfaces which document data like vital signs without nurses having to input these) and could substantially reduce workload. Paying attention to human factors when nurses are entering data could also be helpful.

The implementation of BCMA has been shown to reduce medication administration errors.6,72–74 Our review suggests that BCMA was associated with the greatest time savings (−33%), while EHRs, CPOE, electronic nursing documentation, and eMARs resulted in an increase in time spent on documentation. This contrasts with the previous review, which was limited to studies that only focused on EHRs and, overall, reported increased nursing efficiency.7 Three of the 4 studies, which implemented BCMA reported a decrease in the time spent on medication administration/medication-related activities.38,40,50 Poon et al40 reported decreased time spent managing physician’s orders, the delivery of medicines to the patient, and inefficient tasks, such as travelling and looking for patients records. Poon et al40 highlighted that some of the time savings reported were due to the context of BCMA implementation. For example, time spent travelling was decreased due to the implementation of bedside laptops, which had not been present pre-BCMA. This emphasizes the importance of workflow redesign. Time savings could also be made through HIT by removing the need to manually transcribe, verify, and renew orders.38,50,75 Dwebedi et al56 was the only BCMA study to report an increase in the time spent on medication administration. Although they found a decrease in the time taken to document medications, this was not offset by an increase in time spent on drug preparation, assisting physicians, universal precautions, and conversing with patients.56 In particular, other factors, such as a difference in the types of patients (eg, more patients ≥ 65 years old) and medications encountered (eg, fewer solid oral dosage forms) between the pre- and postimplementation phases may have contributed to an increase in drug preparation time.56 Studies should aim to control for such confounders in the future.56 Wilbanks et al36 evaluated 2 different electronic documentation systems, auto-filling and computer-assisted, compared to paper system. Both systems resulted in time savings, with the auto-filling system having the greatest impact.36 However, auto-filling was associated with lower documentation accuracy demonstrating the need for a balance between the desire to save time while maintaining standards.36

Over half of the included papers (n = 19) reported an increase in the time spent documenting or on medication administration-related activities, which was associated with a range of different HIT systems (eg, CPOE, automated clinical documentation, EHRs, etc).32–35,39,44–46,51–57,60–62,64 Nurses continued documenting information on paper despite the availability of HIT, because this was quicker and the information was close “at hand.”59 Duplication of information could increase the risk of transcription errors, with a previous study reporting how 56% (n = 310) of medication errors were due to transcription errors, 41% (n = 127) of which have potentially significant clinical consequences.76 Cornell et al39 reported no change in the amount of time that nurses spent making personal notes, an aspect that vendors could potentially identify new approaches for in the future. Another study suggested that the increase in documentation time was due to increased time spent finding relevant information on the electronic system.60 However, it is possible that nurses became more familiar with the system over time and possibly used electronic approaches to share information with their colleagues.60

The implementation of HIT often resulted in the redistribution of nurses’ time from activities, such as documentation/medication administration, to direct care activities or “value-adding” activities.9 One study found that nurses spent an increased amount of time talking to relatives (indirect care),39 and 3 described an increase in time spent in patients’ rooms and in personal interactions with patients (direct care).44,49,59 Increased time providing direct care can also improve job satisfaction because nurses feel more valued and rewarded.77 However, HIT can also be disruptive to nurses’ workflow and result in negative consequences, such as increased multitasking and interruptions.25 For instance, false alarms can result in nurses becoming desensitized, leading to poor patient outcomes.78 Malfunctions can also cause stress and lead to what is known as compassion fatigue, eventually contributing to burnout.27,79 These positive and negative impacts are all important to consider at a time when there is a shortage of nurses and attempts are been made to increase recruitment and retention.80

Poissant et al7 previously found that studies that observed clinicians soon after implementation (defined as 3 months or less) showed a reduction in time spent documenting, and studies that waited longer tended to show increases. This review did not reveal a correlation between the time after implementation that the study took place and the impact of the HIT on nurses’ time, although some studies did support Poissant et al’s previous finding.60 Munyisia et al,60 for example, observed nurses at 4 different time points postimplementation (3, 6, 12, and 23 months after) and found that the percentage of time nurses spent documenting reduced at 3 months but increased to a higher percentage than before implementation at 23 months. Like Poissant et al, the authors reflected on how this might be due to nurses becoming more familiar with the system and, in turn, using more of its functions.7,60 How well the system is designed can impact on the time that it takes nurses to conduct tasks.17 None of the studies in this review carried out usability testing as part of their study, which makes it difficult to compare across studies including those that implemented the same type of HIT. Leung et al,45 did acknowledge the potential impact of usability on their results, as they observed nurses having difficulty using the system due to problems with the interface between the software and hardware. Further studies that explore the impact of usability and optimized system design are needed. Such studies should also describe the contextual factors related to a particular implementation (eg, the organizational setting, users, tools used, etc) to help better understand the results.

The majority of our included studies employed a time and motion approach, which is considered to be the most precise and accurate method of assessing the time spent on activities in health services research.81 However, these often had a one-group pretest-posttest study design, which are vulnerable to temporal changes such as an increase in the number of patients and severity of illnesses seen in the winter months.82 Researchers should aim to use a control group to increase validity. Use of standardized methods for time and motion studies (eg, the suggested time and motion procedures [STAMP]) guidelines could improve the “consistency of design, conduct, and results reporting of time and motion research in health informatics” and would facilitate a more straightforward comparison between studies.83 It is also important that studies categorize nursing tasks in a standardized way; this was not the case in this review which made interpreting the results more challenging.56 Although the final number of studies included in this review was large (33 studies), these were split across numerous different HIT systems, which may limit the generalizability of our findings. The lack of studies that have explored the impact of the different HIT systems on nurses’ time might be due to the time-consuming nature of time and motion studies, since they are more labor intensive, requiring one-to-one observations.81

It is also important to consider the statistical significance of results found in the studies before overall conclusions are made. Of the papers reporting time spent in documentation, less than a third (29%) showed a statistically significant change.32,36,44,46,52,59,60,62 Four studies investigated BCMA/barcode scanning technology,43,50,56,57 and only 2 of these reported a statistically significant decrease in time spent in medication administration.43,50 Certain confounding factors (eg, a nurse’s experience and the number of patients a nurse was caring for) were controlled for in less than half of the included studies34,36–38,40,45,49,56,57,61,62,64 and must be taken into account in future studies.

CONCLUSION

Nursing time is one of the single most expensive things in secondary care, and there are trade-offs between direct patient care and documentation specifically. We found that several interventions have been implemented with varying effects on nurses’ time. BCMA showed the most marked average relative time difference of −33% and was the only intervention to demonstrate a statistically significant decrease in time in more than 1 study. Other applications, including adoption of EHRs and CPOE, were associated with increased documentation time across multiple studies. A key need in informatics is more research on using HIT to improve efficiency, focused on finding new efficiencies, and using robust methods to enable comparisons between studies.

FUNDING

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

AUTHOR CONTRIBUTIONS

All authors contributed substantially to the conception and design of this work as well as the analysis and interpretation of the data, and helped to draft and revise the manuscript. All authors approved the final version to be published and are accountable for all aspects of this work.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Conflict of Interest statement

None declared.

Supplementary Material

ocz231_Supplementary_Data

REFERENCES

  • 1.Global Health Workforce Alliance and World Health Organization. A Universal Truth: No Health Without a Workforce; 2013. https://www.who.int/workforcealliance/knowledge/resources/GHWA-a_universal_truth_report.pdf? ua=1. Accessed January 16, 2018. [Google Scholar]
  • 2.NHS Digital. NHS Vacancy Statistics England- February 2015-March 2018, Provisional Experimental Statistics; 2018. https://digital.nhs.uk/data-and-information/publications/statistical/nhs-vacancies-survey/nhs-vacancy-statistics-england—february-2015—march-2018-provisional-experimental-statistics. Accessed November 1, 2017.
  • 3.US Department of Health and Human Services, Health Resources and Services Administration, National Center for Health Workforce Analysis. National and Regional Supply and Demand Projections of the Nursing Workforce: 2014–2030 Rockville, Maryland; 2017. https://www.ic4n.org/wp-content/uploads/2018/02/Supply-and-Demand-Projections-of-the-Nursing-Workforce-2014_2030.pdf. Accessed August 23, 2018.
  • 4.Health Workforce Australia. Australia’s Future Health Workforce–Nurses Overview; 2014. https://www1.health.gov.au/internet/main/publishing.nsf/Content/34AA7E6FDB8C16AACA257D9500112F25/$File/AFHW%20-%20Nurses%20overview%20report.pdf. Accessed August 23, 2018.
  • 5. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10): 742–52. [DOI] [PubMed] [Google Scholar]
  • 6. Koppel R, Wetterneck T, Telles JL, Karsh B-T.. Workarounds to barcode medication administration systems: their occurrences, causes, and threats to patient safety. J Am Med Inform Assoc 2008; 15 (4): 408–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 7. Poissant L, Pereira J, Tamblyn R, Kawasumi Y.. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 2005; 12 (5): 505–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. Rhoads J, Ferguson LA, Langford CA.. Measuring nurse practitioner productivity. Dermatol Nurs 2006; 18 (1): 32. [PubMed] [Google Scholar]
  • 9. Upenieks VV, Akhavan J, Kotlerman J, Esser J, Ngo MJ.. Value-added care: a new way of assessing nursing staffing ratios and workload variability. J Nurs Adm 2007; 37 (5): 243–52. [DOI] [PubMed] [Google Scholar]
  • 10. Hendrich A, Chow MP, Skierczynski BA, Lu Z.. A 36-hospital time and motion study: how do medical-surgical nurses spend their time? Perm J 2008; 12 (3): 25. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Hollingsworth JC, Chisholm CD, Giles BK, Cordell WH, Nelson DR.. How do physicians and nurses spend their time in the emergency department? Ann Emerg Med 1998; 31 (1): 87–91. [DOI] [PubMed] [Google Scholar]
  • 12. Jinks AM, Hope P.. What do nurses do? An observational survey of the activities of nurses on acute surgical and rehabilitation wards. J Nurs Manag 2000; 8 (5): 273–9. [Google Scholar]
  • 13. Lundgren S, Segesten K.. Nurses' use of time in a medical–surgical ward with all‐RN staffing. J Nurs Manag 2001; 9 (1): 13–20. [DOI] [PubMed] [Google Scholar]
  • 14. Westbrook JI, Duffield C, Li L, Creswick NJ.. How much time do nurses have for patients? A longitudinal study quantifying hospital nurses' patterns of task time distribution and interactions with health professionals. BMC Health Serv Res 2011; 11 (1): 319. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15. Kaufman D, Roberts WD, Merrill J, Lai T-Y, Bakken S.. Applying an evaluation framework for health information system design, development, and implementation. Nurs Res 2006; 55 (Supplement 1): S37–42. [DOI] [PubMed] [Google Scholar]
  • 16. Yen P-Y, Bakken S.. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19 (3): 413–22. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 17. Cho I, Kim E, Choi WH, Staggers N.. Comparing usability testing outcomes and functions of six electronic nursing record systems. Int J Med Inform 2016; 88: 78–85. [DOI] [PubMed] [Google Scholar]
  • 18. Bosman RJ. Impact of computerized information systems on workload in operating room and intensive care unit. Best practice and research. Clin Anaesthesiol 2009; 23 (1): 15–26. [DOI] [PubMed] [Google Scholar]
  • 19. Hessels AJ, Flynn L, Cimiotti JP, Cadmus E, Gershon R.. The impact of the nursing practice environment on missed nursing care. Clin Nurs Stud 2015; 3 (4): 60–5. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20. Thomas-Hawkins C, Flynn L, Clarke SP.. Relationships between registered nurse staffing, processes of nursing care, and nurse-reported patient outcomes in chronic hemodialysis units. Nephrol Nurs J 2008; 35 (2): 123. [PMC free article] [PubMed] [Google Scholar]
  • 21. Sochalski J. Is more better?: the relationship between nurse staffing and the quality of nursing care in hospitals. Med Care 2004; 42 (Suppl): II–67–II-73. [DOI] [PubMed] [Google Scholar]
  • 22. Schubert M, Glass TR, Clarke SP, et al. Rationing of nursing care and its relationship to patient outcomes: the Swiss extension of the International Hospital Outcomes Study. Int J Qual Health Care 2008; 20 (4): 227–37. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Lucero RJ, Lake ET, Aiken LH.. Nursing care quality and adverse events in US hospitals. J Clin Nurs 2010; 19 (15-16): 2185–95. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24. Yeung MS, Lapinsky SE, Granton JT, Doran DM, Cafazzo JA.. Examining nursing vital signs documentation workflow: barriers and opportunities in general internal medicine units. J Clin Nurs 2012; 21 (7-8): 975–82. [DOI] [PubMed] [Google Scholar]
  • 25. Dean S, Lewis J, Ferguson C.. Is technology responsible for nurses losing touch? J Clin Nurs 2017; 26 (5-6): 583–5. [DOI] [PubMed] [Google Scholar]
  • 26. Nanji KC, Slight SP, Seger DL, et al. Overrides of medication-related clinical decision support alerts in outpatients. J Am Med Inform Assoc 2014; 21 (3): 487–91. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27. Sawatzky J-A. Stress in critical care nurses: actual and perceived. Heart Lung 25 (5): 409–17. [DOI] [PubMed] [Google Scholar]
  • 28. Downing NL, Bates DW, Longhurst CA.. Physician burnout in the electronic health record era: are we ignoring the real cause? Ann Intern Med 2018; 169 (1): 50. [DOI] [PubMed] [Google Scholar]
  • 29. Washington V, DeSalvo K, Mostashari F, Blumenthal D.. The HITECH era and the path forward. N Engl J Med 2017; 377 (10): 904–6. [DOI] [PubMed] [Google Scholar]
  • 30. The American Recovery and Reinvestment Act: HR1; 2009. http://frwebgate.access.gpo.gov/cgi-bin/getdoc.cgi? dbname=111_cong_bills&docid=f: h1enr.pdf. Accessed November 29, 2019.
  • 31. Sterne JAC, Hernán MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 2016; 355: i4919. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 32. Qian S, Yu P, Hailey DM.. The impact of electronic medication administration records in a residential aged care home. Int J Med Inform 2015; 84 (11): 966–73. [DOI] [PubMed] [Google Scholar]
  • 33. Hollingworth W, Devine EB, Hansen RN, et al. The impact of e-prescribing on prescriber and staff time in ambulatory care clinics: a time-motion study. J Am Med Inform Assoc 2007; 14 (6): 722–30. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34. Westbrook JI, Li L, Georgiou A, Paoloni R, Cullen J.. Impact of an electronic medication management system on hospital doctors' and nurses' work: a controlled pre-post, time and motion study. J Am Med Inform Assoc 2013; 20 (6): 1150–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35. Elganzouri ES, Standish CA, Androwich I.. Medication Administration Time Study (MATS): nursing staff performance of medication administration. J Nurs Adm 2009; 39 (5): 204–10. [DOI] [PubMed] [Google Scholar]
  • 36. Wilbanks BA, Berner ES, Alexander GL, Azuero A, Patrician PA, Moss JA.. The effect of data-entry template design and anesthesia provider workload on documentation accuracy, documentation efficiency, and user-satisfaction. Int J Med Inform 2018; 118: 29–35. [DOI] [PubMed] [Google Scholar]
  • 37. Yee T, Needleman J, Pearson M, Parkerton P, Parkerton M, Wolstein J.. The influence of integrated electronic medical records and computerized nursing notes on nurses' time spent in documentation. CIN: Comput Inform Nurs 2012; 30 (6): 287–92. [DOI] [PubMed] [Google Scholar]
  • 38. Tsai SL, Sun YC, Taur FM.. Comparing the working time between Bar-Code Medication Administration system and traditional medication administration system: an observational study. Int J Med Inform 2010; 79 (10): 681–9. [DOI] [PubMed] [Google Scholar]
  • 39. Cornell P, Riordan M, Herrin-Griffith D.. Transforming nursing workflow, part 2: the impact of technology on nurse activities. J Nurs Adm 2010; 40 (10): 432–9. [DOI] [PubMed] [Google Scholar]
  • 40. Poon EG, Keohane CA, Bane A, et al. Impact of barcode medication administration technology on how nurses spend their time providing patient care. J Nurs Adm 2008; 38 (12): 541–9. [Google Scholar]
  • 41. Poon EG, Keohane C, Featherstone E, et al. Impact of barcode medication administration technology on how nurses spend their time on clinical care. AMIA Ann Symp Proc 2006; 2006: 1065. [PMC free article] [PubMed] [Google Scholar]
  • 42. Devine EB, Hollingworth W, Hansen RN, et al. Electronic prescribing at the point of care: a time-motion study in the primary care setting. Health Serv Res 2010; 45 (1): 152–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43. Pereira JA, Quach S, Hamid JS, et al. The integration of barcode scanning technology into Canadian public health immunization settings. Vaccine 2014; 32 (23): 2748–55. [DOI] [PubMed] [Google Scholar]
  • 44. Schenk E, Schleyer R, Jones CR, Fincham S, Daratha KB, Monsen KA.. Impact of adoption of a comprehensive electronic health record on nursing work and caring efficacy. CIN: Comput Inform Nurs 2018; 36 (7): 331–9. [DOI] [PubMed] [Google Scholar]
  • 45. Leung M, Chan KKC, Wong WL, Law A.. Impact of IPMOE on nursing tasks in the medical ward: a time-motion study. Int J Nurs Sci 2018; 5 (1): 50–6. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 46. Walker RM, Burmeister E, Jeffrey C, et al. The impact of an integrated electronic health record on nurse time at the bedside: a pre-post continuous time and motion study. Collegian 2019: doi:10.1016/j.colegn.2019.06.006. [Google Scholar]
  • 47. Schachner B, Gonzalez Z, Recondo F, et al. Post-implementation study of a nursing e-chart: how nurses use their time. Eur J Epidemiol 2016; 31 (Supplement 1): S172. [PubMed] [Google Scholar]
  • 48. Schachner B, González Z, Recondo F, et al. Post-implementation study of a nursing e-chart: how nurses use their time. Stud Health Technol Inform. 2016;228:638–42. [PubMed] [Google Scholar]
  • 49. Gomes M, Hash P, Orsolini L, Watkins A, Mazzoccoli A.. Connecting professional practice and technology at the bedside: nurses' beliefs about using an electronic health record and their ability to incorporate professional and patient-centered nursing activities in patient care. Comput Inform Nurs: CIN 2016; 34 (12): 578–86. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 50. Huang HY, Lee TT.. Impact of bar-code medication administration on nursing activity patterns and usage experience in Taiwan. CIN-Comput Inform Nurs 2011; 29 (10): 554–63. [DOI] [PubMed] [Google Scholar]
  • 51. Boeckx A, Mauws N, Colpaert K, Janssens B, De Jaeger A, Decruyenaere J.. The impact of an intensive care information system on nursing activities in the pediatric intensive care unit (pICU): work sampling before and after implementation. Intensive Care Med 2010; 2: S274. [Google Scholar]
  • 52. Banner L, Olney CM.. Automated clinical documentation: does it allow nurses more time for patient care? CIN: Comput Inform Nurs 2009; 27 (2): 75–81. [DOI] [PubMed] [Google Scholar]
  • 53. Asaro PV, Boxerman SB.. Effects of computerized provider order entry and nursing documentation on workflow. Acad Emerg Med 2008; 15 (10): 908–15. [DOI] [PubMed] [Google Scholar]
  • 54. Banet GA, Jeffe DB, Williams JA, Asaro PV.. Effects of implementing computerized practitioner order entry and nursing documentation on nursing workflow in an emergency department. J Healthc Inf Manag 2006; 20 (2): 45–54. [PubMed] [Google Scholar]
  • 55. McComas J, Riingen M, Chae Kim S.. Impact of an electronic medication administration record on medication administration efficiency and errors. CIN: Comput Inform Nurs 2014; 32 (12): 589–95. [DOI] [PubMed] [Google Scholar]
  • 56. Dwibedi N, Sansgiry SS, Frost CP, et al. Bedside barcode technology: impact on medication administration tasks in an intensive care unit. Hosp Pharm 2012; 47 (5): 360–6. [Google Scholar]
  • 57. Dwibedi N, Sansgiry SS, Frost CP, et al. Effect of barcode-assisted medication administration on nurses' activities in an intensive care unit: a time-motion study. Am J Health-Syst Pharm 2011; 68 (11): 1026–31. [DOI] [PubMed] [Google Scholar]
  • 58. Weiss D, Weiss Y.. The impact of introducing a clinical information system on ICU workflow. Crit Care Med 2011; 12: 29. [Google Scholar]
  • 59. Munyisia EN, Yu P, Hailey D.. Does the introduction of an electronic nursing documentation system in a nursing home reduce time on documentation for the nursing staff? Int J Med Inform 2011; 80 (11): 782–92. [DOI] [PubMed] [Google Scholar]
  • 60. Munyisia E, Yu P, Hailey D.. The effect of an electronic health record system on nursing staff time in a nursing home: a longitudinal cohort study. Australas Med J 2014; 7 (7): 285. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 61. Franklin BD, O’grady K, Donyai P, Jacklin A, Barber N.. The impact of a closed-loop electronic prescribing and administration system on prescribing errors, administration errors and staff time: a before-and-after study. BMJ Qual Saf 2007; 16 (4): 279–84. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62. Hakes B, Whittington J.. Assessing the impact of an electronic medical record on nurse documentation time. CIN: Comput Inform Nurs 2008; 26 (4): 234–41. [DOI] [PubMed] [Google Scholar]
  • 63. Burkle T, Castellanos I, Tech H, Prokosch HU.. Implementation of a patient data management system: an evaluation study of workflow alterations. Stud Health Technol Inform 2010; 160(Pt 2): 1256–60. [PubMed] [Google Scholar]
  • 64. Saarinen K, Aho M.. Does the implementation of a clinical information system decrease the time intensive care nurses spend on documentation of care? Acta Anaesthesiol Scand 2005; 49 (1): 62–5. [DOI] [PubMed] [Google Scholar]
  • 65. Lavander P, Meriläinen M, Turkki L.. Working time use and division of labour among nurses and health care workers in hospitals: a systematic review. J Nurs Manag 2016; 24 (8): 1027–40. [DOI] [PubMed] [Google Scholar]
  • 66. Kaushal R, Shojania KG, Bates DW.. Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 2003; 163 (12): 1409–16. [DOI] [PubMed] [Google Scholar]
  • 67. Bates DW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medication errors. JAMA 1998; 280 (15): 1311–6. [DOI] [PubMed] [Google Scholar]
  • 68. Reckmann MH, Westbrook JI, Koh Y, Lo C, Day RO.. Does computerized provider order entry reduce prescribing errors for hospital inpatients? a systematic review. J Am Med Inform Assoc 2009; 16 (5): 613–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 69. Edmondson AC, Tucker AL.. Managing routine exceptions: a model of nurse problem solving behavior. Adv Health Care Manag 87–113. [Google Scholar]
  • 70. Patterson ES, Rogers ML, Chapman RJ, Render ML.. Compliance with intended use of bar code medication administration in acute and long-term care: an observational study. Hum Factors 2006; 48 (1): 15–22. [DOI] [PubMed] [Google Scholar]
  • 71. McGinn CA, Grenier S, Duplantie J, et al. Comparison of user groups' perspectives of barriers and facilitators to implementing electronic health records: a systematic review. BMC Med 2011; 9 (1): 46. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 72. Bonkowski J, Carnes C, Melucci J, et al. Effect of barcode-assisted medication administration on emergency department medication errors. Acad Emerg Med 2013; 20 (8): 801–6. [DOI] [PubMed] [Google Scholar]
  • 73. Coyle GA, Heinen M.. Scan your way to a comprehensive electronic medical record. Nurs Manag 2002; 33 (12): 56–9. [DOI] [PubMed] [Google Scholar]
  • 74. Poon EG, Keohane CA, Yoon CS, et al. Effect of bar-code technology on the safety of medication administration. N Engl J Med 2010; 362 (18): 1698–707. [DOI] [PubMed] [Google Scholar]
  • 75. Chan YS, Liang HJ, Lin YH. IEEE using wireless measuring devices and tablet pc to improve the efficiency of vital signs data collection in hospital. In: 2014 IEEE International Symposium on Bioelectronics and Bioinformatics (ISBB) 2014 April 11th-14th, Chung Li, Taiwan; 2014.
  • 76. Lisby M, Nielsen LP, Mainz J.. Errors in the medication process: frequency, type, and potential clinical consequences. Int J Qual Health Care 2005; 17 (1): 15–22. [DOI] [PubMed] [Google Scholar]
  • 77. Hayes B, Bonner A, Pryor J.. Factors contributing to nurse job satisfaction in the acute hospital setting: a review of recent literature. J Nurs Manag 2010; 18 (7): 804–14. [DOI] [PubMed] [Google Scholar]
  • 78. Sendelbach S. Alarm fatigue. Nurs Clin North Am 2012; 47 (3): 375–82. [DOI] [PubMed] [Google Scholar]
  • 79. Bush NJ. Compassion fatigue: are you at risk? Oncol Nurs Forum 2009; 36 (1): 24–8. [DOI] [PubMed] [Google Scholar]
  • 80. Haddad LM, Toney-Butler TJ.. Nursing Shortage; StatPearls [Internet]. Treasure Island (FL): StatPearls Publishing 2019. Available from https://www.ncbi.nlm.nih.gov/books/NBK493175. Accessed January 20, 2019. [Google Scholar]
  • 81. Finkler SA, Knickman JR, Hendrickson G, Lipkin M, Thompson WG.. A comparison of work-sampling and time-and-motion techniques for studies in health services research. Health Serv Res 1993; 28 (5): 577–97. [PMC free article] [PubMed] [Google Scholar]
  • 82. Goodacre S. Uncontrolled before-after studies: discouraged by Cochrane and the EMJ. Emerg Med J 2015; 32 (7): 507–8. [DOI] [PubMed] [Google Scholar]
  • 83. Zheng K, Guo MH, Hanauer DA.. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research. J Am Med Inform Assoc 2011; 18 (5): 704–10. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

ocz231_Supplementary_Data

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of Oxford University Press

RESOURCES