Evaluating Outcomes of Electronic Tools Supporting Physician Shift-to-Shift Handoffs: A Systematic Review

Abstract

Background

Multiple organizations have recognized that handoffs are prone to errors, and there has been an increase in the use of electronic health records and computerized tools in health care.

Objective

This systematic review evaluates the current evidence on the effectiveness of electronic solutions used to support shift-to-shift handoffs.

Methods

We searched the English-language literature for research studies published between January 1, 2008, and September 19, 2014, using National Library of Medicine PubMed, EBSCO CINAHL, OvidSP All Journals, and ProQuest PsycINFO. Included studies focused on the evaluation of physician shift-to-shift handoffs and an electronic solution designed to support handoffs. We assessed articles using a quality scoring system, conducted a review of barriers and strategies, and categorized study outcomes into self-report, process, and outcome measures.

Results

Thirty-seven articles met inclusion criteria, including 20 single group pre- and posttest studies; 8 posttest only or cross-sectional studies; 4 nonrandomized controlled trials; 1 cohort study; 1 randomized crossover study; and 3 qualitative studies. Quality scores ranged from 3.5 to 14 of a possible 16. Most articles documented some positive outcomes, with 2 of the 3 studies evaluating patient outcomes yielding statistically significant improvements. The only other study that analyzed patient outcomes showed that interventions other than the electronic tool were responsible for most of the significant improvements.

Conclusions

The majority of studies supported using an electronic tool, yet few measured patient outcomes, and numerous studies suffered from methodology issues. Future studies should evaluate patient outcomes, improve study design, assess the role of faculty oversight, and broaden the focus to recognize the role of human factors.

Editor's Note: The online version of this article contains tables of research studies of electronic tools supporting shift-to-shift handoffs (January 2008 to September 2014) and a list of articles on development, implementation, and use of electronic tools for shift-to-shift handoffs.

Introduction

According to The Joint Commission, 63.5% of all sentinel events in 2013, and 62.9% in the first 6 months of 2014, involved miscommunication between health care providers.¹ Communication errors, especially those occurring during transitions of care from one provider to another, have been shown to cause preventable adverse events.^2–4 In the Agency for Health Care Research and Quality's 2014 Hospital Survey on Patient Safety Culture, 405281 staff members responded⁵; of these, 53% endorsed the statement that “important patient care information is often lost during shift changes,” and 47% endorsed the statement that “shift changes are problematic for patients in this hospital.”⁵

Several health care organizations have recognized this potential for error and the need for standardized handoff communication, including the Institute of Medicine,⁶ The Joint Commission,⁷ the World Health Organization,⁸ and the Accreditation Council for Graduate Medical Education (ACGME).⁹ Current ACGME Common Program Requirements state that residency programs “must ensure and monitor effective, structured hand-over processes to facilitate both continuity of care and patient safety.”⁹ Despite these calls for standardized, structured handoffs, a systematic review of resident and attending physician handoffs published in 2008 concluded that very little quality research had been done to identify best practices of any kind for conducting shift-to-shift handoff.¹⁰

Since 2008, handoffs have received more attention, and the use of electronic health records (EHRs) and computerized tools in health care has increased.^11–14 One strategy that has been suggested for improving quality of shift-to-shift handoff and helping meet standardization requirements is the use of electronic handoff tools.⁶ Therefore, we conducted a systematic review of the research literature evaluating the effectiveness of electronic solutions used for physician shift-to-shift handoffs.

Methods

Literature Search

Our team librarian (I.V.) conducted a systematic literature search of English-language articles published on electronic handoff solutions between January 1, 2000, and September 19, 2014, using National Library of Medicine PubMed, EBSCO CINAHL, OvidSP All Journals, and ProQuest PsycINFO. We chose relevant controlled vocabulary and keywords to capture the concepts of handoff and electronic solutions.

All titles were independently reviewed for inclusion by a team of 2 trained reviewers (L.A.R. and J.Da.). If either reviewer selected a reference, the full text was ordered for further review. Using this strategy, 349 articles were obtained for further review. The percentage of agreement on initial independent selection of articles was 97.8%. Interrater reliability using Cohen's κ was κ = 0.91 (P < .001). The reference sections of all included articles were checked by 2 independent research assistants for additional potentially relevant articles, as well as the reference sections of 2 systematic reviews identified during our search process (figure).^15,16 We also searched PubMed for other publications written by authors included in this review. We used each author's name combined with our handoff search terms.

FIGURE .

Process Used to Select Research Studies of Computerized Tools Supporting Shift-to-Shift Handoff

Inclusion and Exclusion Criteria

Articles meeting the following criteria were eligible for review: English language; published between January 1, 2000, and September 19, 2014; focused on physician shift-to-shift handoffs; had either quantitative or qualitative data; and focused specifically on the evaluation of an electronic tool. We excluded articles that were not handoff-specific (ie, discussed duty hours, general communication, ward rounds); that focused solely on the development, implementation, or use of a handoff tool; that addressed inter- or intrahospital transfer; and those with anecdotal or no data; along with letters to the editor, commentaries, editorials, and newsletter articles.

In our preliminary review, we noted that the majority of articles were published in 2008 or later. Considering advances in technology and the above finding, we decided to limit the review to articles published from January 1, 2008, forward.

Abstraction Process

Trained reviewers used an iterative process to develop an abstraction form designed to confirm final eligibility for full review, assess article characteristics, and extract data relevant to the study question. Each article was independently abstracted by 2 trained reviewers (M.M., B.B., or M.Y.). The 2 abstractors and an author independent to the abstraction process (J.Da. or L.A.R.) discussed and combined the 2 abstractions into a final abstraction. All abstraction disagreements were minor and were resolved during discussion.

Types of Data Reported

Organizing results according to the type of data reported can help to summarize what is known and what still needs to be studied. We categorized data reported into 3 types: self-report, process measures, and outcome measures. These were defined as (1) self-report measures, which ask subjects to report on their attitudes, beliefs, perceptions, and satisfaction; (2) process measures, which evaluate or assess activities conducted by health care providers (they describe actions performed); and (3) outcome measures, which evaluate or assess actual patient outcomes (they describe the patient's condition or response to care).

Barriers and Strategies

All barriers and strategies directly related to the electronic tool were identified by 2 independent reviewers (M.M. and B.B.). The 2 reviewers then met to compare lists and, through discussion, agreed on a final combined comprehensive list, which was reviewed by 1 author (J.Da.) for coherence and consistency.

Quality Scoring

One of the authors of this manuscript (L.A.R.) worked with a previous team to develop a quality scoring system that can be used to assess experimental and observational studies in the same systematic review, described in more detail in previous publications.^10,20

Quality scores were independently obtained from 2 reviewers (J.Da., M.Y., or L.A.R.). Overall interrater agreement was 85%, and Cohen's κ for agreement between the 2 reviewers was κ = 0.75 (P < .001). Any disagreements were resolved via discussion.

Results

We identified 37 unique articles with relevant research related to electronic tools designed to support physician shift-to-shift handoffs (box 1 and provided as online supplemental material). Of these, 9 (24%) used multiple outcome types, 7 (19%) used both self-report and process data,^21–27 1 used process and patient outcomes,²⁸ and 1 used self-report, process data, and patient outcomes.²⁹

box 1 Characteristics of Articles in a Systematic Review of Electronic Tools for Physician Shift-to-Shift Handoffs

Year Published

• 2 (5.4%) in 2008^33,41

• 5 (13.5%) in 2009^{17,25,26,49,50}

• 7 (18.9%) in 2010^{18,22,23,27,31,40,43}

• 7 (18.9%) in 2011^{32,39,42,46,52–54}

• 9 (24.3%) in 2012^{19,21,29,30,34,37,38,45,48}

• 4 (10.8%) in 2013^24,28,35,36

• 3 (8.1%) in 2014^44,47,51

Location of Study

• 24 (64.8%) in the United States^{18,19,21–24,26–28,30–35,37–41,44,47,51,53}

• 6 (16.2%) in the United Kingdom^{25,45,46,48,50,52}

• 2 (5.4%) in Australia^29,42

• 2 (5.4%) in Ireland^49,54

• 3 (8.1%) in Canada^17,36,43

Population(s) Studied

• 35 (94.6%) resident physician handoffs (or their equivalents in other countries)^{17–19,21–34,36–47,49–54}

• 4 (10.8%) nurses^19,33,42,50

• 8 (21.6%) attending physicians^{17,27,29,31–33,35,37}

• 8 (21.6%) midlevel providers (eg, nurse practitioners, physician assistants)^{17,19,32,35,37,43,50,51}

• 12 (32.4%) fellows^{17,29,32,34,37,42,43,47,49–51,54}

Specialties Involved

• 5 (13.5%) surgical specialties/subspecialties^{17,25,33,45,54}

• 22 (59.4%) medical specialties/subspecialties^{19,21–24,26–32,35,36,38,41,42,44,46,48,52,53}

• 10 (27.0%) involved both medical and surgical specialties^{18,34,37,39,40,43,47,49–51}

Study Designs

• 20 (54.1%) pretest/posttest designs^{17,22–25,29,31–34,36,41,42,45–50,54}

• 8 (21.6%) posttest only or cross-sectional^{26,27,35,37,39,43,44,51}

• 4 (10.8%) nonrandomized controlled trials^21,28,30,38

• 3 (8.1%) qualitative studies^18,19,52

• 1 (2.7%) randomized crossover study⁴⁰

• 1 (2.7%) cohort study⁵³

Quality Assessment Scores (range, 3.5–14.0; possible range, 1–16)

• 14 (37.8%) with scores less than 10^{23,27,29,36,37,39,42,44–46,48–50,54}

• 13 (35.1%) with scores from 10–11.5^{17,21,25,30–35,38,40,43,53}

• 10 (27.0%) with scores of 12–14.0^{18,19,22,24,26,28,41,47,51,52}

Self-Reported Measures

In 22% of the included studies, respondents reported increased or improved handoff content,^22,29–35 and 22% of studies also reported high provider satisfaction with use of an electronic handoff system.^{22–27,31,32} However, in 1 of these studies, 22.6% of respondents reported there was information they did not receive at handoff that would have helped them care for patients.²⁶ Other studies^{22,25,30,33,36,37} noted that respondents reported increased quality of handoffs with an electronic handoff system. In 1 study, residents used 3 different handoff protocols (written, electronic, and face-to-face), with reports of protocol deviations lowest in the face-to-face phase (28%), highest with written handoffs (67%), and electronic handoffs falling in the middle (50%).³⁸

In 9 studies (24%), perception of patient safety and/or quality of care improved^{22,27,31,36,39}; better patient management^21,25 or fewer near misses³⁰ were reported; and there was a reduction in missed patients.³⁴ However, a randomized crossover study found no statistically significant differences in resident-reported unexpected events, medical errors, or adverse drug events when comparing the electronic handoff system to the standard systems used (ie, written lists, card based, or a team-developed spreadsheet),⁴⁰ and another study found no difference in reported unexpected events pre- to postimplementation of a new electronic handoff system.⁴¹ Residents responding to a postcall survey reported an unexpected patient event that should have been anticipated during handoff in about one-third of patients in the old as well as the new system.⁴¹

Most studies that provided self-reported data on time devoted to activities related to handoffs showed reductions in time allocated to handoffs.^{23,27,31,34,39,42} In contrast, in 1 study respondents self-reported a decrease in time devoted to handoff preparation and an increase in the amount of time needed to update handoff information.³² Another study found conflicting results, with 37% of respondents reporting the electronic system was faster, and 57% reporting the card-based system (control) was faster to use.²¹ Yet another study found no significant difference between the number of patients handed off and the time spent during handoff; however, the respondents reported an increase in perceived efficiency and ease of use.²² In other studies questioning about use of the electronic system, 82% of respondents in 1 study³⁹ reported using the electronic system, and 84% of respondents in a second study²⁷ reported they would use the electronic system over other systems.

The implementation of a standardized, partially automated handoff system yielded a decrease in resident reports of inappropriate tasks being transferred to them at handoff.³³ In a different study, 78% of respondents reported that an electronic handoff system improved team communication.²⁵

One study included a survey of faculty physicians who provided oversight of the electronic resident handoff content; respondents estimated that corrections were required in 12% of handoffs, and that about 7% of those “represent[ed] a serious matter related to patient safety or quality of care.”²⁷

In 1 study, 70% of residents reported that the electronic handoff system helped them adhere to the 80-hour work week.³⁴ In another, respondents reported an increased ease in adherence with the 80-hour work week from 28% preimplementation to 50% postimplementation of an electronic inpatient handoff system.³¹ A survey of 39 departments at a pediatric hospital found that only 9 (23.1%) used an electronic handoff system.⁴³

Process Measures

Nineteen of 37 studies (51%) included process data, and of these, 16 (84%) documented actual increases or improvements in handoff content related to the implementation of an electronic solution.^{22,24–26,28,29,35,44–52} One study found that 92% of errors were committed in a core data element and could have been prevented with auto-population.⁴⁴ While many studies with process outcomes specifically showed fewer data omissions,^{24,25,28,29,34,44–47,49–52} 1 found that less than 50% included allergies or code status when using the electronic handoff system.²⁶ An observational study found that that only 11% included “Do Not Resuscitate” or advance directive information, and only 50% of successive handoffs were updated daily.⁵³ Another study found that 96% of electronic handoffs were updated within 24 hours,⁵¹ while a third study found an increase in the documentation of 12 of 16 specific data elements studied, a decrease in 3 elements, and no difference in 1 element.⁴⁸ Three studies noted that an electronic handoff tool should include free-text options for current anticipatory guidance and comments,^25,35,48 and another reported that users need to be encouraged to be vigilant in revising and updating free-text sections.²⁹

In a study that used faculty estimates for inaccuracies in handoffs and data from actual reviews, when attending physicians provided oversight of the content of residents' electronic handoffs they reported that 7% contained inaccuracies.²⁷ A nonrandomized control study found that by providing ongoing feedback on allergy and code status errors and omissions, there was a sharp decline in these errors, with the electronic group outperforming a card-based handoff system.²¹

Patient Outcomes

Three studies assessed patient outcome measures. One measured general medical emergency team calls and weekend discharges 12 months prior to and 12 months after implementation of an electronic handoff system, finding a decrease in calls (P = .01) and an increase in weekend discharges (P < .001).²⁹ Another study found a significant decrease in median length of stay (P = .05) 2 weeks after implementation of an electronic handoff system.⁵⁴ Starmer et al²⁸ conducted a large nonrandomized control trial investigating the implementation of a resident handoff bundle (standardized communication, handoff training, a verbal mnemonic, and a new team handoff structure). In this study, 1 unit also implemented a computerized handoff tool integrated into the EHR.²⁸ Medical errors decreased on both units studied; however, the decrease was greater on the unit that did not have an electronic handoff tool, and the decrease in preventable adverse events was statistically significant only for the unit that did not have an electronic handoff tool.²⁸

Barriers and Strategies

Twenty-one studies (57%) noted barriers to successful implementation of an electronic handoff system (box 2). Sixteen studies (43%) identified strategies that helped them to successfully implement an electronic system (box 2).

box 2 Barriers and Strategies to Implementation of Electronic Physician Shift-to-Shift Handoff System

Barriers

• Tool's inability to automatically keep up-to-date or requiring manual updates^17,29,33,53

• Persistence of inaccurate data^{19,29,33,34,37,44}

• Clinician resistance to change^31,46,47,50

• Perception that the handoff tool might interfere with direct physician communication^26,40,41

• Duplication of work or more work^17,32

• Errors created by free-text entry⁵⁴

• Lack of provider training⁴⁶

• Access issues⁴⁸

• Computer problems²⁴

• Cost³³

• Lack of integration with EHR³⁰

• Lack of specialty-specific format⁵¹

Strategies

• Collaborating with key stakeholders during development and implementation, with continuous feedback and improvement^{24,25,29,34,36,37,43}

• Providing adequate training^{23,25,36,37,42,51}

• Integrating into the EHR^{28,31,37,40,44}

• Keeping tool simple, reliable, and consistent with existing software^25,43

• Having clinician champions²⁹

• Balancing maximum information with the need for readability and highlighting critical information⁴¹

• Providing frequent updates and encouraging users to delete old entries^29,37

• Posting a printed version of the checklist where handoffs occur⁴⁷

• Communication about handoff changes⁴⁷

• Specialty-specific templates³⁷

Electronic Handoff Use and Development

We identified 23 articles that reported on the development, implementation, or use of an electronic handoff tool (provided as online supplemental material).

Discussion

Our systematic review identified 37 articles describing electronic tools designed to support physician shift-to-shift handoffs, published between January 1, 2008, and September 19, 2014. All but 2 of the studies (95%)^24,35 were conducted to evaluate shift-to-shift handoffs in physicians in training. Academic medical centers appear to be conducting the majority of the research on handoffs. This highlights the importance of graduate medical education in the development of handoff systems and future research on this important topic.

Inadequate communication during transition of care from one provider to another has been shown to cause adverse patient events.^2–4 Numerous organizations and accrediting bodies have called for better communication and improved handoffs. The ultimate goal is to improve patient outcomes. Unfortunately, only 3 of the identified studies^28,29,54 collected patient outcomes data, and these studies demonstrated conflicting results. Two pre-post studies^29,54 with quality scores less than 10 documented positive patient outcomes; however, 1 nonrandomized control study²⁸ with a quality score of 14 failed to demonstrate improvements in patient outcomes. In addition to assessing patient outcomes, including multiple types of outcomes in a study is a way to increase the strength of the evidence. Yet, we identified only 9 studies (24%) that included multiple outcomes.

Only 1 of the 37 studies in our review used a randomized crossover design,⁴⁰ and 4 (11%) used nonrandomized control groups.^21,28,30,38 A recent editorial notes that “randomization is not the ‘gold standard’ for medical education research,”⁵⁵ but it emphasizes that use of “. . . a comparison group is essential in education research.”⁵⁵ Future studies should strive to include a comparison group.

Electronic solutions used to support handoffs may be a way to standardize handoff information. Indicators of standardization include process measures that evaluate content, data accuracy, and data omissions. The majority of studies with process measures (16 of 19, 84%) support the assertion that the use of an electronic handoff tool can result in improved content quality. However, studies that have evaluated both content quality and patient outcomes have shown conflicting results on whether information transfer correlates with improved patient outcomes.^28,29 One reason could be that an information-centric view focused on information completeness, accuracy, and the transfer of information may fail to consider other important factors.⁵⁶ Some of these factors include culture, environment, effective teamwork, leadership skills, situational awareness, task management, relationships, and human interactions with the environment and equipment.^56,57 Future research on the utility of electronic handoff tools should consider factors other than information completeness and accuracy.

Relying on humans to copy information is fraught with opportunities for errors, and 49% of the studies in this review pulled at least some data for the electronic handoff tool from the EHR (provided as online supplemental material). However, it was not always clear how much data came directly from the EHR and how much was input by users; it is also possible that the remaining studies include some data that had been pulled from the EHR.

Data supporting best practices on handoffs remain lacking. Based on the results of our review, we encourage those designing electronic handoff tools to optimize the amount of data pulled from existing records, eliminating error introduced by humans retyping information; include key stakeholders in design efforts; provide plenty of opportunity for user feedback and status updates; and include adequate training. In addition, the handoff tool should include free-text options for current anticipatory guidance and comments, and a system to ensure these sections are regularly updated.

Faculty oversight, review of handoff content, and feedback resulted in statistically significant content error reductions and correction of content that could have led to a serious patient safety issue; therefore, we recommend the use of faculty oversight and real-time feedback of resident handoffs in medical education.

The current study is limited by the search strategy used. Specifically, the selected search terms may not have included all relevant terms. However, we strengthened the quality of our systematic review by developing a study protocol at the outset, with an explicit search strategy and clear inclusion/exclusion criteria. Although our strategy minimizes the risk of missing germane articles, it does not eliminate the possibility.

Several studies used multiple interventions in addition to implementing an electronic tool, and most included some form of education/training on handoffs and/or the electronic tool. In addition, 2 studies used oversight or feedback, 1 facilitated face-to-face handoffs, and another implemented a new handoff process along with the electronic tool. The simultaneous implementation confounds the results, making it difficult to attribute changes to the electronic handoff system alone.

Another limitation is the potential for publication bias, resulting from the possibility that high-quality studies with negative results may not have been published and that many quality improvement projects are not published in peer-reviewed journals.⁵⁸

Conclusion

Our systematic review identified 37 peer-reviewed studies of electronic handoff tools. Most studies used self-report and process outcome measures, which are surrogates for actual patient outcomes. The majority of studies had results supporting electronic handoff tools, including 2 of the studies with patient outcomes. A third study that analyzed patient outcomes showed that interventions other than the electronic tool were responsible for most of the significant improvements. Overall, there remains a paucity of evidence supporting best practices for electronic handoff tools designed to support physician shift-to-shift handoffs.

Future research should (1) utilize high-quality study designs, including a comparison or control group whenever possible; (2) evaluate patient outcomes and, if that is not possible, include a combination of both self-reported and process measures; (3) investigate further the role of faculty oversight in resident handoff improvement; and (4) include measures of the sociocultural aspects of handoffs when using an electronic handoff tool.