Impact of an electronic handoff documentation tool on team shared mental models in pediatric critical care

Abstract

Objective

To examine the impact of the implementation of an electronic handoff tool (the Handoff Tool) on shared mental models (SMM) within patient care teams as measured by content overlap and discrepancies in verbal handoff presentations given by different clinicians caring for the same patient.

Materials and Methods

Researchers observed, recorded, and transcribed verbal handoffs given by different members of patient care teams in a pediatric intensive care unit. The transcripts were qualitatively coded and analyzed for content overlap scores and the number of discrepancies in handoffs of different team members before and after the implementation of the tool.

Results

Content overlap scores did not change post-implementation. The average number of discrepancies nearly doubled following the implementation (from 0.76 discrepancies per handoff group pre-implementation to 1.17 discrepancies per handoff group post-implementation); however, this change was not statistically significant (p = 0.37). Discrepancies classified as related to dosage of treatment or procedure and to patients’ symptoms increased in frequency post-implementation.

Discussion

The results suggest that the Handoff Tool did not have the desired positive impact on SMM within patient care teams. Future electronic tools for facilitating team handoff may need longer implementation times, complementary changes to handoff process and structure, and improved designs that integrate a common core of shared information with discipline-specific records.

Conclusion

While electronic handoff tools provide great opportunities to improve communication and facilitate the formation of shared mental models within patient care teams, further work is necessary to realize their full potential.

Graphical abstract

Overall, there were no significant changes in either overlap scores or discrepancy rates after implementation of Handoff Tool. However, teams had almost twice the rate of discrepancies post-implementation. Four teams in the post-implementation phase contributed 36% of all discrepancies identified.

1. Introduction

Teamwork is an essential component of modern medicine[1], and successful teamwork is strongly tied to improved patient outcomes[2,3]. Many factors can impact clinicians’ ability to work together[4]; these include team orientation[5,6], an open leadership style[7–9], and unrestricted communication[10,11]. Previous research also indicated that successful teamwork depends on the team members’ ability to develop and maintain a shared understanding of task requirements, procedures, and role responsibilities as well as the context of their work, often referred to as a team’s shared mental model (SMM, Figure 1)[12]. Previous research on teamwork showed that teams with better developed SMMs and a higher degree of shared understanding of their tasks and responsibilities demonstrated higher levels of performance in a variety of domains[13–15]. In the clinical context, SMMs were described as a foundation of clinical teamwork[16].

Figure 1.

Shared Mental Model (SMM) within a patient care team.

1.1. Shared Mental Models, Clinical Handoff, and Electronic Handoff Tools

Communication and, in particular, interdisciplinary communication, has been identified as a critical factor for facilitating clinical teamwork and for the development of SMMs among team members[13,17]. Gaps in communication among members of interdisciplinary patient care teams can lead to adverse events and patient harm[18,19]. Clinical handoffs, such as those that occur during shift change, have been shown to be of particular importance to the formation of SMMs [20]. Studies showed that information transferred during handoffs serves as the foundation of patient care during the following shift[21,22]. Similarly, researchers argued that handoffs help members of interdisciplinary care teams establish a shared understanding of patient care needs and to coordinate responsibilities[23,24].

Yet multiple studies suggested that miscommunication occurs frequently during handoff, thus jeopardizing a team’s ability to form robust SMM, and contributing to errors and adverse events[25]. Because clinical handoffs are typically conducted as individual conversations between clinicians in the same role, they can contribute to divergence, rather than alignment among members of patient care teams, for example, in cases when handoffs by different clinicians on a team include inconsistent or contradictory information. Because of the increased attention to the critical role of handoffs, many interventions have been introduced to improve their quality. Examples of previous interventions include handoff training[26–28], standardizing the approach to verbal handoff using mnemonics[29,30], and electronic handoff documentation tools[31,32]. However, few of the previous initiatives explicitly targeted improvement in SMMs among members of patient care teams, or studied the impact of these interventions on clinical teamwork.

Previous studies of electronic handoff documentation tools suggested that these tools are increasingly adopted not only by their intended users (typically nurses or residents), but also by other members of patient care teams [33]. This represents an important contrast to the conventional practice in which handoff documentation is shared exclusively with clinicians of a similar role and discipline on a patient care team. Because new electronic tools allow clinicians in different roles to access the same condensed set of information documented for facilitating handoff, they have the potential to increase alignment among team members. We hypothesize that such tools can help teams to develop and maintain a more synchronized shared understanding of a patient’s conditions and care, as well as expectations in regards to care responsibilities—SMM—as compared to the typical discipline-specific handoff documentation practices (Figure 2).

Figure 2.

Shared access to the same handoff documentation via a shared electronic handoff tool can lead to richer SMM within patient care teams.

1.2. Shared Mental Models and Handoff Content Overlap

Previous research introduced a wide variety of techniques for empirically studying and quantifying SMMs within teams. Despite their differences, most of the traditional methods for quantifying SMMs rely on the same core strategies: 1) eliciting members’ knowledge and perspectives on issues pertaining to their tasks and 2) quantifying the degree of content overlap in these perspectives among team members. Higher degrees of content overlap are usually taken as an indication of richer and better-developed SMMs. While some researchers argued for the importance of complementary rather than overlapping mental models[34],[35], there is much evidence to suggest that a shared core of common understanding is essential to effective teamwork[36]. Given that SMMs and mental models in general are not directly observable, many previous methods have incorporated knowledge elicitation techniques and often required prolonged engagement of team members outside of their direct job responsibilities[37–39]. These approaches, while well-established outside of healthcare, may not be feasible in clinical settings, particularly in the context of intensive care, as they could place a high burden on clinicians and introduce additional disruption to their already highly interrupted work[40].

To address these limitations, we previously introduced a novel method for measuring teams’ SMMs by quantifying the degree of content overlap in verbal handoffs of different clinicians caring for the same patient[41]. We further argued that better aligned teams with richer SMMs have a higher degree of content overlap in their handoffs. This would mean, for example, that a team for which verbal handoffs by all team members included an overlapping core set of observations, issues, and actions related to patient care is more likely to have a better-developed SMM than a team that included a non-overlapping or even contradictory set of observations, concerns, and actions.

This method focuses exclusively on the analysis of naturally occurring discourse and does not account for common ground among team members that includes their general medical knowledge, as well as previously discussed information (in encounters not observed by researchers) regarding a specific patient (Figure 3)[42]. Moreover, there exist no current guidelines as to the desired degree of content overlap in handoffs by different clinicians. This can only be established empirically across studies. However, the method has a number of advantages comparing to the traditional approaches to measuring SMMs. It places low burden on clinicians and has higher ecological sensitivity due to its reliance on the naturally occurring discourse. It produces measures of SMMs comparable with those captured by the more traditional ways of assessing teams’ SMMs[14]. Other researchers have argued that content similarity in verbal handoffs is a good representation of shared understanding among members of patient care teams[43]. This method also places particular focus on discrepancies between team members, which may be indicative of gaps in understanding and miscommunication. Finally, while the desired degree of content overlap has not yet been established, we can use this measure for comparative purposes, for example, to measure change in the degree of SMM as a result of an intervention.

Figure 3.

Handoff content overlap and common ground.

In this paper, we report on the results of a non-randomized, pre-post observational study that examined the impact of a custom-built electronic handoff documentation tool (Handoff Tool) on SMMs among members of interdisciplinary patient care teams in a Pediatric Intensive Care Unit (PICU) at an urban teaching hospital as measured by the content overlap scores and the number of discrepancies in patient presentations from different clinicians caring for the same patient. Our hypothesis in this study was that the introduction of the Handoff Tool will lead to a higher degree of content overlap, and by extension, better developed SMMs and a higher degree of synchronization among team members.

MATERIAL AND METHODS

Setting

This study was conducted in a 3-unit, 41-bed pediatric intensive care unit (PICU) of a large urban teaching hospital in New York City during the fall of 2011 and winter and spring of 2012. At the time of the study, the unit provided care for about 1,800 patients annually. The PICU provided care for patients from birth to 18-years old, who had a range of acute and/or chronic issues of varying severity. The unit was staffed by attending physicians, house physicians, fellows, residents, nurse practitioners, and bedside nurses. It was divided into two physical sections, with one attending physician managing all patients in each section. Each section had a dedicated fellow who oversaw care for all patients under the attending physician. On average, there were 5 residents who rotated through the unit each month. Residents equally divided patient care responsibilities amongst themselves during the handoff process. Bedside nurses cared for one or two patients during each shift.

Handoff Workflows

The handoff workflows varied based on clinicians’ roles. Most handoffs included a verbal conversation between two clinicians of the same role (i.e. resident to resident handoff). The majority of handoffs were performed independently and without supervision. Nursing handoffs were usually performed at the patient’s bedside at 7am and 7pm. Resident handoffs usually occurred in one of two designated resident work areas. Morning resident handoffs occurred consistently at 7am; however, evening handoffs were more variable and generally took place in the late afternoon. Fellows had the least structured handoff process; their handoffs occurred at different times and locations depending on circumstances.

The staff in the PICU used a commercial electronic health record system (Allscripts Sunrise, Allscripts Corp., Chicago, IL). The system included a number of modules, separated into tabs, such as Results, Flowsheets, and Orders, among several others. All physician documentation was done electronically, with the exception of handoff notes. Prior to the implementation of the Handoff Tool, residents used structured paper-based handoff sheets. The document was organized by bodily systems. The outgoing resident filled out the sheet with pertinent information and passed the document to the incoming resident. Fellows used brief, free-form hand-written notes to facilitate handoff. Nursing handoff documentation included free-form handwritten handoff notes that were passed from the outgoing to the incoming nurse.

Handoff Documentation Tool

The Handoff Tool examined in this study was originally built for adult internal medicine resident physicians using an iterative, user-centered design process[32]. It was integrated with and accessed through the EHR and was comprised of a series of free-text boxes for data entry. Each of the text fields was labeled. At the time of the study, the labels included: 1) Patient Summary, 2) Active Issues, 3) Contact Info (for the patient), 4) Primary To Do List, 5) Notes/Comments, 6) Coverage To Do List, 7) Hospital Course, 8) Discharge Planning, and 9) Consult Handoff. However, adherence to documenting based on these labels was not enforced, and users could enter information about their patients as they saw fit. Additionally, the Handoff Tool included features for creating a print version of the handoff document. This printout included a cover page with a list of all patients under the clinician’s care and subsequent pages with detailed information about each patient. This included information from the free-text boxes within the Handoff Tool as well as structured EHR data, such as medication orders and laboratory results.

The Handoff Tool was implemented for use in the PICU in February 2012. All physicians in the unit received brief training on the use of the Handoff Tool. Since the Handoff Tool was primarily designed for residents, updating the information in the handoff document was considered the responsibility of the residents in the unit. At the time of the study, residents were free to choose whether they wished to use the tool or the structured paper document. Residents also updated the information in the tool as frequently as they saw fit. Nurses could view the information stored in the Handoff Tool but could neither edit existing notes nor create new ones. The use of the Handoff Tool for any clinician was not regulated or mandated by members of the unit leadership.

To the best of the authors’ knowledge, no other handoff or communication interventions occurred during the study period.

Data

Verbal handoffs by residents, fellows, and bedside nurses were recorded during November and December of 2011 (pre-implementation of the Handoff Tool) and between March and May of 2012 (post-implementation of the Handoff Tool). Handoffs were recorded during both morning and evening shift-changes. Shortly before each shift change, members of the research team (DK, LM) requested permission to record handoff conversations for specific patients. Verbal handoffs were recorded, professionally transcribed, and verified for accuracy by members of the research team (SJ, AM).

Overall, there were 50 handoff groups observed in the study. These groups included 133 handoffs, of which 30 were from fellows, 46 from residents, and 47 from nurses. Twenty-one handoff groups were recorded prior to the implementation of the Handoff Tool, and twenty-nine handoff groups after the implementation. Almost half of the handoff groups included handoffs by all three team members (23 out of 50). The remaining 27 groups included only two team members each, either a resident and a nurse (n = 19), a fellow and a nurse (n =5), or a fellow and a resident (n = 3).

This study was approved by the Institutional Review Board of the Columbia University Medical Center (CUMC). All participating clinicians signed the informed consent form prior to participation. Since patients were not actively involved in the study, patient consent was waived.

Content Overlap and Discrepancy Analysis

The transcripts were analyzed in a multi-step approach, described extensively in Mamykina et al.[41]. First, all handoff transcripts were organized into team handoff groups by their clinical team, date, and time; each group represented a set of verbal handoffs given by different clinicians caring for the same patient during a single transition of care. Then, each transcript was divided into the smallest segments, each with a coherent meaning and a single unit of information to reduce ambiguity during overlap analysis. For example, the statement “The plan is to wean over the next 12 hours” was divided into two segments: 1) “the plan is to wean” and 2) “over the next 12 hours”. In this scenario, the first segment relates to the plan (to wean), while the second segment provides more information or context about the plan (temporal frame). Subsequently, all transcripts within each handoff group were compared to each other, statement by statement, on their content overlap. Statements that were deemed overlapping in both the subject and the corresponding modifier were identified as being an “overlap”. Importantly, the statements were compared on their semantic, rather than syntactic, similarity; if two statements included different syntactic structures but conveyed the same meaning, they were marked as overlap. For example, such statements as “[She was on nitric for a while], but that’s been weaned to off” (in a resident’s handoff), and “She’s off [nitric]” (in a nursing handoff) were considered overlapping because while these statements are syntactically different, they nonetheless include the same subject (nitric) and the same modifier (weaned to off).

In addition to overlaps, we identified all instances that referred to the same subject but included different modifiers and labeled these as discrepancies. Again, our focus here was not on linguistic discrepancies in sentence structures or choice of wording, but on factual meaning. For instance, using the example above, if a resident’s handoff included a statement: “[She was on nitric for a while,] but now she has a lower dose” and a handoff of a nurse on the same team given during the same transition of care included a statement: “She’s off nitric”, this would be considered a discrepancy because the subject (nitric) in both handoffs was the same, but the modifiers (one stating that the patient was still on nitric, while the other stating that the patient was not on nitric) were different. Notably, not all discrepancies included opposing or contradictory modifiers; for some, modifiers simply did not align. For example: a resident saying “She needed just one additional [Versed bolus overnight]” and a nurse saying “There was a couple of times [last night that I ended up asking for like extra Versed)”.

Further, all the factual differences identified by the coders (SJ and AM), were assessed on their clinical significance by an attending physician from the same PICU (SH). The discrepancies were presented in a random order and the attending physician was blinded to which discrepancies were captured during the different study phases (pre and post implementation). All discrepancies that were deemed clinically insignificant were discarded and not included in the subsequent analysis. For instance, a discrepancy between a resident stating that a 16-week old patient’s drain output was 600 mL and a fellow stating that the patient’s drain output was 200 mL was considered clinically significant. In contrast, a discrepancy between a blood pressure report of 70–80mmHg by a nurse and a blood pressure report of 65–80mmHg by a resident was not considered clinically significant.

It should be noted that lack of overlap and discrepancies are not equivalent. Lack of overlap can be more ambiguous in its relation to shared mental models and may indicate divergence in priorities regarding different aspects of a patient’s conditions and care, rather than misalignment in knowledge and understanding. In contrast, discrepancies specifically signify instances of disagreement or difference in understanding of a patient’s condition or care plan.

To calculate the content overlap scores, we used an approach adapted in our previous work from the Pyramid method proposed by Nenkova et al, in the context of natural language processing research on text summarization[44]. In this method, each team handoff group is allocated a pyramid that indicates their content overlap. All statements in the handoffs within the group are placed into tiers of a pyramid, where each tier corresponds to how many documents (here handoff transcripts) the statement appeared in. All statements that appeared only in a single handoff are assigned a weight of zero and placed in the bottom tier of the pyramid (T₁). For the rest of the tiers (T_i), the weight of the statements in each tier is calculated as the ratio of the number of times this statement appeared in handoff discussions i, to the maximum overlap n, which corresponds to the number of handoffs in the group. With this approach, statements included in all handoff discussions receive a perfect score of 1; statements that are unique to a single handoff receive a score of 0, and the rest of the statements receive a score between 0 and 1, depending on how many clinicians included them into their handoffs. Below is the formula we used to calculate the total overlap index O for the pyramid; here |T_i| denotes the number of statements in tier T_i and the total overlap index O for the team is calculated as the ratio of the actual weight of the pyramid for that team during a specific transition of care to its maximum weight, achieved if there was a perfect overlap on all statements for all team members:

$$\mathcal{O}=\frac{{\sum }_{i=2}^n\frac{i}{n}\times \mid Ti\mid }{{\sum }_{i=1}^n\mid Ti\mid },$$

A more detailed description of the method is available elsewhere[41,45].

To assess inter-rater reliability, 5 handoff cycles (10% of the total dataset) were coded independently by a clinically trained rater (a student pursuing a Ph.D. in nursing). The rater began by conducting shared coding exercises (breaking transcripts into segments, and identifying overlapping and discrepant segments on one sample cycle), comparing results with the original coding and resolving disagreements in discussions with the original coders. After that, the independent coder was asked to review 5 handoff cycles, break them into individual segments, and identify overlapping and discrepant segments independently. The number of segments that matched between the additional rater and the original raters, as well as the number of segments marked as overlapping and discrepant were counted, with disagreements noted. We calculated agreement rates for the identified segments, and used Cohen’s kappa score for content overlap scores and discrepancies, separately.

Categorizing Discrepancies

All discrepancies were categorized on their content and language structure based on a taxonomy of handoff communication adapted from Apker et al[46]. Although Apker’s taxonomy was originally developed for handoffs that occurred as part of patients’ transition from the Emergency Room (ER) to other units in a hospital, it proved to be sufficiently applicable for handoffs in Critical Care Units, including the PICU in this study[41]. However, the taxonomy was modified to account for differences in handoff communication between the ER and PICU and to allow for a more granular classification. Specifically, under the original category “Assessment/Treatment” we included subcategories referring to treatment administration (if a particular treatment or medication was administered), and dosage (the actual dosage of administered treatment) because of the high frequency of discrepancies in these categories. In addition, we changed the definition of the category “Outcome” to include statements regarding outcomes (or results) of past treatments or procedures (e.g. “the patient tolerated weaning well”) from its original meaning of outcome of handoff (e.g., accept or not accept to the unit) and “Assessment/Transfer of Responsibility” to include any specific instructions in regards to the anticipated care for the upcoming shift (instead of its original meaning of “Statements about what was being asked of the hospitalist (e.g., patient admission, clinical consult, other reason)”). All the other categories and sub-categories remained unchanged. Table 1 gives a list of the possible discrepancies and operational definitions.

Table 1.

Discrepancies Coding Schema and Definitions

Category of Discrepancies	Subcategory of Discrepancies	Operational Definition*
Patient Presentation	Patient Identifiers	“Statements that convey patient’s specific room location or name.”
	Symptom	“Descriptions, explanations that address current clinical scenario in regard to providing information about symptoms of concern (medical or psychosocial).”
	History	“The patient’s past medical problems/conditions that are pertinent to the current diagnosis or clinical impression. Includes events that occurred before patient presented to [hospital]. Patient comments about history. Physician comments about patient personality, lifestyle, [and] demeanor.”
	Procedure (administration)	Statements about whether laboratory orders, medications, and evaluations have already been performed.
	Procedure (dosage)	Statements about how often laboratory orders, medications, and evaluations should be performed.
Assessment	Treatment	“Statements about future medical procedures to be taken, if such steps are deemed necessary, to address the patient’s current problem.”
	Clinical Impression	“Identification of the current clinical impression, naming the problem or reasons for the problem.”
	Prognosis	“Probabilistic statement about patient’s future condition, based on completed or proposed treatments.”
	Outcome	Statements regarding outcomes of different treatments and procedures.
	Transfer of Responsibility	Specific instructions in regards to the anticipated care for the upcoming shift.

^*Definitions provided in quotes were directly adopted from Apker et al[46]

Modifications to the original taxonomy are included without quotes.

Statistical Analysis

The content overlap scores and the number of discrepancies were calculated for each handoff group. Differences in content overlap scores and the number of discrepancies before and after the implementation of the Handoff Tool were evaluated on statistical significance using both Wilcoxon-Rank sum (non-parametric) and two-sample t-tests. The distributions of content overlap scores and discrepancies were tested for normality using the Shapiro-Wilks test. To examine the potential impact of missing data (when the researchers were unable to capture one of the handoffs in a given team), we compared differences in content overlap scores between handoff groups with different role composition (i.e. a resident and a nurse team vs a resident and a fellow team) using the Kruskal-Wallis test. This also helped to test whether there is a higher degree of alignment among clinicians of the same discipline (e.g., residents and fellows) as compared to interdisciplinary groupings (e.g. residents and nurses). Finally, because changes in content overlap scores and discrepancies between study phases may be associated with the corresponding changes in the number of statements in handoffs (arguably, longer handoffs have a higher potential for both content overlap scores and discrepancies), we used Wilcoxon Rank-Sum test to test whether there was a difference in the number of statements in handoffs before and after the implementation of the tool.

RESULTS

The content overlap scores captured in the study ranged from 0.002 to 0.12; the average content overlap score of all handoffs across study phases was 0.06 (Table 1). Across the study phases, there were three distinct categories of handoffs based on their content overlap scores. The low content overlap score category included handoff groups with content overlap scores between 0 (minimum theoretical score) and 0.028; this category included 6 handoff groups. The middle content overlap score category included handoff groups with content overlap scores between 0.042 and 0.082; this category included 40 handoff groups. Lastly, the high content overlap score category included handoff groups with content overlap scores between 0.094 and maximum content overlap score in our study, 0.12; this category included 4 handoff groups. The average content overlap scores before and after the tool was implemented were 0.06 and 0.06, respectively. There was no statistical difference in content overlap scores before and after the tool was implemented (W = 288, p = 0.75).

There were no differences in overlap scores between groups with different team composition (X² = 4.3236, df = 3, p = 0.2286). The average content overlap score for handoff groups that included handoffs of all three team members was 0.06. The content overlap scores of handoff groups consisting of a resident and a nurse had the highest mean overlap score with 0.07. The average content overlap score for groups with fellows and residents were 0.06, and the average content overlap score for groupings of fellows and nurses were 0.04. These results show that groups that included handoffs from fellows and residents on average did not have a higher overlap score than groups that included nursing handoffs.

Overall, 50 discrepancies were identified. The average number of discrepancies per handoff group before and after the Handoff Tool was implemented was 0.76 and 1.17, respectively, with an overall average of 1 discrepancy per handoff group across both study periods. In the pre-implementation phase, teams had a maximum of 2 discrepancies in their handoffs, this increased to a maximum of 5 discrepancies in a given handoff after the implementation of the Handoff Tool. The difference in the average number of discrepancies per handoff was not statistically significant between pre- and post-implementation phases when analyzed using two-samples t-test (t = −1.41 df = 46.5, p = 0.1656), or the non-parametric Wilcoxon-Rank Sum test (W = 261, p = 0.3693). The Shapiro-Wilk test showed that the number of discrepancies before and after the implementation of the Handoff Tool did not follow a normal distribution (p < 0.01, p < 0.01 respectively). Table 2 shows proportions of teams that experienced different numbers of discrepancies before and after the implementation.

Table 2.

Descriptive Statistics of Content Overlap Scores and Numbers of Discrepancies Between Study Phases

	Overlap		Discrepancies
	Pre	Post	Pre	Post
(Min, Max)	(0.006, 0.08)	(0.002, 0.12)	(0, 2)	(0,5)
Mean	0.06	0.06	0.76	1.17
Median	0.06	0.06	1	1
Standard Deviation (SD)	0.02	0.03	0.72	1.6

The average length of handoffs did not change as a result of the implementation of the Handoff Tool. Prior to the implementation of the tool, handoffs had on average 509.33 segments; the average number of segments in handoffs following the implementation of the Handoff Tool was 563.28 (W = 255.5, p = 0.3404). This suggests that the increase in discrepancies was not due to a change in the length of handoff conversations.

The categories of discrepancies with their distribution and representative examples are presented in (Table 3). Most discrepancies were categorized as Patient Presentation, which included several subcategories. There was a slight increase in the rate of discrepancies in the Patient Presentation category after the implementation of the Handoff Tool (from 0.71 discrepancies per handoff group pre-implementation to 0.93 discrepancies per handoff group post-implementation). There was also a change in the number of discrepancies categorized as Assessment, since these types of discrepancies were only identified after the implementation of the Handoff Tool.

Table 3.

Proportions of teams with different frequencies of discrepancies

	Proportion of teams with a given number of discrepancies
Number of discrepancies per team	Pre: proportion (n); total n=21	Post: proportion (n); total n=29
0	43% (n=9)	34% (n=10)
1	38% (n=8)	38% (n=11)
2	19% (n=4)	14% (n=4)
3	0	7% (n=2)
4	0	3% (n=1)
5	0	3% (n=1)

There was a high inter-rater reliability between the independent rater and the study raters. In segmenting the statements, the raters agreed 74.8% of the time. The study raters disagreed with the independent rater 14.3% of the time, and the independent rater disagreed with the study raters 10.9% of the time. In identifying overlaps, the independent rater and study raters achieved a kappa of 0.93 (confidence interval: 0.91, 0.95). In identifying discrepancies, the independent rater and study raters achieved a kappa of 0.69 (confidence interval: 0.49, 0.88).

DISCUSSION

This study investigated the impact of implementing the Handoff Tool on shared mental models within interdisciplinary patient care teams as measured by the degree of content overlap and the number of discrepancies in handoffs given by different clinicians caring for the same patient during the same transition of care. Previous research in a variety of domains, including healthcare, suggested that SMMs are an important foundation of teamwork and that better developed SMMs are associated with higher functioning teams[4,47]. Moreover, previous studies of clinical communication, and, specifically, handoff suggested that handoff communication serves as a foundation for the formation of SMMs among members of patient care teams and for providing coordinated care [21,22]. Previous studies of electronic handoff tools indicated that such tools are increasingly adopted by multiple clinicians on patient care teams in addition to their intended users [33]. In this study, we hypothesized that as a result of this wider adoption, electronic handoff tools can help team members develop richer SMMs, which will manifest in a higher degree of content overlap in their verbal handoffs.

The content overlap scores and the numbers of discrepancies captured in this study were comparable with those found in our previous studies that used the same method. For example, our previous study of content overlap as a measure of SMM in a different ICU of the same hospital found overlap scores ranging between 0.007 and 0.065 (comparing with 0.002 and 0.12 in the current study). Cumulatively, these studies may begin to establish empirical benchmarks for different ranges of content overlap scores. For example, both of these studies suggest that even for high-functioning teams, the amount of overlapping content in their handoffs is still relatively small. In our previous study, teams with overlap scores of 0.04 and higher were rated as medium to high on a more traditional scale of SMM within teams. This further confirms appropriateness of role and discipline-specific focus in clinicians’ handoffs and that it is not necessary for team members to have complete overlap in their handoff coverage to be exhibit characteristics of a high performing team. The current study further supports this observation: the majority of overlap scores clustered around the lower ranges of the overlap scale. However, the study also suggested that within this range there exists rich variability in regards to content overlap scores and corresponding SMMs.

In contrast to our expectations, content overlap scores within patient care teams did not change after the implementation of the Handoff Tool. In addition, there was a non-significant yet considerable (by over 50% in the post-implementation phase) increase in the number of discrepancies in the content of handoffs between team members post-implementation. Notably, in this study discrepancies were not simply different choices of words or phrases, but clinically meaningful differences in stated facts, impressions, or plans of care that were validated by an attending physician with pediatric ICU experience. The study also showed that the captured discrepancies were unevenly distributed across teams. While most teams experienced only 1 or 2 discrepancies per handoff, two of the teams in the post-implementation phase had 4 or more discrepancies each, which accounted for 18% of all discrepancies identified in the study.

These findings suggest that the implementation of the Handoff Tool did not have the anticipated positive impact on SMMs within PICU patient care teams. There exist several plausible explanations for these findings. First, since post-implementation observations happened relatively soon after the implementation, it is possible that this lack of expected positive results was due to temporary disruptions of work practices commonly associated with new Health Information Technology (HIT) solutions. Prior work examining the implementation of HIT in clinical settings indicated that such disruptions are not uncommon and are greatly affected not only by the design of the technologies, but also by the implementation process[48–51]. It is possible that if the post-study observations occurred several months later, clinicians would have had more opportunities to form new practices, which could lead to the eventual increase in content overlap scores and, by extension, gains in SMMs[52].

Alternatively, these results may suggest that novel HIT tools in themselves may not be sufficient for changing such complex constructs as shared mental models and need to be accompanied by structural and process-oriented changes, such as standardization of practice, and education. These observations are consistent with previous studies of handoff interventions that highlighted the benefits of bundling multiple components together rather than focusing on technological solutions by themselves[53,54].

Finally, it is possible that the Handoff Tool in its current design did not inspire the expected positive changes in SMMs within clinical teams. For example, it is possible that lack of explicit content structure in the Handoff Tool, which included several free-form text boxes with only suggestive headers, contributed to lack of increased alignment among team members post-implementation. To address this, future research could help to identify data elements common among handoffs of different clinicians on patient care teams and suggest these for inclusion in electronic handoff tools in a way similar to previous studies of written handoff documentation[55].

The analysis of changes in the types of discrepancies post-implementation suggests a possible link between the availability of information in the Handoff Tool, and the information clinicians included in the verbal handoffs. It is possible that the implementation of the Handoff Tool led to increased reliance on handoff notes as the primary source of information pertinent to handoff and to reduction in information seeking elsewhere in the electronic health record. For example, the two types of discrepancies that increased considerably (almost doubled) post-implementation—those related to the dosage of treatment or therapy and those related to reported symptoms—were not explicitly included in the Handoff Tool, even though they were available elsewhere in the EHR.

Overall this research suggests that there remain many questions in regards to the role of electronic handoff tools in facilitating SMMs and greater synchronicity and alignment among members of patient care teams. While the association between electronic handoff tools and improved patient outcomes and user satisfaction have been well studied[31,56,57], the impact these tools have on clinical teamwork is less clear[58].

Moreover, while the new electronic handoff tools do inspire increased uptake by clinicians in different roles, few, if any, of them are specifically designed to be used by interdisciplinary teams. Future tools for interdisciplinary handoff could more proactively focus on promoting and supporting interdisciplinary documentation. For example, they could help different team members document according to the priorities of their clinical disciplines and patient care roles, while at the same time maintaining a common information core across disciplines. Of course, problems associated with cross-disciplinary care coordination are not unique to handoff or handoff tools, but rather are systemic issues that cut across many spheres of healthcare[59]. Previous work suggests that there is consistency in information needs across different clinicians on patient care teams[33,55]. Having access to both interdisciplinary information as well as discipline-specific handoff documentation may help clinicians develop a greater shared understanding of their patients and the required care, and avoid discrepancies such as those identified in this study. This could lead to a synergistic approach to increasing teams’ SMMs as well as maintaining role-specific complementary perspectives.

This study has a number of limitations. First, it was conducted in the single unit of a large urban academic medical center; as a result, its findings may not generalize to other units within the same center or other centers. This was a non-randomized observation pre-/post- study with a convenience sample; as a result, there could be other factors outside of the implementation of the Handoff Tool that contributed to both the content overlap scores and the frequency of discrepancies. This study also featured patients who had very different clinical conditions, case severity, length of stay, and other clinical variations. Additionally, the handoffs in this study were captured at various times of patients’ hospital courses. Some handoffs were recorded early on, while others were captured after a plan of care was well established. All of these factors were not controlled for and could have had an impact on content overlap scores. Another limitation is that the method for assessing shared mental models only takes into consideration participants’ spoken discourse and does not account for shared understanding that was not expressly verbalized. Additionally, this method may be sensitive to team composition. Due to the similarity in training among residents and fellows, content overlap scores for handoff groups that only captured physicians’ handoffs (in cases when nurses were either not available or not willing to be recorded) may be different than content overlap scores for handoff groups that included nurses. In this study team composition did not have a significant impact on content overlap scores, but this remains a possibility. Because the method used here relies on manual comparison of statements in verbal handoffs across team members, it is time-consuming and labor intensive, and is prone to human error in recognition of both content overlaps and discrepancies. However, inter-rater reliability analysis suggested a high degree of reliability across analysts. Moreover, automating the analysis of content similarity using computational methods, such as that proposed by Abraham et al[43], opens the possibility of using this approach in future studies of handoff interventions on a large scale.

CONCLUSIONS

In this study, we examined the impact of the implementation of an electronic handoff tool on shared mental models within patient care teams in a Pediatric Intensive Care Unit. Shared mental models were assessed using the degree of content overlap and the number of discrepancies in verbal handoffs between different clinicians caring for the same patient and captured at the same shift-change time. The study showed that the degree of content overlap did not change post-implementation. However, the discrepancies nearly doubled in frequency; moreover, their nature considerably changed from pre-implementation to post-implementation. The study suggests that examining HIT and their use from the perspective of interdisciplinary patient care teams can bring a new depth to the discussion of HIT implications for patient safety.

Figure 4.

Content Overlap Score Distribution

Table 4.

Description of Discrepancies

Category of discrepancy	Subcategory of discrepancy	Rate of discrepancy (total: pre/post)	Example
Patient Presentation		(0.84: 0.71/0.93)
	Patient Identifiers	(0.02: 0.02/0)	Resident: [patient name] is 5 months old Nurse: He is, I believe, nine months
	Symptom	(0.38: 0.29/0.44)	Resident: He’s still tachypneic Nurse: He’s not tachypneic.
	History	(0: 0/0)	NA
	Procedure (administration)	(0.26: 0.43/0.14)	Resident: But we didn’t put any thrombin on it last night Nurse: I put thrombin on it and got a nice dry
	Procedure (dosage)	(0.20: 0/0.34)	Fellow: So I went up on the Lasix drip 0.08 Resident: So we wean on the Lasix drip slowly, 0.3, 0.2
Assessment		(0.08: 0/0.14)
	Treatment	(0: 0/0)	NA
	Clinical Impression	(0.04: 0/0.07)	Resident: The x-ray just got like whited out again Nurse: But his right lung is definitely better
	Prognosis	(0.04: 0/0.07)	Resident: She can actually probably go to the floor today Nurse: So I don’t think the floor today
	Outcome	(0: 0/0)	NA
	Transfer of responsibility	(0: 0/0)	NA
Other		(0.06: 0/0.10)

Highlights.

1. Assessed impact of handoff documentation tool on shared mental models

2. No significant changes in overlap scores or rate of discrepancies

3. Future work needed to develop tools for reducing discrepancies in team understanding

4. Technical solutions alone may not be sufficient to improve complex care processes, such as handoff.