An integrative framework for sensor-based measurement of teamwork in healthcare

Abstract

There is a strong link between teamwork and patient safety. Emerging evidence supports the efficacy of teamwork improvement interventions. However, the availability of reliable, valid, and practical measurement tools and strategies is commonly cited as a barrier to long-term sustainment and spread of these teamwork interventions. This article describes the potential value of sensor-based technology as a methodology to measure and evaluate teamwork in healthcare. The article summarizes the teamwork literature within healthcare, including team improvement interventions and measurement. Current applications of sensor-based measurement of teamwork are reviewed to assess the feasibility of employing this approach in healthcare. The article concludes with a discussion highlighting current application needs and gaps and relevant analytical techniques to overcome the challenges to implementation. Compelling studies exist documenting the feasibility of capturing a broad array of team input, process, and output variables with sensor-based methods. Implications of this research are summarized in a framework for development of multi-method team performance measurement systems. Sensor-based measurement within healthcare can unobtrusively capture information related to social networks, conversational patterns, physical activity, and an array of other meaningful information without having to directly observe or periodically survey clinicians. However, trust and privacy concerns present challenges that need to be overcome through engagement of end users in healthcare. Initial evidence exists to support the feasibility of sensor-based measurement to drive feedback and learning across individual, team, unit, and organizational levels. Future research is needed to refine methods, technologies, theory, and analytical strategies.

Breakdowns in teamwork and communication are an independent cause of, and a cross-cutting theme in, many of the system failures leading to patient harm.^1–3 Teamwork improvement strategies can be effective,^4,5 but the lack of ongoing measurement, evaluation, and feedback impedes sustainment of good team performance.⁶ Few valid and reliable teamwork measurement systems exist in healthcare, and labor costs associated with implementing these systems can be prohibitive.

Sensor-based technology offers a novel low-cost method for augmenting the current approaches to teamwork measurement. In turn, this creates an opportunity for medical informatics to contribute in new ways to teamwork improvement and patient safety. This article reviews the emerging literature and proposes a framework rooted in the science of teams for designing multi-method measurement systems for teamwork in healthcare.

TEAMWORK AND MEASUREMENT IN HEALTHCARE

The large, multidisciplinary science of teams has informed the development of measurement systems.^7–9 Figure 1 illustrates the input–mediator–output (IMO) framework underlying much of this research.⁹ Table 1 provides definitions from the science of teams and sensor-based measurement used here. Team ‘inputs’ are relatively stable features of the team, its members, the task and environment. Team ‘mediators’ are dynamic team member interactions (ie, processes) or transient products of interactions (ie, emergent states) that translate team inputs into team ‘outputs’ such as effectiveness, viability, and learning.

Figure 1:

Overview of input–mediator–output (IMO) framework and examples of dimensions of teamwork and factors influencing team effectiveness.

Table 1:

Key definitions from the science of teams and sensor-based measurement

Teamwork definitions	A team consists of two or more individuals with specific roles working together interdependently and adaptively towards a shared goal.51 Teams can be partially or wholly distributed in space (ie, collocated vs virtual teams)52 and time (ie, using synchronous vs asynchronous communication technologies).53
	Taskwork is defined as clinical activities that do not demand interdependence (ie, tasks each team member complete without input from other team members).51 Understanding the taskwork of individuals in teams is important because of workload balancing (ie, team members must complete their individual tasks as well as their team tasks).
	Teamwork is defined as dynamic interactions among team members such as coordination and communication events.54
	Team performance is defined as the summation of taskwork and teamwork activities.51
	Team effectiveness is an assessment of the quality of team performance outcomes in relation to specified standards (ie, task outcomes, team member satisfaction and viability, learning outcomes).54
	Multi-team systems (MTSs) are defined as a network of component teams that share at least one mutual goal that is interdependent with another team, although each component team may also pursue different objectives at times.55
Sensor-based measurement definitions	The terms sensors and sensor-based technology for human and team performance describe automated data collection tools including radio frequency identification (RFID) tags, infrared sensors, video and audio recording devices, and accelerometers56 implemented for the purpose of capturing real-time sociometric data (eg, behavior, speech analysis, proximity to other team members, devices, and workplace location).
	Sensor-based measurement refers to the use of sensors to capture team performance data. Unlike traditional approaches to team performance measurement in healthcare, sensor-based measurement is automated and objective, and activity data are collected in real time.
	Activity traces are defined as information collected about team member interaction as a byproduct of completing tasks or using information systems. This includes an increasing array of data streams such as paging and phone systems, emails, and use patterns of and entries into electronic medical records. Such activity traces complement sensor-based technology, but do not dynamically capture sociometric data in a physical environment (ie, ‘sense’ behaviors, relative proximity, etc).

There are two general methods for measuring teamwork: self-report and observation.¹⁰ Self-report methods involve asking team members to rate: themselves as individuals; their team; or the entire facility. These methods capture inherently subjective attitude competencies (eg, mutual trust, collective efficacy and orientation, psychological safety) as well as perceptions of teamwork,¹¹ and have notable limitations, including systematic bias in self-ratings and challenges achieving adequate response rates.^12–14 Observational measures incur labor costs of time spent observing, training, and monitoring raters over time.¹⁵ This can be a large investment in effort, up to 20 h for some systems,¹⁶ and largely limit observations to funded research. Although self-reporting and observation can serve critical roles in the training and periodic peer review processes, the frequency of feedback to clinicians likely remains low. Sensor-based measurement of teamwork (ie, use of sensors such as RFID (radio frequency identification) tags, audio recording, and video recording to capture team performance data) and activity traces (ie, information collected about team member interaction as a byproduct of task completion through email, paging, and electronic health record systems, for example) offer approaches to augment current capabilities.

Past reviews of team performance measurement in healthcare indicated a lack of tools with strong theoretical grounding and methodologically rigorous development and validation processes.^17,18 More recent reviews indicate improvements in the quality of tools,^19,20 with some notable exemplars.^16,21–23 Continued progress in this area is needed, but the burdens of administration will probably constrain routine use. Less costly methods are needed to complement traditional approaches.

A FRAMEWORK FOR SENSOR-BASED MEASUREMENT OF TEAM PERFORMANCE

The traditional (self-report and observation) and novel (sensor-based measurement and activity traces) methods of team performance measurement each have unique strengths and weaknesses. Figure 2 presents the IMO framework and the emerging literature on sensor-based measurement of teams to illustrate factors to consider when building multi-method measurement systems. For team mediators—the primary focus of team measurement systems—the framework delineates how physical distribution of team members and the specificity of behavioral expectations for effective teamwork influence the utility of different measurement and analysis approaches. For team inputs and outputs, we describe applications of both direct and inference-based measurement. Table 2 summarizes existing validity evidence for sensor-based measurement of components of the IMO framework.

Figure 2:

Framework mapping applications of sensor-based measurement to the input–mediator–output (IMO) framework of team performance.

Table 2:

Summary of existing applications of sensors to team performance measurement

Team inputs, mediators, and outputs	Feasibility evidence
Input: team composition—personality	Information about a person's interactions, locations, activities, mood and language use coded from a relatively small sample of audio recordings (2 min/ h over a 2-day period) significantly predicted Big Five personality traits).28 Big Five Personality Traits include: Extraversion, Emotional Stability, Agreeableness, Conscientiousness, and Openness to Experience. In a sample of 67 post-anesthesia care unit (PACU) nurses, sensor-based measurement of physical activity, speech activity, face-to-face interaction and physical proximity predicted four of the Big Five personality traits with significant correlations ranging from 0.41 to −0.43 for different combinations of personality traits and interaction patterns.29
Input: task structure and interdependence	Sensor-based measurement was used to uncover work patterns within a pediatric unit to uncover potential trajectories for the spread of infection.38 Much of the nurses work in this unit was not interdependent (ie, focused only on their patients with little potential for cross-contamination).
Mediator: taskwork	Hendrich and colleagues57 used sensor-based location detection, activity monitoring, and physiological data collection to characterize workload and work processes in a sample of 767 nurses across 36 hospitals. Sensor-based indices were compared with traditional methods to establish initial validity of sensor-based measures.
Mediator: teamwork	Parlak and colleagues56 demonstrated the feasibility of environmental sensors for tracking performance processes of trauma resuscitation teams. Vankipuram and colleagues37 achieved a high level of reliability in classifying trauma team activities in simulated environments using motion and location sensors and a Hidden Markov Modeling analysis. They identified 15 key tasks and achieved 87.5% accuracy in classifying activity across all tasks. Kannampallil and colleagues43 implemented both location detection sensors and human observers within a trauma center and found a significant correlation between data sources (r = 0.96, p<0.001). Isella and colleagues38 describe how proximity detection sensors can be used to describe interaction patterns between different role types on a pediatric ward. They were able to identify specific interactions to target for infection-prevention strategies based on an analysis of the frequency and duration of contacts between care providers of different types and patients.
Mediator: emergent states	Kranzfelder and colleagues58 described critical attributes of team situational awareness during intraoperative care and pilot work with sensors for monitoring and feedback during surgeries.
Output: effectiveness	Length of stay and the number of delays could be predicted by composites of physical activity intensity, face-to-face interaction time, and proximity to a phone in a sample of PACU nurses.29
Output: staff satisfaction	In a study from outside the healthcare domain, the total amount of communication captured through both email and a combination of proximity and voice sensors as well as the betweenness of individuals within the organization's communication predicted ∼30% of the variance in individual satisfaction with interaction.36

Team inputs

Sensor-based measurement can be applied to at least two categories of team inputs: team composition and task interdependence structures.

Team composition (ie, the mix of attributes of individual team members) greatly influences team processes and outcomes.^24,25,26 Measuring team composition can be challenging in complex and dynamic work environments where team boundaries are permeable, individuals may be members of multiple teams, and team membership frequently changes.²⁷ When combined with activity traces, sensors can help answer basic questions, such as who is currently on the team, and infer attributes of individual team members based on their interaction patterns. The existing literature on individual attributes pertains to personality traits,^28,29 but could be broadened to include teamwork-related attitudes^30,31 and individual teamwork competencies.

Task and interdependence structures (ie, the configuration of how task inputs and outputs are distributed across team members, and the types of interactions required to complete tasks) influence team outcomes.³² In healthcare, many work practices appear to have evolved over time, rather than being engineered.³³ Sensor-based measurement of teamwork can descriptively map these organic interdependence structures within a unit or facility, explore variations in work practices, and ultimately develop alternative methods of organizing teams. For example, using a more traditional survey-based social network analysis approach, Effken and colleagues³⁴ showed that communication patterns within units correlated with safety and quality metrics. Higher levels of adverse drug events were associated with higher levels of betweenness centrality (ie, more information gatekeepers). Sensor-based measurement can capture these types of structural attributes of teamwork in a low-cost way.

Team mediators

Strengths and weaknesses of different measurement approaches can be delineated on two dimensions: (1) the physical distribution of team members, and (2) the specificity of behavioral expectations for effective teamwork. First, physical distribution of team members varies dramatically in healthcare organizations (eg, a surgical team with primarily colocated members; a cancer treatment multi-team system with primarily distributed team members) as well as within teams (eg, an intensive care unit with phases of collocated—rounds—and distributed activity). Second, as with the technical work,³⁵ expectations for teamwork behaviors are highly specified in certain situations (eg, protocolized cardiac or trauma resuscitations) and much less specified in others. Self-report methods are equally applicable across these team configurations and situations, but physical distribution and specificity of expectations have unique implications for the relevance of observation, sensor-based, and activity trace methods as well as for the appropriateness of analysis techniques applied to the data collected.

Physical distribution primarily affects the relevance of sensor-based measurement and activity trace data. Sensor-based measurement of teamwork primarily detects face-to-face interaction patterns including proximity and conversational dynamics. As the physical distribution of teams and multi-team systems decreases, the relevance of sensor-based measurement is likely to increase. As physical distribution increases, the relevance of activity traces is likely to increase because these data are captured primarily through interaction in distributed communication systems (eg, emails, pages, texts, entries into electronic medical records). Understanding when and how the physical distribution of a team may vary over time is necessary for planning an appropriate measurement system. However, aside from entirely colocated or entirely distributed teams, most situations will call for a blended approach using both sensor-based measurement and activity traces to capture a team's interaction patterns. Olguín and colleagues³⁶ found that staff satisfaction was significantly associated with the total amount of communication measured through both sensors and activity traces, but not with each measure of communication in isolation.

The degree of behavioral specificity of expectations for effective teamwork has implications for observational methods and analysis techniques applied to sensor-based measurement. For observational methods, structured observation is applicable with high degrees of behavioral specificity, and unstructured or ethnographic observation with low specificity of performance expectations. Similarly, methods of analysis for sensor-based measures emphasizing the detection of a priori defined patterns of interaction will be most relevant for areas of a team's work with defined behavioral expectations. For example, Vankipuram et al.³⁷ identified very specific interaction patterns in sensor-based measures of teamwork in trauma resuscitation teams with highly protocolized interactions. Conversely, more exploratory or descriptive analysis methods are appropriate in areas of low behavioral specificity. In a pediatric unit with relatively low specificity of teamwork expectations, Isella and colleagues³⁸ applied an exploratory analytical approach to discover patterns of interaction that could be targeted to prevent infections.

Team outputs

Sensor-based measures of teamwork can be applied to at least three categories of team outputs: task efficiency, team learning, and affective outcomes. Task efficiency is the most straightforward where sensor-based measurements of teamwork capture reaction times to alerts and alarms. Assessing team learning through sensor-based measurements can include evaluating changes in a priori defined patterns of effective and ineffective teamwork, or changes in more descriptive measures of communication structures. Affective team outcomes, such as staff satisfaction, can be inferred through analyzing patterns of team interaction.²⁹

Summary

Sensor-based measurement is one of several strategies for evaluating teamwork. It is most applicable for teams, or phases of team performance, where members are at least partially colocated. Different analytical approaches should be applied to sensor-based measurement data collected in aspects of work with high (ie, prospective pattern-detection methods such as tensor decomposition described below) versus low (ie, exploratory techniques) specificity of behavioral team performance expectations. This review has focused primarily on sensor-based measurement and not activity traces, reflecting a disproportionately low number of studies on activity traces used for team performance measurement and indicating a strong need for work in this area.

APPLICATIONS: MULTILEVEL PERFORMANCE EVALUATION AND LEARNING

Sensor-based measurement can contribute to interventions for improving individual, team, unit, and organizational learning in healthcare. For individuals, sensor-based measurement can provide real-time support for clinicians to balance their individual workload efforts and provide feedback on the quantity and quality of interactions with other clinicians or patients. For teams, sensor-based measurement can serve to augment traditional methods of team improvement, such as self-guided team reflective activities,^39,40 by providing visualization of performance patterns to aid diagnosis of performance deficits. Unit-level learning involves the detection of work system issues that can be addressed through policies, procedures, equipment, work processes, staffing, and so forth.⁴¹ Sensor-based measurement can automate process mapping in an ongoing way to identify bottlenecks in flow or other inefficiencies. Mechanisms for sharing innovation and knowledge within an organization are a marker of high reliability and safety.⁴² Widespread adoption of sensor-based measurement can provide an analog to aviation's flight data recorder, allowing playback of real events in simulated environments and sharing knowledge generated from that experience.⁴³

ANALYTICAL STRATEGIES

Sensor-based measurement captures a dynamic network of who interacts with whom, and when, where, and how, together with myriad covariates and dependent variables. Dynamic network analysis involves visualization, exploratory data analysis, and statistical inference. Methods exist for simple Euclidean representation of time series of graphs, and inference thereon.^44–47 Recent theoretical results have proven this representation of the dynamic network to be a principled representation for visualization, exploratory data analysis, and quantitative statistical inference regarding team-based causes and effects for the evaluation of teamwork and the development of improvement strategies.

DISCUSSION

Teamwork is critical for safe patient care. Interventions can improve teamwork, but their impact is limited by the absence of reliable, valid, and practical measurement approaches. Sensor-based measurement can augment existing approaches and improve access to developmental and real-time feedback on team interactions. There is much to learn, both in the science and practice of this new domain. The initial work is exciting and encouraging, but new methods, technologies, theories, and analytical approaches must be developed and refined to make the most of this approach.

Managing the cultural complexities of implementing sensor systems may be one of the biggest challenges. Building trust in the system will be critical, or staff will devise workarounds to corrupt the quality and utility of the data.^48,49 Privacy and security concerns associated with accessing these data must be addressed. The aviation industry faced a similar crossroads in the 1960s, when the introduction of cockpit voice recorders depended on ‘the bold support of the airline pilots and the wisdom of the aviation community’⁵⁰ (p6). However, to reach the ultimate outcomes of improved safety and quality, team performance measurement must be put to use.

CONCLUSION

Large practical benefits to care providers, patients, and their loved ones can be realized by addressing the technical, theoretical, cultural, and methodological issues involved in sensor-based measurement. Consequently, this represents a problem where practical application can drive fundamental advancements in our conceptual understanding of human dynamics and technological capability.