Table 2.
Teamwork measurement using direct observation
| Tool | Design | Content validity | Predictive validity | Concurrent validity | Test–retest reliability | Inter‐rater reliability (ICC) and agreement (κ, Rwg) |
|---|---|---|---|---|---|---|
| NOTECHS27, 28, 29, 30, 31, 32, 33 | Scale: observed behaviours | Translated from aviation by theatre experts and human factors experts27 |
Improved scores after team training (P = 0·005)27
Improved scores after team and systems training (P = 0·025)33 |
Expected and observed correlation with glitch rate (P = 0·045)28 | 0·09 < P < 0·64 across 5 sites (non‐significant variation)28 Non‐significant variation across different time intervals27 | Rwg = 0·9627 ICC = 0·73–0·8828 |
| OTAS10 34, 35, 36, 37, 38, 39, 40 | Checklist: tasks and scale: observed behaviours | Theatre and human factors experts involved in development | No | Adverse correlation between impact of distractions and completion of patient‐related tasks (P < 0·050)6 10 | n.r. | Cohen's κ > 0·4034 Pearson's coefficient = 0·7138 ICC = 0·42–0·9040. In German operating theatres: κ > 0·40 in 70% of scale items, ICC = 0·78–0·8939 |
| SO‐DIC‐OR41 | Checklist: observed behaviours | Representative sample of theatre team involved in development | No | No | n.r. | Cohen's κ = 0·74–0·95 including for ‘tired’ observers |
| Coding of field notes42 | Scale: impact of coded field notes | No | No | No | n.r. | No, each observer had a different role |
| Mayo‐HPTS43 44 | Checklist: tasks and scale: behaviours | Validated for crew resource management44 | Improved scores after team training (P = 0·01) | No | n.r. | Cohen's κ = 0·46–0·9743 |
| METEOR45 | Checklist: tasks | Scale items verified by agreement between theatre experts | No | No | n.r. | Observers ‘calibrated’ until Cohen's κ > 0·70 Observer agreement for cases n.r. |
| NOTSS40 46 | Scale: behaviours | Theatre experts involved in development | No | Good correlation with Cannon‐Bowers scale32 | n.r. | ICC = 0·12–0·8347 |
| Cannon‐Bowers46 48 | Literature review | Based on psychological theory | No | Good correlation with NOTSS | n.r. | Cronbach's α = 0·80 |
| HFRS‐M47,49 | Scale: behaviours | Took elements of LOSA checklist for aviation | Briefing workshops and simulation had no significant effect on scores | No | n.r. | Cronbach's α = 0·8947 |
| Study‐specific survey7 | Scale: observed behaviours | Based on behavioural markers | No | No | n.r. | Observers ‘calibrated’ Rwg = 0·85 after training. Observer agreement for cases n.r. |
| Study‐specific survey50 | Checklist: coded events | Based on previously validated tool for assessing mental fitness and concerns | No | No | n.r. | Cohen's κ = 0·77 |
ICC, intraclass coefficient; NOTECHS, Non‐Technical Skills; OTAS, Observational Teamwork Assessment for Surgery; SO‐DIC‐OR, Simultaneous Observation of Distractions and Communication in the Operating Room; Mayo‐HPTS, Mayo High Performance Teamwork Score; METEOR, Metric for Evaluating Task Execution in the Operating Room; NOTSS, Non‐technical Skills for Surgeons; HFRS‐M, Modified Human Factors Rating Scale; LOSA, Line Oriented Safety Audit.