Synergy of wearable technologies and proficiency-based progression for effecting improvement in procedural skill training

The move from time-based to competence-based training has been limited by practical (often resource) issues and by the variability of effect offered by different training methodologies. Two independent advances, one technical (wearable recording devices (WRDs)) and the other methodological (proficiency-based progression—PBP),¹ may act synergistically to enable consistently effective training in procedural skills. In this article, we describe our ongoing work in which both are integrated in ‘real-world’ training and the potential for these together to transform training in procedural skills.

A potential synergy

Although the proficiency of physicians undertaking procedural skills directly influences patient outcome,² valid assessment of doctors’ procedural skills is yet a reality. The WRD alone will not be sufficient (as it simply enables acquisition of more data) but these devices can be central to acquiring digital recordings without consuming the learner's attention. Gallagher and colleagues have described PBP for training in procedural skills. This approach consistently achieves greatly superior training effect—including clinical performance—compared with other methods of competency assessment approaches³ but requires the development of unambiguously defined and detailed procedure-specific metrics and errors, so-called ‘procedure characterisation’.¹ The success of PBP is dependent on the definition and recognition of specific observable behaviours. In practice, this requires direct (and resource-consuming) expert observation or video acquisition and analysis. The emergence low cost, high-quality WRDs may address this impediment to widespread introduction of PBP. This synergy may enable doctors to acquire a cumulative personal ‘visual data set’ suitable for routine highly focused, deliberate practice as well as providing a practical means of quality assured training overseen by a clinical supervisor.

Wearable recording devices

For the purposes of this article, we define WRDs as any electronic device carried on the person that records aspects of performance while not impeding or distracting from task completion. In recent years, head-mounted, high-resolution audio-visual recording devices such as Google glass and Go-pro have been studied in the setting of medical training.⁴ These WRDs appear to be non-distracting to the operator, and also have the potential to decrease any Hawthorne effect or observer bias.⁵ In our experience of developing carefully defined metrics and errors, most are amenable to detection from a first-person view, such as that acquired using a WRD (table 1).

Table 1.

A sample list of metrics (errors in this case with critical errors in italics), recently developed by the authors for labour epidural catheter insertion. These were elicited and defined using facilitated expert input, ‘stress tested’ for clarity and ambiguity using sample videos, and underwent prospective evaluation of construct validity (unpublished data)

No.	Metrics (errors)	Error
1	Patient not positioned at the edge of the bed
2	Patient not positioned in the middle third (lengthwise) of the bed
3	Bed not horizontal and parallel to the floor
4	A clear working environment not established (eg, one or more of the following actions not completed—dress/gown taped, CTG monitor belt moved away from field, intravenous lines and monitor cables away from the working field)
5	Adequate time not allowed for antiseptic solution to act (interval from application of antiseptic to insertion of needle: 3 min for betadine and 60 s for chlorhexidine)
6	Stylet of epidural needle not replaced in the sterile field, once withdrawn from needle
7	Loss of resistance syringe connected with less than half barrel (5 mL) of air
8	More than two needle passes made in the same direction
9	Alteration in direction of needle advancement not limited to a single plane in any new pass
10	Second attempt made in the same interspace without a change in angulation in one or both planes
11	More than 5 min expended in same attempt
12	Insertion attempt made in an unprepared and unsterilised interspace

Integrating WRDs in procedural skill training

In addition to defining procedure-specific metrics and errors, PBP requires establishment of performance benchmarks based on a mean of expert performance. The trainee practices and is instructed specifically to those benchmarks, at first in a simulated setting. Having achieved proficiency in a simulated setting, we propose that each trainee uses (1) the characterised reference procedure (in the form of a set of metrics/errors) and (2) a WRD and mobile device for download and review, into the clinical setting. These tools together enable them to recurrently review and update versions of their own performance of particular procedural skills. They are thus enabled to continue deliberate practice and self-assessment on a daily basis while the degree of ‘real-time’ clinical supervision is unchanged.

Our proposed approach is that each procedure the trainee subsequently performs in the clinical setting is recorded using a WRD and that he/she performs formal self-assessment of each. The trainee is generally motivated to self-improvement and so performs self-assessment diligently; he or she should also bring a detailed and developing knowledge of their own performance to each successive review. The clinical supervisor will review a sample of video recordings of performance and score them against the benchmarks, independently of the trainee. This review will supplement the supervisor's memory of live observation in determining feedback and ‘sign off’ for a particular skill. The selection of performances for review as well as the timing of the review(s) may be dictated by trainee or trainer or by the duration of a training module. The paradigm shift in procedural skill training which we describe here may address the deep misgivings that many trainers and trainees have regarding current practices for supervision and especially ‘sign off’, and do so in a an efficient way. However, this process is completely new to procedural skill training and we do not underestimate the cultural shift required to successfully integrate this suggested approach into ‘real-world’ training.

At our institution, the use of WRDs has been successfully piloted for this purpose. Since February 2015 we have used WRDs to record the stabilisation of preterm (<32 weeks gestational age) infants in the delivery room by trainees and consultant physicians in neonatology. Metrics specific to this period of complex care and procedures have been developed and validated for WRDs. In addition to using traditional education methods for newborn stabilisation and demonstrating competency using low-fidelity mannequin models in the simulation laboratory, trainee neonatologists also use WRDs to monitor maintenance of technical skills when performing these preterm stabilisations. We compared learning and clinical outcomes before and after the implementation of a new metric-based training incorporating WRD-assisted learning at our institution. There were 38 preintervention video recordings compared to 29 postintervention video recordings. Our (previously unpublished) findings indicate that trainee neonatologists find this mode of learning useful, that WRDs improve skill acquisition and that the use of WRDs correlates with improvements in patient safety, as supported by an improvement in many aspects of newborn stabilisation, including a reduction in the average time to the placement of an electrocardiogram leads (60 vs 26 s, p=0.013), and a better use of team members during the stabilisation process (47% vs 16%, p=0.016).

In the longer term, this technology also offers an ideal opportunity to evaluate the association between physician performance and clinical outcomes.

Challenges

One potential limitation of WRDs is that the first-person perspective of an operator may not capture all important or relevant information to allow the accurate assessment of specific procedural skills or the surrounding environment (eg, team communication). The recording and storage of patient and physician data raises certain medicolegal, ethical and logistical concerns, none of which is insurmountable.