NightShift simulation to train newly qualified doctors in non-technical skills: a feasibility study

ABSTRACT

There is growing evidence of greater rates of morbidity and mortality in hospitals during out-of-hours shifts, which appears to be exacerbated during the period in which newly qualified doctors commence work. In order to combat this issue, an online simulation of a night shift was developed and trialled in order to improve the non-technical skills of newly qualified doctors and, ultimately, improve clinical outcomes. A randomised feasibility trial of the electronic training simulation was performed with medical students (n=30) at the end of their training and in the initial weeks of working at a large teaching hospital. The study showed that participants in the intervention group completed their non-urgent tasks more rapidly than the control group: mean (SD) time to complete a non-urgent task of 85.1 (50.1) versus 157.6 (90.4) minutes, p=0.027. This difference persisted using linear regression analysis, which was undertaken using rota and task volume as independent cofactors (p=0.028). This study shows the potential for simulation technologies to improve non-technical skills.

Introduction

Hospitals in the UK are struggling to cope with the demands of ever-increasing numbers of admissions. This pressure is felt keenly during the 75% of the year that does not fall between 9 am and 5 pm Monday to Friday. During this ‘out-of-hours’ period staffing levels are generally much lower. This effect is further exacerbated during the 10 pm to 9 am ‘night shift’ when care is largely provided by a skeleton staff of junior doctors, support workers and nurses.

Over the past decade, reducing length of stay, and an increasingly complex range of diagnostic and treatment options mean more interventions and tasks must be coordinated in a shorter time, a situation that has shifted care further towards the evenings, nights and weekends. Over the same time period, individual junior doctors’ working hours have fallen by more than 35% to comply with the European Working Time Directive.¹ These factors have contributed to increased shift intensity and cross-speciality cover, and to problems of communication between and across shifts.²

Failure to perform well may delay patient care and impact on its quality, with well-documented effects on error rate and adverse outcomes.^3,4 There is evidence that this high demand role leads to emotional burnout and poor quality of life.⁵

Junior doctors’ awareness and use of best practice for night shift work appears low,⁶ with many reporting instances of not being able to take breaks during shifts. Objective assessment of the non-technical skills of junior doctors is increasingly possible from routinely collected data from electronic records, ordering and taskflow systems.^7,8 However, this is either not captured or not interrogated in most hospitals. We have previously described objective granular data on tasks undertaken by new junior doctors that support the need for training in task prioritisation⁹ and route planning,¹⁰ and highlighted the gap between senior and junior clinicians in understanding non-technical skills in out-of-hours care.⁹ Such results relate to recurrently voiced concerns over the preparedness of new junior doctors and may be associated with the significant increase in mortality seen over the month when new doctors start work.¹¹

In response to this ‘apprenticeship gap’, simulation has become an increasingly common and effective tool in medical education, particularly for invasive or high-risk procedures.^19,20 However, standard clinical simulation techniques are usually resource intensive with limited scope for repeated attempts, particularly if applied to non-technical skills.¹⁴ Virtual simulation is an appealing prospect in this regard, as it has the potential to prepare people and improve performance.¹⁵ The application of video game technology and methodology is becoming more influential in delivering successful medical education aimed at junior doctors¹⁶ and patients.¹⁷

In this paper, we describe the leveraging of existing large and detailed data on junior doctors’ actual activity to develop a serious video game simulating night shift working. We build on previous focus group and survey results,¹⁸ and testing with medical students to evaluate the impact of the simulator on the efficiency of junior doctors working night shifts. We aimed to trial this intervention using actual work activity as the outcome measure to highlight the potential benefits and challenges of this approach, and to inform future, multicentre studies.

In this paper, we explore the research question: does data-driven nightshift simulation have the potential to improve the training of junior doctors in non-technical skills. To this end the investigation had the following specific aims:

1. To test the impact of a ‘nightshift’ simulator on task completion times during out-of-hours working for junior doctors during the first 2 months after the August handover.

2. To identify the practical considerations for development and deployment of a ‘nightshift’ simulation focused on non-technical skills.

Method

‘NightShift’ simulation development

The simulation was designed to run on Android (Google, CA, USA) enabled tablet PCs. Wireframe computer-aided design (CAD) map data on sections of the Nottingham City Hospital were provided to medical contributors to confirm the location of wards and departments. These CAD files were then used as templates to produce an accurate but simplified floor plan. Photographs were taken of the hospital to ensure correct placement and representation of internal aspects of the hospital (eg slopes, floor colour, staircases etc) and to provide orientation to the user during the simulation. The decision to use a lower fidelity simulation was consistent with the medical simulation literature.^2,3

We used 1 calendar year of data (2011) from the NerveCentre on-call task management system (NerveCentre, Wokingham, UK) to inform our simulation. As described elsewhere,⁴ the NerveCentre system allocates tasks a colour code (red, amber or green) based on clinical urgency. The taskflow data were summarised overall, by urgency colour code, task and per ward by hour of shift (>100,000 out-of-hours task requests in total).

We used the actual activity data to calculate the mean (and standard deviation (SD)) time to completion for each specific task type recorded in the NerveCentre data (eg ‘clinical review for chest pain’, ‘interpret ECG’). Each specific task therefore had its own time distribution in the simulation. The team agreed a list of potential delaying or complicating factors (eg unable to find notes, first cannulation failed) with approximate frequencies. These factors could then be displayed to the user when tasks took longer than the minimum time to complete to add to the realism of the simulation. In order to reduce complexity, cardiac arrest calls were the only tasks not included in the simulation.

To reduce usage time to a reasonable duration, simulation time was compressed: walking time ran 1.5 times faster and performing clinical tasks ran 5 times faster. To convey a sense of increasing fatigue, the character in the simulation walked more slowly with time and the screen darkened; this was improved by a visit to the mess or to a vending machine. The simulation was structured to provide summary statistics on performance.

The simulation was initially tested by the research team and volunteers (non-medical researchers at the University of Nottingham) who did not have a working knowledge of the layout of the hospital. These sessions also informed the level of the paper gameplay instructions given to each subsequent user, as well as adding to the iterative design process.

A version of the simulation was released for further appraisal among final-year medical students in Nottingham. This appraisal comprised four sessions where 6–10 students undertook at least one shift in place of a usual 1-hour formal teaching session.

Trial of simulation

Setting

The assessment was undertaken over the course of the junior doctor changeover in August 2013. The setting was Nottingham City Hospital, a large (approximately 800 beds) teaching hospital.

Participants

Medical students were approached before their ‘shadowing’ fortnight: the phase immediately prior to starting work as a doctor for the first time involves following the existing, outgoing doctor in the role they are about to commence. The simulation was described to all eligible students via a presentation while they were assembled for their trust induction. The first 30 students who expressed an interest were randomly assigned to standard induction or induction plus serious game training. All participants were given an information sheet and signed a consent form relating to the use of their performance data from both the simulation and the NerveCentre task tracking system. Free use of the simulation was permitted, and the participants were encouraged to contact a named team member if there was a technical issue preventing their engagement: tablet computers were made freely available for a no-cost loan during the study.

Data collection and analysis

Actual out-of-hours activity for each participant was determined using data from NerveCentre and cardiac arrest logs (cardiac arrests are the only tasks that can be submitted outside the NerveCentre system). Data collection ran from the start of the first shift (8 am) on 1 August 2013 to the end of the last shift on 30 September (10 am on 1 October). The primary outcome for a future efficacy trial was taken to be time taken to complete a non-urgent task (coded ‘green’ by NerveCentre). Secondary outcomes were time taken to complete an urgent task (coded ‘red’ by NerveCentre), and crash calls per shift. Analyses were undertaken in SPSS (IBM, NY, USA); continuous outcome variables were compared using t tests and linear regression.

Feedback from study participants was gathered through a focus group feedback session after the completion of the trial. Non-medical researchers led the session as we believed this would encourage more open reporting of difficulties. We also circulated links to online feedback forms to gather comments made directly after shifts and from those who could not attend the focus group.

Ethical approval for the study was obtained through the Ethics Committee of the Department of Engineering of the University of Nottingham. See Fig 1 for CONSORT flow diagram.

Results

Overall, 27 of the 30 individuals who initially volunteered to participate completed the study, and full task level datasets were available for 22 of these, as 5 did not undertake night shifts during the study period. Seven (32%) of those with full datasets were male. 9 of the 12 participants randomised to the simulation used it, and 2 individuals used it repeatedly. Table 1 shows the distribution of participants across the medical and surgical rotas.

Table 1.

Distribution of participants by study arm and working pattern.

		Rota		Total
		medicine	surgery
Group	Standard	6	4	10
	Simulator	5	7	12
Total		11	11	22

Activity data

Participants in the intervention group completed their non-urgent tasks more rapidly than the control group though there was a broad range of completion times: mean time to complete a non-urgent task was 85.1 minutes (SD, 50.1; median, 84.5; IQR, 71.02) versus 157.6 (SD. 90.4; median, 128.3; IQR, 128.6) (p=0.027). This difference was also observed when linear regression was performed using rota and task volume as cofactors (p=0.028). There was no difference in the time to complete urgent tasks (Fig 2). These task completion times and distribution values are comparable to those seen in previous work¹⁰ and are slightly skewed from a normal distribution.

Fig 2.

Boxplot of time taken to complete individual tasks by study arm illustrating range, quartiles and median. Red boxes represent urgent tasks; green boxes represent less urgent tasks.

Fig 1.

Consort diagram of flow of participants in pilot trial of the NightShift simulation.

Standard cardiac arrest logs were complete for all study days. There was no difference in the mean number of cardiac arrest calls attended per shift (0.39 vs 0.45).

Feedback

Following the trial, the new junior doctors were clear that they had been overwhelmed by the experience of starting work, and with so many new clinical protocols and systems to learn they had not had time to devote to a simulation. In their focus group, they also raised issues that mirrored those described in feedback forms by the medical students who had undertaken testing.

Reported shortcomings

A key issue reported by users of the simulation was difficulty in navigation around the virtual hospital using a plan (top-down) view and a static set of pictures relating to what the user would see at that location (Fig 3). Users recognised that hospitals have a plethora of signs, but felt the level of visual information provided tended to be too simple (such as some signs above ward doors not being in place). Users also were frustrated by the ‘fatigue’ element (slower walking pace with gradual darkening of the screen).

Fig 3.

Example screenshot of alpha version of digital simulation.

Actual difficulties of working out-of-hours

The second area of feedback related to realistic aspects of the night shift. The simulation involves a great deal of walking around in comparison to the medical activity undertaken. Although this is realistic (and time was compressed in the simulation) it reduced engagement in the activity. Similarly, students reported it was impossible to attend to multiple urgent tasks that appeared in a short time window. Again, this is representative of actual shifts rather than a game where one can ‘win’, but highlights an important issue with the representations of real-world tasks within serious games or simulations.

Discussion

This study shows it is feasible to create an educational simulation of on-call shifts with the primary aim of teaching non-technical skills. In the study, those in the simulation group took less time to complete non-urgent tasks while on-call. These data, taken with previous studies, suggest an efficacy trial for an intervention of this nature would require junior doctor recruitment at a number of medical schools. Hospital-specific simulations are therefore not likely to be practical for such a study. However, if only small time savings per task in the first weeks of work were realised through this approach, the cumulative effect would be substantial.

Strengths and weaknesses of the study

This work has strengths in its use of a high-volume of task level data and precise mapping to inform the underlying mechanics of the simulation. However, given the practicalities of collecting and integrating such data, it is unclear if this approach should serve as a template for future studies.

The project also focuses on the teaching of non-technical skills for junior doctors, a clear unmet need as discussed previously. It is possible that relatively brief exposure to the simulation could teach the layout of the hospital, increase navigational efficiency and make the user more aware of the importance of clinical task prioritisation, but it seems unlikely there was a major training effect.

A major limitation of the study was the limited engagement of users, with initial enthusiasm not translating into extended use as described above. Our study suggests successful simulations will need significant resources to support preparation, deployment and debriefing. The suggestion from this experience is that any such interventions are targeted at medical students as part of their curriculum, or that new doctors are provided with protected time in the busy induction period immediately before starting work. It may be most efficient to develop a suite of software with elements that focus on specific skills. The initial candidate would appear to be training in task prioritisation skills as it is likely valuable experience could be provided in a limited time period without the need to resolve complexities of accurate individual hospital mapping, time compression of full shifts, and making walking around the hospital engaging for the user.

Conclusions

This study has highlighted that virtual simulations based on routine clinical data could be developed and deployed as part of medical school training in non-technical tasks. Potentially, this could partially mitigate the previously reported ‘August effect’ on out-of-hours care.¹⁰

While apparently effective, our study suggests improvements that could enhance engagement with the system and streamline the development of future systems.

The use of a ‘top-down’ highly detailed virtual environment was time consuming to develop and a cause of frustration for the participants. We suggest that future work in this area could focus on simulators that target specific skills (task prioritisation, hospital navigation or the management of personal needs) rather than attempting to emulate the entire nightshift experience. More constrained simulation/games such as these could more efficiently deliver training need and would likely require less tailoring to specific sites.

Similarly, it is unclear if the ‘fatigue’ feature added sufficiently to the training value of the simulator to justify the development effort required to develop it and the commonly reported participant frustration with it.

Future work

As suggested above, future work is needed in developing and testing more focused and engaging simulators in order to provide training as efficiently as possible and with minimum user frustration. Large scale deployments would be required to definitively study a full night shift simulation, with our data suggesting an efficacy trial would need to include in excess of 800 students to have 80% power to detect a 10 minute difference in task completion time. However the potential benefits of this type of training appear to justify the development and formal testing of more generalisable and focused tools.

Acknowledgements

This work was supported by the National Institute for Health Research and the East Midlands Health, Innovation, Education Cluster grant EM24. It was carried out in collaboration with Horizon Digital Economy Research, through the support of Research Councils UK grants (EP/G065802/1, EP/M0008776/1); the MATCH project, through the support of an Engineering and Physical Sciences Research Council grant EP/G063822/1; and The Health Foundation, through the Insight 2014 project Informatics to identify and inform best practice in out of hours seconday care (7382).