Effects of the Simulation Using Team Deliberate Practice (Sim-TDP) model on the performance of undergraduate nursing students

Abstract

Background

The use of simulation has grown in prominence, but variation in the quality of provision has been reported, leading to calls for further research into the most effective instructional designs. Simulation Using Team Deliberate Practice (Sim-TDP) was developed in response. It combines the principles of simulation with deliberate practice, therefore, providing participants with opportunities to work towards well-defined goals, rehearse skills and reflect on performance whilst receiving expert feedback. This study aimed to compare the effects of Sim-TDP, versus the use of traditional simulation, on the performance of second year adult nursing students.

Methods

Using a longitudinal quasi-experimental design, the effects of the two approaches were compared over a 1-year period. Sixteen groups, each containing an average of six participants, were randomised into an intervention arm (n=8) or comparison arm (n=8). Data collection took place at 3 monthly intervals, at which point the performance and time to complete the scenario objectives/tasks, as a team, were recorded and analysed using a validated performance tool.

Results

The independent t-tests, comparing the performance of the groups, did not demonstrate any notable differences during the three phases. However, in phase 1, the independent t-tests suggested an improvement in the Sim-TDP participants’ time spent on task (t ₍₁₄₎ = 5.12, p<0.001), with a mean difference of 7.22 min. The mixed analysis of covariance inferred that the use of the Sim-TDP led to an improvement, over time, in the participants’ performance (F_{(1, 5)} = 12.91, p=0.016), and thus, an association between Sim-TDP and the enhanced performance of participants.

Conclusion

The results suggest that Sim-TDP, potentially, optimised participant performance, while maximising the use of Simulation-based education (SBE) resources, such as simulation facilities and equipment. The model could be of practical benefit to nurse educators wishing to integrate SBE into their programmes.

What this paper adds.

What is already known on this subject

• Deliberate practice, as an educational approach in simulation-based education, has been demonstrated to improve the performance of individual learners.

• It has also been demonstrated that the use of simulation-based education in team training leads to positive outcomes in the performance of healthcare teams.

• At a team level, the use of team deliberate practice has been used successfully to improve team performance in various sports, however, there is little evidence of its use within healthcare simulated-based education.

What this study adds

• Our study suggests that the use of team deliberate practice in simulation-based education could potentially improve the performance of adult nursing preregistration students.

• The study further suggests that the model ‘Simulation using Team Deliberate Practice’ was a viable approach to use within adult nursing preregistration education.

Introduction

The use of simulation-based education (SBE) in healthcare education has grown rapidly over the past decade,¹ buoyed by a growing evidence base and a greater understanding of the underpinning pedagogy.² However, a scoping exercise undertaken in the UK found a wide variation in the quality of delivery.³ Anderson et al ³ reported that integrating SBE across healthcare curricula, and underpinning it with robust pedagogical research, would support a more consistent high quality approach. Deliberate practice (DP), developed by Ericsson,⁴ is one such approach as some authors^{2 5} posit that it boosts learner performance. Clapper and Kardong-Edgren⁶ reason that implementing DP would benefit nursing programmes through enhancing student performance.

DP is described as an approach that engages individual learners in repetitive learning activities encompassing well-defined learning objectives, set at an appropriate level, and supported by an expert facilitator providing immediate feedback.^{2 4} Ericsson⁴ identifies that SBE, incorporating DP, provides individual learners with opportunities to improve performance. However, it can be constrained by timetabling restrictions and the availability of resources such as specialised staff, SBE rooms and equipment.⁷ When combined with large student cohorts, nurse educators wishing to use SBE face significant challenges.⁸ Harris et al ⁹ proposed that TDP offers a potential solution to these challenges. In concordance with the DP approach, TDP combines well-defined learning objectives, set at an appropriate level, with opportunities for repetitive team practice under the supervision of a coach providing immediate feedback.^{10 11} As an approach, Helsen et al ¹⁰ report that international football players combine both individual DP and TDP to improve their performance and master their team sport. However, in the healthcare context, although nursing studies^{12 13} have indicated that SBE improves team performance, there remains little guidance as to how TDP should be integrated into an SBE curricula.

Katzenbach and Smith¹⁴ describe a team as a small group, committed to a common goal, whose success is dependent on them interacting efficiently. As effective team working is viewed as essential to high quality, safe healthcare,^{15 16} team training interventions are vital to achieve high standards of performance.¹⁷ Salas et al ¹⁷ describe team training as a set of tools and methods used to enhance teamwork and refer to three components: teamwork, task work and a combination of both. Teamwork, in this context, refers to the behaviours that facilitate effective team interaction¹⁵ including decision making, assertiveness, situational awareness and communication skills.¹⁸ Task work denotes what team members are doing with regard to team goals,^{16 19} including core technical competencies,¹⁷ such as performing and recording vital observations and undertaking patient assessments. Consequently, the tasks performed by one member of the team contribute to the overall performance of the team and, as a result, reflect the effectiveness of team coordination. This mirrors the coordination component of teamwork,²⁰ and links to phase one of the model of team development proposed by Morgan et al,²¹ especially the teamwork pathway.²¹ They purport that teams progress through several phases of development and identify two discrete developmental pathways (figure 1). The first relates to task work, the second to teamwork. For team training to be successful the two pathways need to develop separately and then combine as learners progress.²² Given that teams develop along both pathways, Mathieu and Rapp²³ argue that teams need to establish a solid foundation for each pathway during the early stages of their development. However, exactly how this should be integrated into educational programmes has not been clarified.²² Nelson et al ²⁴ echo this lack of clarity, having found little evidence to indicate the best way to implement team training into an undergraduate programme. Thus, to implement the team training pathways and potentially address resource constraints, the ‘Simulation using Team Deliberate Practice’ (Sim-TDP) model was developed.

Figure 1.

Team evolution and maturation. The merging of task-related skills and team-related skills during team. From Morgan et al,²¹ Copyright 2001 by Taylor & Francis. Reprinted by permission of the publisher (Taylor & Francis, http://www.tandfonline.com).

The aim of this study was to compare the effects of Sim-TDP on the performance of second year adult nursing students, compared with that achieved using traditional SBE. The latter approach followed the classic three stage model; the prebrief, clinical scenario and debrief.²⁵ It was hypothesised that the mean performance scores of the Sim-TDP intervention groups would be different from the scores of the comparison groups using traditional SBE.

Ethics

Participants were given study information and informed that participation was completely voluntary. They could refuse to participate and opt out of the study at any time. As the SBE sessions were part of their nursing programme they still had to participate in the activity but, if they chose to opt out the data relating to them would not be used. All data were stored securely.

Methods

A longitudinal quasi-experimental design was adopted to compare the effects of Sim-TDP on the performance of second year adult nursing students, to that achieved using traditional SBE. Both interventions were integrated into the practical modules of an existing curriculum.

Data collection took place at three time points over the year. At these points, the performance and time to complete the scenario objectives/tasks, as a team, were recorded and analysed using a validated performance tool. The data were analysed using the software package Statistical Package for the Social Sciences (SPSS) (IBM SPSS Statistics V.22). An independent t-test was used to compare both the mean performance scores and the time on task of the two arms during each phase. Effect sizes were calculated using Cohen’s d statistical test. As Sim-TDP is time dependent, a mixed analysis of covariance (ANCOVA) was undertaken and Pearson’s correlation coefficient r used to estimate the effect size.

Sample

The sample was taken from a cohort of adult nursing students who had commenced year two of a 3-year adult nursing programme. Due to the structure of the curricula and timetabling demands, these students had been placed into tutor groups that composed of an average of 24 students. This tutor group structure dictated when the students were taught. Due to the data collection time frame and timetabling constraints, a convenience sample of four of these tutor groups (n=4) was used. These were randomised, following the process outlined in figure 2, into 16 subgroups (n=8 in the intervention arm and n=8 in the comparison arm), each containing an average of six participants. Once randomised, the comparison and intervention arms undertook their SBE experiences separately.

Figure 2.

The randomisation process.

The traditional SBE approach

The traditional SBE method (figure 3) undertaken by the subgroups within the comparison arm, followed a standardised approach that was based on the International Nursing Association for Clinical Simulation and Learning’s Standards of Best Practice: Simulation^SM.²⁵ This encompassed three stages; the prebrief, clinical scenario and debrief. The prebrief focused on the effective preparation of the participants, outlining the aims and objectives of the scenarios, as well as participant roles, professional expectations, orientation to the environment and simulator. The scenarios were designed to represent a clinical situation that the participants may encounter during their clinical practice and followed a standard scripted scenario template incorporating salient signs and symptoms. These were piloted prior to the commencement of the study. The scenario ended when the participants, as a team, completed their assessment and rang for senior help. The subgroup’s performance was video recorded during the scenario but no video feedback was used during the debrief. The debriefing stage was facilitated by an experienced faculty member using a standard proforma based on the three phase structured debriefing model developed by Steinwachs.²⁶ The three phases were: description, analysis and application phases, which were underpinned by the debriefing with good judgement approach.²⁷ This meant the teams in the comparison arm undertook one scenario and one debrief.

Figure 3.

Traditional simulation model.

The intervention

The Sim-TDP model (figure 4) using the same templates and models followed the first three phases of the traditional SBE approach. However, following the debriefing stage the Sim-TDP subgroups were provided with further opportunities to rehearse the same scenario. Each team, under the guidance of an expert facilitator using the ‘within-event’ debriefing approach,²⁸ first undertook a ‘coached walk through’ of the scenario in the SBE environment. Once completed, the team then repeated the same scenario, which was video recorded. The teams then undertook a final debrief following Steinwacks²⁶ model. This meant the intervention subgroups repeated or rehearsed the scenario three times and undertook two debriefings.

Figure 4.

Simulation with team deliberate practice model.

The SBE programme

In total six scenarios were used, with each phase incorporating two scenarios (figures 3 and 4). These were undertaken in the University’s simulation centre over a 3-hour time period, with 1 hour and 25 min for each rotation of SBE or Sim-TDP. Due to the numbers of participants per group, the simulations followed the process outlined in figure 5, enabling both the Sim-TDP and traditional SBE groups to undertake one scenario and observe a second. No other educational interventions were used.

Figure 5.

Scenario delivery.

The SBE environment was set up to represent a surgical or medical ward and contained relevant clinical equipment, for example, oxygen masks. The patient simulators used were Laerdal’s SimMan (Laerdal Medical, Stavanger, Norway). Their functionality meant that participants could record relevant vital observations, for example, respiratory rate and blood pressure. They could also be ‘voiced’ so that participants could communicate with the patient and to increase realism further relevant moulage was used, for example, to replicate cyanosis. Another experienced faculty team member facilitated the scenario as the student’s mentor following a standardised script. To ensure consistency, all facilitators were fully trained in using both traditional SBE and Sim-TDP.

Each scenario focused on the recognition of a deteriorating patient and, as recommended by the Resuscitation Council (UK),²⁹ the use of the ‘ABCDE’ (Airway, Breathing, Circulation, Disability and Exposure) systematic assessment framework and the ‘SBAR’ mnemonic (Situation, Background, Assessment and Recommendation). These formed the basis of the scenario learning objectives, which were to recognise a deteriorating patient, use the ABCDE assessment and SBAR handover tool. These were set at the participants’ current level of development. This process was repeated at 3 monthly intervals.

Performance tool development

The performance tool (figure 6) was structured using the ‘ABCDE’ assessment framework²⁹ and ‘SBAR’ handover²⁹ mnemonics. Content validity was established over several phases. Initially, content was identified through a literature review and this was used to develop a checklist of representative tasks.⁴ The checklist was reviewed by an expert panel (n=12) composed of university academics and hospital-based practitioners who had expertise in both SBE and critical care. The Content Validity Index³⁰ was used to assess the relevance of each item and a scale-CVI rating of 0.98 was found, which was above the 0.90 recommended by Polit and Beck.³⁰

Figure 6.

TDP performance observation tool—hypovolaemia. BP, blod pressure; EWS, early warning score; SBAR, Situation, Background, Assessment and Recommendation; TDP, team deliberate practice. Query - table line missing on right hand side.

To ensure the reliability of the data, all the videos captured (n=59) were reviewed by two independent raters and the researcher (n=3). The data collected was analysed for inter-rater reliability using the intraclass correlation coefficient. A Cronbach’s α of 0.71 (95% CI 0.55 to 0.84) was found, which was above the 0.70 threshold demonstrating the reliability of the tool.³¹

Results

Data were assessed for any potential violation of assumptions. Demographic data, including the participants’ gender and age, were extracted (table 1).

Table 1.

Demographic data

			Intervention	Comparison	T-test	P value	??2 this should be: χ2	P value
Participants	Total	98	52 (53%)	46 (57%)	1.04	0.303	3.93	0.686
	Withdrew	1	1	0
	Left programme	4	2	2
	Grand total	93	49	44
Gender	Male	3 (3%)	2	1	−0.492	0.624	0.246	0.620
	Female	95 (97%)	50	45
Age	18–24	73 (76%)	34	39	2.090	0.039	4.26	0.039
	25–30	24 (24%)	17	7
	31–36	6 (6%)	5	1
	37 +	8 (8%)	4	4

Homogeneity of participant numbers and gender was evident across the intervention and comparison arms, however, the analysis suggested heterogeneity across the age groups.

Performance analysis

Descriptive statistics were performed at the subgroup level (n=16) for both the intervention arm and the comparison arms (table 2).

Table 2.

Postperformance group statistics

	Condition	Group statistics
		N	Mean	SE	P value	Effect size (Cohen’s d)
Phase 1 performance post	Comparison	8	37.13	1.81	p=0.305	d=0.53
	Intervention	8	39.50	1.31
Phase 2 performance post	Comparison	7	35.57	2.22	p=0.131	d=1.24
	Intervention	3	42.00	2.52
Phase 3 performance post	Comparison	6	31.83	2.10	p=0.779	d=0.17
	Intervention	5	32.80	2.65

In phase 1, the performance scores between the Sim-TDP intervention group (M=39.50, SE=1.31) and the traditional SBE comparison group (M=37.13, SE=1.81), had a mean difference of −2.38, 95% CI (−7.16 to 2.41), with a t₍₁₄₎ = −1.06 and p=0.305. A moderate effect size (d=0.53) was noted. In phase 2, the mean difference between the performance scores for the Sim-TDP intervention subgroups (M=42.00, SE=2.52) and the comparison subgroups (M=35.57, SE=2.22) was −6.43, 95% CI (−15.25 to 2.39), with a t ₍₈₎ = −1.68 and p=0.131. The Sim-TDP intervention had a large effect size (d=1.24). In phase 3, the mean difference in the performance scores between the Sim-TDP intervention subgroups (M=32.80, SE=2.65) and the comparison subgroups (M=31.83, SE=2.10) was −0.97, 95% CI (−8.51 o 6.59), with a t₍₉₎ = −0.29 and p=0.779. A very small effect size (d=0.17) was found. In terms of the performance of the teams during the individual phases, the analysis suggests there were no differences between the Sim-TDP intervention and the comparison groups, inferring that Sim-TDP during the individual phases did not influence performance.

In phase 1, the analysis found that the mean difference between the postperformance time on task, in minutes, for the intervention subgroup (M=8.52, SE=0.70) and the traditional SBE comparison subgroups (M=15.74, SE=0.70), was 7.22, 95% CI (4.19 to 10.24), with a t₍₁₄₎ = 5.12, and p<0.001, and a very large effect size (d=2.56). This suggests that Sim-TDP potentially reduced the time on task compared with the traditional SBE and facilitated the achievement of the scenario objectives earlier. However, the analysis in phases two and three did not demonstrate any differences, although the effects sizes for both were large. In phase 2, this was d=0.85, and in phase 3, it was d=1.34. This analysis suggests that Sim-TDP did have a potential positive effect on the teams by reducing their time on task.

As age was found to be a covariant, a mixed ANCOVA was undertaken on the participants’ performance across the three phases, and a difference between the groups in the two arms was found (F_{(1, 5)}=12.91, p=0.016). Pearson’s correlation coefficient r was used to estimate the effect size, and this was found to be large, r²=0.85, with an observed power of 0.82. This infers that Sim-TDP, independent of age, had a potential positive effect overtime on the performance scores of the teams.

Discussion

The findings highlight the potential of the Sim-TDP model as an effective instructional design for SBE. In terms of team performance, the results suggest that the model had a potential positive effect. As Ericsson⁴ identifies, providing opportunities to practice is a vital component of DP, and the Sim-TDP model was designed to maximise these opportunities. Although the independent t-tests on the participants’ performance did not demonstrate any difference during the phases, the mixed ANCOVA identified a possible practical benefit of the intervention. This was the achievement of continuous skill improvement,² and the attainment of progressively higher levels of performance over time.⁴ These findings are in line with the results of other DP studies in nurse education,³² highlighting the potential efficacy of using Sim-TDP in nursing curricula. They also echo the results reported by Ward et al,³³ who found that the accumulated hours spent in TDP consistently discriminated between elite and subelite football players. Furthermore, studies by Baker and Young³⁴ and Lund et al ¹¹ found that effective performance depended on the cohesive interaction among team members, gained through individual and coached team training.

The reduction in the participants’ time on task in phase one was promising. In terms of patient safety, this could have beneficial effects on patient care if teams are able to recognise patient deterioration earlier. It also appeared that Sim-TDP had its largest effect on the participants’ time on task when they were at an earlier stage in their professional development, which further supports its early integration into an undergraduate nursing curriculum. In terms of team training, the findings echoed the improvement in neonatal resuscitation performance of paediatric residents found by Sawyer et al.³⁵ In this study, participants working in teams of two undertook three simulation scenarios over a 2-month period. Although Sawyer et al ³⁵ did not identify their intervention as TDP, they reported a positive impact of using DP in SBE.

The results provisionally support the use of the Sim-TDP model early in the education of undergraduate adult nursing students to support the assimilation of their task-working and their team working skills.^{20 36} As emphasised by Kardong-Edgren et al,³⁷ finding the SBE methodology with the most impact with regard to learning and retention is vital. The improvement in participants’ performance, over time, infers that by using small teams of participants the Sim-TDP model potentially achieves a balance between optimising team performance and maximising available SBE resources. As the enhanced performance Sim-TDP achieved was within the same location, time frame and resources as the traditional SBE approach. This provisionally points to a more efficient model of SBE that potentially overcomes the resource challenges faced by SBE educators. These challenges include the availability of SBE rooms, equipment and appropriately trained staff.⁷ Consequently, Sim-TDP offers SBE educators a model that could be integrated into a wide range of professional undergraduate curricula effectively.

Study limitations and strengths

The study had several limitations. First, the quasi-experimental design means that the study findings are not generalisable, and only associative, not causal, inferences can be made. Second, the study sample size was relatively small (n=16) and a convenience sampling technique was used to select the initial tutor groups (n=4). This was compounded by the heterogeneity in the age groups. However, to reduce threats to internal validity, the naturally occurring groups were randomly assigned into their respective arms.

A third limitation related to the structure of the two models. The performance of those observing the initial scenarios could have affected their results. As the traditional SBE approach only included one scenario and debrief compared with three scenarios and two debriefs in the Sim-TDP approach, the process of repeating the scenarios rather than the model itself could have influenced the results. However, as O’Regan et al ³⁸ reported, observation conveys no advantage to participants.

Fourth, as the data collection tool was designed by the authors this created a potential source of bias. However, the design and development of this tool was very specific to the study population and followed a rigorous development process. Finally, several logistic and technological issues, such as timetabling cancellations and the loss of video captured materials, may have potentially affected the results.

Nevertheless, it is envisaged that the results will act as a catalyst for SBE educators to either incorporate Sim-TDP into their SBE programmes or to undertake additional research into its use. The study had several key strengths, the first of which was the approach we adopted in relation to the interpretation of p values. Acknowledging the debate on the use of p values and statistical significance,³⁹ we adopted an open and cautious approach to the interpretation of the findings. Second, the study was undertaken in an actual curriculum setting, using participants studying on a nursing programme, and not as an additional SBE activity. This increases the potential for translation into other curricula and adds credibility to the findings. Another strength includes the use of standardised scenarios and debriefing methods for both arms and, the use of experienced SBE facilitators trained in the use of both approaches. The use of three raters to evaluate and rate the performance of the subgroups added further strength to the study, since this led to a consensus score for each sub-group reducing the potential risk of bias.

Conclusion

Overall, the results suggested that the Sim-TDP model, as an instructional design, had a positive impact on the participants’ performance. The greater levels of performance over time and the reduced time on task achieved within the same time frame and resources highlights the potential efficacy and practical benefit of Sim-TDP. The results were promising and signalled the possible feasibility of developing the task-work and team working skills of student nurses. Consequently, Sim-TDP offers an approach that could potentially aid SBE educators in developing the professional competencies of student nurses.