Can narration and guidance in video-enhanced learning improve performance on E-BLUS exercises?

Abstract

Introduction

This study aimed to compare trainees’ laparoscopic performance concerning the peg-transfer (PT) and needle-guidance (NG) exercises after watching the original European Basic Laparoscopic Urologic Skills (E-BLUS) video or after watching a video-mentored tutorial (VMT) with ‘tips and tricks’, narration and didactic illustrations.

Material and methods

An experimental, unblinded, parallel, 2-intervention, 2-period randomized trial with an allocation ratio of 1:1 was conducted. Forty-two participants were randomized into 2 groups. Prior to task initiation, Group 1 watched the VMT in both trials and Group 2 watched, firstly, the original E-BLUS examination video and, in the second trial, the VMT. Each participant performed 2 trials for each exercise. Outcome measures were task time and total number of errors.

Results

In the first period, participants who visualized the PT and NG VMT had fewer errors than participants who visualized the E-BLUS video (p = 0.001 and p = 0.014, respectively). In the second period, after watching the VMT, a decrease in the total number of errors in PT and NG exercises was observed in the participants who previously watched the E-BLUS video (p = 0.001 and p = 0.002, respectively). In the second period, a decrease in median task time was observed for Group 1 and 2 after watching the PT VMT (p ≤0.001 and p = 0.003, respectively) and NG VMT (p = 0.005 and p = 0.01, respectively).

Conclusions

The use of VMT can lead to a smaller number of errors and, if coupled with deliberate practice, could lead to a shorter task time in exercise performance among participants with no previous laparoscopic experience.

INTRODUCTION

In the early twentieth century, the onset of laparoscopy revolutionized surgery and became a cornerstone of minimally invasive surgery (MIS) [1]. MIS is currently a frontline option in comparison to traditional open-surgery techniques [2] in most surgical specialties, namely Urology [3].

Moreover, the acquisition of surgical skills is currently difficult, due to shorter hospital stays, lower operative loads and demand of increased clinical productivity [4]. Technically speaking, laparoscopic procedures also require mastery of a specific set of skills when compared to the traditional open-surgery techniques. These include: accurate instrument handling, depth perception, hand-eye coordination and non-dominant hand dexterity [5, 6, 7] which are also not addressed by the current apprenticeship model.

Thus, new strategies for surgical education are emerging to improve the flaws of the present apprenticeship model. Although challenging, evidence suggests that improving training methods away from the hospital environment could be transferable to the operating room while allowing residents to focus on more complex issues. Therefore, current interest in this area is justifiable [8].

In this sense, simulation-based training (SBT), if appropriately integrated into surgeons’ curricula, may provide a time-efficient, cost-effective and safe method of training [3]. This strategy has already proved to be effective in improving basic technical skills in subjects with no previous experience in laparoscopic surgery [8–11].

The delivery of training or educational programs by electronic means, defined as electronic learning (e-learning), can potentially improve the efficiency and effectiveness of surgical education [1, 12]. Video-based surgical learning (VBSL) is becoming a hallmark of surgical preparation [1, 13]. Additionally, short structured instructional videos containing didactic illustrations, narration and practical ‘tips and tricks’ are recognized as highly valued features for both residents and specialists [13].

Currently, standardized programs on SBT are used to evaluate residents’ basic laparoscopic skills [14]. In 2011, the Program for Laparoscopic Urologic Skills (PLUS) was introduced in Europe for the final-year Urology residents’ examination, and has been renamed as European Basic Laparoscopic Urologic Skills (E- BLUS) examination [15]. It contemplates five tasks, performed in SBT, including peg- transfer, pattern cutting, intracorporeal knot tying, clip-and-cut and needle guidance, aiming at basic laparoscopic skills evaluation [16].

In order to prepare for this examination, a theoretical course is proposed, alongside with visualization of descriptive videos of the exercises, which are available on the European Association of Urology (EAU) website [17, 18]. However, an explanatory video-mentored tutorial (VMT) is lacking.

With this study, we aim to develop a complementary approach by producing two VMT, concerning two exercises of E-BLUS examination: peg-transfer (PT) and needle-guidance (NG). In addition, we will compare the impact of narration and technical tips on exercise performance, regarding task time and total number of errors and evaluate how participants reacted to features of each VMT.

MATERIAL AND METHODS

Ethics

This project was approved by the Committee of Life and Health Sciences Research Institute. An informed and written consent, describing all the procedures and goals of this research protocol, was obtained before any data collection.

Study design and procedures

The project included two phases. In the first phase, a VMT for PT and NG exercises was created. The second phase consisted of an experimental study on the impact of the developed VMTs. Outcome measures were total number of errors and task time. A post-experiment questionnaire was administered.

Development of the video-mentored tutorial

The PT and the NG tasks were performed by an assistant physician and certified fellow of the European Board of Urology.

The entire execution of the exercises was recorded with a laparoscopic camera and an external 600D Canon^® camera placed on a tripod. Didactic illustrations for each exercise were created using CorelDraw X7^® software. In order to include narration throughout the exercises, a free online tool of text-to-speech software was used, namely www.fromtexttospeech.com.

Afterwards, videos were edited using Adobe Premiere Pro CC^® 2018 software, in order to create a video-mentored tutorial for each of the designed tasks, including didactic illustrations and narration, to provide practical ‘tips and tricks’ and clarification of critical task points.

Experimental study

The study was an experimental, unblinded, parallel, 2-intervention, 2-period, with an allocation ratio of 1:1, randomized trial.

Materials and laparoscopic tasks

The exercises were carried out on a single Karl-Storz working station, within a trainer box model with one fixed 10 mm, 300 degree laparoscope and Karl-Storz instruments.

The PT task required one dissector and one grasper. The NG task required two needle holders and 2-0 suture material.

Description of the exercises

1. Peg-transfer: six rubber rings are grasped, one at a time, with the left instrument, transferred midair to the right instrument, and placed on a peg in the opposite side. Once the six rings have been transferred, the process is reversed to complete the exercise. Time starts counting as soon as an object is touched. An error is recorded whenever an object is dropped or not transferred correctly.

2. Needle-guidance: a needle is guided through 10 metal rings along a depicted route marked by sequential numbers from 1 to 10. Time starts counting as soon as the needle passes the first ring and stops as soon the last ring is bypassed. An error is recorded whenever the needle is dropped, rings are bypassed, or the route is altered. When the task is not completed, the ring number where the participant stopped is noted.

Study sample and randomization

Starting September 2018, volunteer final year medical students were invited to participate in this study, through e-mail contact. No exclusion criteria were established.

For sample size determination, the interaction between video-mentored tutorial and participants performance in each exercise was considered. A minimum sample size of 40 participants was estimated on the basis of an expected large effect size (f = 0.8), a type I error of 0.05, and a power (1-β) of 0.80.

All volunteers that met the inclusion criteria were enrolled in the experimental session. Since a pre-experiment inquiry allowed to verify that none of the participants had previous experience with laparoscopic training procedures, E-BLUS examination exercises or ever had visualized the E-BLUS videos, participants were randomly allocated into two parallel groups, through a simple method, with an allocation ratio of 1:1, to Group 1 and Group 2.

Experimental session

Participants were instructed that they could take five minutes to practice and get familiar with both tasks, the trainer box model and the laparoscopic instruments. A written descriptive guide of the designed tasks was available and given to the participants. After the familiarization period, participants watched a video according to the predetermined sequence and task for the group.

In the first trial, Group 1 watched the VMT, whereas Group 2 watched the original E-BLUS examination video (EBLUS-V). Subsequently, all participants started the exercises. In the second trial, Group 1 watched again the VMT, whereas Group 2 watched the VMT for the first time (Table 1). In both trials and groups, the first exercise to be performed was the PT followed by the NG.

Table 1.

Schematic representation of the study design

Design	Group 1	Group 2
Trial 1	VMT	EBLUS-V
Trial 2	VMT	VMT

VMT – video-mentored tutorial; EBLUS-V – original European Basic Laparoscopic Urologic Skills examination videos

Participants were instructed that no maximum time was established, and exercises could be continued despite the number of errors. However, when participants demonstrated signs of fatigue, frustration or were unwilling to continue the exercise was interrupted.

All exercises were evaluated separately by the same evaluator according to the endpoints defined above in the ‘Laparoscopic task description’ topic (Appendix 1). At the end of the experimental session, the EBLUS-V was shown to Group 1 participants. A post-experiment questionnaire was then completed.

Questionnaires

Participants answered 2 different questionnaires: a pre-experiment and a post-experiment version (Appendix 2 and 3, respectively).

The post-experiment questionnaire (Appendix 3) consisted of a five-level Likert item regarding the VMT. Participants were also asked to score the usefulness of both videos and to state which video feature they valued the most. A preference section was designed so participants could compare VMT and EBLUS-V. Lastly, participants were invited to write a brief comment on difficulties felt and features that they would like to see improved in the VMT. Since no fully validated questionnaires are available concerning the study’s purposes, this form was developed by the authors.

Statistical analysis

The statistical analysis was performed with IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp. Release 2016.

Data normality was assessed through the Shapiro-Wilk test (n <30), skewness, kurtosis and visual evaluation of the histograms. Normality assumption could not be assumed; therefore, for each exercise we used non-parametric statistics. Wilcoxon-Signed Rank test was used to compare task time and total number of errors within- groups in between trials. To compare task time and number of errors between groups in the same trial number, a Mann-Whitney U Test was used. Median and interquartile ranges were used to report descriptive results. A p value of less than 0.05 was considered statistically significant and the confidence interval used was 95%. Additionally, we provide r-values as effect size approximations.

A descriptive analysis was performed to interpret the post-experiment inquiry responses. Descriptive statistics were presented as frequencies (n) and percentages (%). In the NG exercise, when task time was analyzed, participants who did not complete more than 50% of the task (calculated on the basis of number of rings surpassed in a total of 10) were excluded and the analysis was carried out without them. However, since most of the results were the same, all participants were included in the analysis.

RESULTS

Experimental study

Participants were forty-two final year medical students (n = 42), with no previous experience with laparoscopy or laparoscopic exercises. Two randomized groups were formed: Group 1 (n = 21) and Group 2 (n = 21).

Peg-transfer

Table 2 summarizes the between and within-groups comparisons concerning median task time. No statistically significant differences between groups were found, in the first or second trial. However, in the second trial, after watching the VMT, participants of both groups had a statistically significant decrease in median task time, in comparison to the first trial (p <0.001; p = 0.003; Group 1 and 2, respectively).

Table 2.

Between (horizontally) and within-groups (vertically) comparison of median task time (seconds) in the peg-transfer exercise

	Task time (seconds)
	Group 1: VMT/VMT Median (IQR) (n = 21)	Group 2:EBLUS-V/VMT Median (IQR) (n = 21)	Test statisticsa
Trial 1	233 (102)	210 (126)	U = 191.50; p = 0.466; r = -0.11
Trial 2	159 (67)	174 (79.5)	U = 193.00; p = 0.489; r = -0.10
Test statisticsb	Z = -3.752; p <0.001*; r = -0.58	Z = -2.938; p = 0.003*; r = -0.45

VMT – video-mentored tutorial; EBLUS-V – original European Basic Laparoscopic Urologic Skills examination videos; IQR – interquartile range;

^a– Mann-Whitney U Test;

^b– Wilcoxon-Signed Rank test;

^*– p-value <0.05

Table 3 summarizes the between and within-groups comparisons concerning the median total number of errors. In the first trial, a lower median in total number of errors was seen in participants who watched the VMT, in comparison to the participants who watched the EBLUS-V (p = 0.001). However, in the second trial, when both groups watched the VMT, no statistically significant differences between groups were found. Furthermore, participants who previously watched the EBLUS-V had a statistically significant decrease in total number of errors in the second trial, after watching the VMT (p = 0.001).

Table 3.

Between (horizontally) and within-groups (vertically) comparison of median total number of errors in the peg-transfer exercise

	Total number of errors (n)
	Group 1: VMT/VMT Median (IQR) (n = 21)	Group 2: EBLUS-V/VMT Median (IQR) (n = 21)	Test statisticsa
Trial 1	0 (1)	2 (4)	U = 94.50; p = 0.001*; r = -0.51
Trial 2	0 (1)	0 (1)	U = 198.50; p = 0.514; r = -0.10
Test statisticsb	Z = -1.387; p = 0.166; r = -0.21	Z = -3.194; p = 0.001*; r = -0.49

VMT – video-mentored tutorial; EBLUS-V – original European Basic Laparoscopic Urologic Skills examination videos; IQR– interquartile range;

^a– Mann-Whitney U Test;

^b– Wilcoxon-Signed Rank test;

^*– p- value <0.05

Needle-guidance

When compared to the PT exercise, participants exhibited more difficulties performing this particular exercise, and some participants could not complete the whole exercise. In fact, only 19 participants in Group 1 and 14 participants in Group 2 completed the exercise.

Table 4 summarizes the within-groups comparisons concerning median task time. A statistically significant decrease in median task time was only observed in participants who visualized the VMT in the first and second trial (p = 0.007). However, when we only included participants that completed more than 50% of this exercise, in both trials, we found different results. In the second trial, after watching the VMT, participants of both groups had a statistically significant decrease in median task time in comparison to the first trial (p = 0.005; p = 0.010; Group 1 and 2, respectively).

Table 4.

Within-groups comparison of median task time (seconds) in the needle-guidance exercise, firstly when all participants were included and secondly when only participants that completed more than 50% of this exercise in both trials were included

	All participants		Participants who finished >50%
	Task time (seconds)		Task time (seconds)
	Group 1: VMT/VMT Median (IQR) (n = 21)	Group 2: EBLUS-V/VMT Median (IQR) (n = 21)	Group 1: VMT/VMT Median (IQR) (n = 19)	Group 2: EBLUS-V/VMT Median (IQR) (n = 14)
Trial 1	525 (134)	480 (123)	540 (124)	540 (151)
Trial 2	451 (123)	450 (151)	435 (135)	450 (155)
Test Statisticsb	Z = -2.711; p = 0.007*; r = -0.42	Z =-1.699; p = 0.089; r = -0.26	Z = -2.817; p = 0.005*; r =-0.43	Z = -2.587; p = 0.010*; r = -0.40

VMT – video-mentored tutorial; EBLUS-V – original European Basic Laparoscopic Urologic Skills examination videos; IQR – interquartile range; ^a – Mann-Whitney U Test;

^b– Wilcoxon-Signed Rank test;

^*– p-value <0.05

Table 5 summarizes the between and within-groups comparisons concerning median total number of errors. In the first trial, a lower median in total number of errors was seen in participants who watched the VMT, in comparison to participants who watched the EBLUS-V (p = 0.014). However, in the second trial, when both groups watched the VMT, no statistically significant differences between groups were found. Furthermore, participants who had previously watched the EBLUS-V had a statistically significant decrease in total number of errors in the second trial, after watching the VMT (p = 0.002).

Table 5.

Between (horizontally) and within-groups (vertically) comparison of median total number of errors in the needle-guidance exercise

	Total number of errors (n)
	Group 1: VMT/VMT Median (IQR) (n = 21)	Group 2: EBLUS-V/VMT Median (IQR) (n = 21)	Test statisticsa
Trial 1	0 (1)	1 (2)	U = 130.00; p = 0.014*; r = -0.38
Trial 2	0 (0)	0 (1)	U = 168.00; p = 0.083; r = -0.27
Test statisticsb	Z = -1.897; p = 0.058; r = -0.29	Z = -3.169; p = 0.002*; r = -0.49

VMT – video-mentored tutorial; EBLUS-V – original European Basic Laparoscopic Urologic Skills examination videos; IQR – Interquartile Range;

^a– Mann-Whitney U Test;

^b– Wilcoxon-Signed Rank test;

^*– p- value <0.05

Post-experiment questionnaire

All participants filled a post-experiment questionnaire, and a total of 42 responses were registered. To analyze participants’ opinion on each video, we combined participants’ responses ‘Totally Agree’ and ‘Agree’ into an ‘Agree’ category, and the ‘Totally Disagree’ and ‘Disagree’ into a ‘Disagree’ category.

Concerning the PT exercise, primary difficulties reported were: lack of depth perception, lack of ambidexterity coordination and difficulties in instrument handling. Regarding the PT-VMT, 38 participants (90.5%) agreed that video length was appropriate and 41 participants (97.6%) agreed that narration was thorough, succinct and clear. Additionally, the same proportion of participants agreed that the ‘tips and tricks’ were simple, succinct, clear and helpful. All 42 (100%) participants agreed that the didactic illustrations were thorough and clear, the narration was helpful and that, globally, VMT represented a useful learning resource. When compared to the original EBLUS-V, all 42 participants agreed that the presence of didactic illustrations, ‘tips and tricks’ and clarification of critical task points on the VMT enhances learning capabilities. Participants scored, on a scale from 0–10, the usefulness of the PT-VMT to improve performance with a mean and standard deviation of 8.95 ±1.27.

Concerning the NG exercise, primary difficulties were: instrument handling, lack of depth perception and needle orientation. Regarding the NG-VMT, 29 participants (69.1%) agreed that video length was appropriate. Of the 42 participants, 41 (97.6%) agreed that narration was thorough, succinct, clear and 38 (90.5%) thought that narration was helpful. Also, 39 participants agreed that the ‘tips and tricks’ were simple, succinct and clear. All 42 participants agreed that the didactic illustrations were thorough and clear, and the specific ‘tips and tricks’ were useful.

Globally, all participants stated that VMT represents a useful learning resource to improve exercise performance. When compared to the original EBLUS-V, 40 participants (95.2%) agreed that the presence of didactic illustrations and clarification of critical task points, on the VMT, enhances learning capabilities. Also, 41 participants (97.6%) agreed that ‘tips and tricks’ enhances learning capabilities and all 42 participants agreed that, globally, VMT represents a better learning resource to improve exercise performance, in comparison to the original EBLUS-V. Participants scored, on a scale from 0–10, the usefulness of the NG-VMT to improve performance in this exercise, with a mean and standard deviation of 8.76 ±1.38.

Additionally, 31 participants (73.8%) stated that the most valued feature of both VMT was “the presence of ‘tips and tricks’.

DISCUSSION

In 2013, the 2011 E-BLUS examination results showed that the level of basic skills in laparoscopy among European residents was low, as most residents did not meet the examination pass criteria [6]. Thus, it seems reasonable that new strategies to overcome these results are urgent. In this study, we created the PT-VMT and NG-VMT, based on the experience of a certified FEBU member in order to enhance performance of both exercises, included in the 5 exercises of E-BLUS examination.

It is clear that to fully optimize teaching contents, especially practical skills, we must understand how motor skills are acquired. In this sense, we must consider the widely accepted Fitts and Posner’s theory of motor skills acquisition [19]. This theory argues that motor skills acquisition consists of three different stages: firstly, in the cognitive stage, the learner tries to understand the skills’ mechanics despite erratic execution. In the integrative stage, motor behavior and performance are gradually improved, and learners focus on specific performance details. Finally, in the autonomous stage, learner motor skills are performed unconsciously, and movements are more accurate, consistent and efficient.

Thus, based on the Fitts and Posner’s theory and these study findings, we must argue that VMT appears to be helpful in the cognitive phase of motor skills acquisition, as it enhances and eases learners’ capability to understand skills’ mechanics and task principles. This ultimately leads to a decreased number of errors, either on the PT and NG exercises, when compared to the original EBLUS-V.

Nevertheless, this direct effect was not observed regarding task time. Concerning the PT exercise, task time only successfully decreased in the second trial. For the NG exercise, the same results were observed only when participants that completed more than 50% of this exercise were included. This can be explained by the greater difficulty level associated with this exercise, as well as the lack of laparoscopic experience among participants.

Feedback and mentoring concerning critical task points and basic skill mechanics is essential to develop an efficient teaching and training method, in order to shorten the learning curve and optimize learning [19]. Based on this study results, we argue that using VMT containing didactic illustrations, ‘tips and tricks’, narration and clarification of critical task points represents a valuable strategy to improve performance and, ultimately, shorten the learning curve of basic laparoscopic skills.

Although the use of a non-validated questionnaire to evaluate participants’ preferences might be limiting, its use was essential to demonstrate that participants considered both VMTs useful, representing an appropriate learning resource that enhances exercise performance. Additionally, the presence of ‘tips and tricks’, highlighted by the expert, was the most valued video feature, possibly by anticipating common difficulties on critical task points. Furthermore, this feature could also enhance participants comprehension of skills’ mechanics.

The number of trials needed to fulfill the pass criteria of the E-BLUS examination exercises, with the use of the VMT, was not assessed, as this was not the scope of this investigation.

While this study only states a positive impact of the VMT on exercise performance, among medical students without previous laparoscopic experience, we might also expect similar results in surgical residents as it could improve laparoscopic skills acquisition and possibly enhance performance on E-BLUS examination. Ultimately, these skills could be transferable to the surgical environment in the operating room, raising surgeons’ performances and patients’ outcomes.

CONCLUSIONS

In summary, these study findings suggest that the use of VMT can lead to a smaller number of errors and, if combined with deliberate practice, could lead to shorter task time in exercise performance, among learners with no previous laparoscopic experience.

Additionally, this study unlocks new topics for further investigation, such as the impact on exercise performance of VMT among surgical residents. Furthermore, given the promising results stated in this study, production of new VMT to the remaining tasks of the E-BLUS examination could stand as a chance to improve preparation for this exam.

CONFLICTS OF INTEREST

The authors declare no conflicts of interest.

Appendix 1. Laparoscopic Task Description – Evaluation Form

First exercise: ‘Peg-transfer’:

• Errors:

Number of objects dropped: ___ objects.

Objects not transferred: ___ objects.

• Task time: _________ seconds.

• Task completed (despite number of errors):

  Yes / No.

Second Exercise: ‘Needle guidance’:

• Errors:

  Number of times that the needle fell: _____ times.

  Number of rings bypassed: _____ rings.

  Number of times that the route was altered: _____ times.

• Task time: __________seconds.

• Task completed (despite number of errors):

  Yes / No

Observation: Whenever the needle is dropped, clock time stops until needle is grasped by the needle drivers.

Appendix 2. Pre-experiment questionnaire

Personal data:

1. Name initials (follow this example: Vítor Martinho da Silva Fernandes-VMSF): ________

2. Age: ________ years.

3. Gender: A. Male B. Female

4. Mobile number: ______________________________

5. E-mail address: ______________________________

Technical data:

6. How many laparoscopic training procedures on a surgical box trainer have you performed? (circle your answer)

1. Less than thirty;

2. More than thirty;

7. Have you ever performed the European Basic Laparoscopic Urological Skills (E- BLUS) examination exercises? (circle your answer)

1. Yes;

2. No.

8. Have you ever watched the videos regarding the E-BLUS examination exercises available on the website of European Association of Urology (EAU)? (circle your answer)

1. Yes;

2. No.

9. Do you allow the use of these data for the purposes of this study, as defined in the formal written consent? (circle your answer)

1. Yes;

2. No.

To maintain confidentiality and protection of the provided data, please fill this section in order to be assigned a participant code according to:

• The first three letters of your city of birth;

• The last three digits of your mobile number;

• And the last two digits of your day of birth (use two digits to represent the day).

Follow this example:

• City of birth: Guimarães;

• Mobile Number: XXXXX543;

• Day of birth: 05.XX.XXXX

In this particular case, the code to fill would be: GUI543/05.

• The first three letters of your city of birth: _____

• Last three digits of your mobile phone number: ______

• The last two digits of your day of birth: _______

Participant Code: ______________/______

Appendix 3. Post-experiment questionnaire

Participant Code: _______________________________

1. Please state how much you agree or disagree with the statements bellow about the Video Mentored Tutorials. Please insert a single cross in the square that better fits your opinion.

2. On a scale from 0 to 10, with 0 being ‘not useful’ and 10 ‘very useful’, how would you globally rate:

   1. the usefulness of the Peg-Transfer Video Mentored Tutorial to improve performance in this exercise: _________________________________

   2. the usefulness of the Needle Guidance Video Mentored Tutorial to improve performance in this exercise: ____________________________

3. On a scale from 0 to 10, with 0 being ‘did not create any difficulties’ and 10 ‘raised very important difficulties’, how did you consider if the use of English as the language in the narration of the Video Mentored Tutorial created specific difficulties for you to perform the exercises correctly: _______________________________________________