Taking advantage of asynchronous digital feedback: development of an at-home basic suture skills training program for undergraduate medical students that facilitates skills retention

Abstract

Purpose

To date, there are no training programs for basic suturing that allow remote deliberate practice. This study seeks to evaluate the effectiveness of a basic suture skills training program and its 6-month skill retention applying unsupervised practice and remote digital feedback.

Methods

Fourth-year medical-student trainees reviewed instructional videos from a digital platform and performed unsupervised practice as needed at their homes. When they felt competent, trainees uploaded a video of themselves practicing the skill. In < 72 h, they received expert asynchronous digital feedback. The course had two theoretical stages and five video-based assessments, where trainees performed different suturing exercises. For the assessment, a global (GRS) and specific rating scale (SRS) were used, with a passing score of 20 points (max:25) and 15 (max:20), respectively. Results were compared to previously published work with in-person expert feedback (EF) and video-guided learning without feedback (VGL). A subgroup of trainees underwent a 6-month skills retention assessment.

Results

Two-hundred and forty-three trainees underwent the course between March and December 2021. A median GRS of 24 points was achieved in the final assessment, showing significantly higher scores (p < 0.001) than EF and VGL (20.5 and 15.5, respectively). Thirty-seven trainees underwent a 6-month skills retention assessment, improving in GRS (23.38 vs 24.03, p value = 0.06) and SRS (18.59 vs 19, p value = 0.07).

Conclusion

It is feasible to teach basic suture skills to undergraduate medical students using an unsupervised training course with remote and asynchronous feedback through a digital platform. This methodology allows continuous training with the repetition of quality practice, personalized feedback, and skills retention at 6 months.

Introduction

Simulation training in medical education enables trainees to practice a specific procedure or skill in a safe and controlled environment, without adding risk for patients. Through simulation, trainees can acquire both technical and non-technical skills. Due to its impactful results enabling the possibility for deliberate practice and the transfer of acquired skills to real-life scenarios, this methodology has been widely used to achieve competence for several medical procedures [1–9].

In the United States of America, the American College of Surgeons and the Association for Surgical Education have joined forces and developed a medical student simulation-based surgical skills curriculum [10]. Among the skills taught through these modules, the acquisition of basic suturing skills is part of third-year medical students’ curriculum. These skills have also been considered part of our institution’s 6-years undergraduate medical school curriculum, and are currently taught during their fourth year [11]. Traditionally, these skills were learned with in-person lectures and hands-on training at our simulation center using a synthetic skin model. These simulation sessions usually lasted 3 h and trainees were taught four different suturing techniques: simple interrupted suture, vertical mattress suture, interrupted and continuous subcuticular sutures [11, 12].

In 2020, due to the COVID-19 pandemic and the subsequent lockdowns, academic institutions needed to adapt their traditional teaching methods. One of the most widely used methodologies was incorporating digital platforms, such as Zoom^® and Microsoft Teams^®, to develop remote lectures and tutoring sessions [13, 14]. This rapidly increasing number of remote simulation training methodologies made it mandatory for the Society of Simulation in Healthcare to develop a dictionary that made the most frequently used terms understandable [15].

Our faculty of medicine and simulation center were not exempt from the need to adapt our traditional learning environment. To do so, a previously developed digital platform intended to teach laparoscopic skills with remote and asynchronous feedback was adapted to support other healthcare personnel and students’ training needs [16, 17]. This new methodology has the advantage of allowing both trainers and trainees to access the platform 24/7 using electronic devices with internet access. The technology provides them with more flexible schedules to complete a course or achieve proficiency in technical skills. Additionally, it allows trainees to receive personalized expert feedback through digital enhanced inputs.

The experience using this methodology with remote and asynchronous digital feedback at our institution showed that it is feasible to acquire a wide variety of skills such as basic and advanced laparoscopic skills [16], the use of personal protection equipment (PPE) and endotracheal intubation [18]. This showed us that not only can this methodology be used to provide technical skills, but it can also enable an easy scale-up, allowing to reach a large number of trainees with a limited amount of trainers.

This study aimed to assess skills acquisition in basic suturing skills using this new remote and asynchronous digital feedback methodology in undergraduate medical students while comparing it to previously published data from our institution, and further on, analyzing skills retention from this new learning platform.

Methods

Basic suturing skills course

As part of our institutions’ curriculum, all students undergoing fourth-year medical school (MS4) had to complete a standardized basic suture skills training course, which was adapted to the digital platform to enable continuous training with remote and asynchronous feedback using previously developed audiovisual material [19]. All trainees were provided with a basic suture skills sets by the Simulation Center that included: a needle driver, forceps, scissors, and 2–0 silk sutures (Fig. 1). When needed, trainees could ask for additional sutures throughout the course.

Fig. 1.

Basic suture skills set

The course lasted 3 weeks and consisted of seven stages, the first two included theoretical content regarding asepsis, use of sterile gown and gloves, types of sutures, surgical instruments, and anesthesia infiltration. The next five stages included a video-based assessment of the following exercises: simple interrupted suture, vertical mattress suture, interrupted and continuous subcuticular suture, and final assessment. For each exercise, trainees had to record themselves performing three stitches, except for the continuous subcuticular suture, where they only had to perform one suture. The number of stitches performed in every exercise was established based on prior studies to properly assess trainees’ skills [11, 12, 19]. Students were encouraged to practice at home and to carefully review the feedback provided by teachers through the platform. To properly pass the course, trainees were required to achieve a passing score in their final assessment, which consisted of performing three simple interrupted sutures in less than 3 min, and achieving a global rating and specific rating passing score (see Assessment).

Study design

Throughout March and December 2021, fourth-year medical students completed the course, they were then followed and a subgroup was re-assessed during their fifth year of medical school. It is relevant to mention that in our country, medical school is 6–7 years long. Their data were then extracted from the digital platform and further analyzed. As part of the usual teaching model, trainers knew the identity of the students; however, the research team was blind to the identity of those evaluated. Results from the final assessment were compared to previously published data from our institution, by Tejos et al. [12].

Additionally, a follow-up assessment was conducted incorporating only those trainees that underwent this course for the second time during their fifth year within 6–9 months from their first training. This subgroup consisted of 37 students that were included for analysis.

Assessment

Trainees were assessed through the digital platform by a group of five expert trainers defined as surgical simulation and research fellows of our institution with previous training in suturing skills and educational methods, who also underwent a standardization trial before the beginning of the course.

Two different previously validated assessment scales were used, the first one was a global rating scale (GRS) using a modified Objective Structured Assessment of Technical Skills (OSATS), with a maximum score of 25 points, with 20 points required to pass (Table 1) [11, 12, 20]. The second scale used was a task-specific rating scale (SRS), which consisted of a maximum of 20 points with at least 15 points needed to pass the evaluation (Table 2) [12].

Table 1.

Global rating scale (GRS) (Modified Objective Structured Assessment of Technical Skill scale used for skills assessment)

	1	2	3	4	5
Respect for tissue	Frequently used unnecessary force on tissue or caused damage by inappropriate use of instruments	–	Careful handling of tissue but occasionally caused inadvertent damage	–	Consistently handled tissue appropriately with minimal damage
Time and motion	Many unnecessary moves	–	Efficient time/motion, but some unnecessary moves	–	Economy of movement and maximum efficiency
Instrument handling	Repeatedly makes tentative or awkward moves with instruments	–	Competent use of instruments although occasionally appeared stiff and awkward	–	Fluid moves with instruments and no awkwardness
Flow of operation	Frequently stopped operating and seemed unsure of next move	–	Demonstrated some forward planning with reasonable progression of procedure	–	Obviously planned course of operation with effortless flow from one move to the next
Knowledge of specific procedure	Deficient knowledge needed specific instruction at most operative steps	–	Knew all important aspects of the operation	–	Demonstrated familiarity with all aspects of the operation

Table 2.

Task-specific rating scale (SRS) used for skills assessment

	1	2	3	4	5
Needle handling	Deficient needle handling; takes longer than expected and does not position correctly	–	Optimal needle handling; manages to position correctly, but sometimes takes longer than expected	–	Consistently handles needle correctly
Unskilled hand exposure	Does not use/partially uses non-dominating hand	–	Always uses non-dominating hand, but does not expose tissues correctly	–	Always uses non-dominating hand, exposing tissues correctly without harming them
Knotting	Does not know how to tie knots/incomplete knot -tying	–	Completes knot-tying, but sometimes hesitates or makes clumsy movements	–	Completes knot-tying with fluidity
Final product quality	Completes the exercise partially; suture does not affront structures	–	Completes the exercise affronting structures yet does not achieve symmetry regarding borders or distance between stitches	–	Completes the exercise affronting structures correctly, with symmetric and equidistant stitches

In addition, for their final assessment, trainees had to fulfill three interrupted sutures within 3 min; this cut-off time was established based on previous studies and experts' opinions [11, 12, 19].

Digital platform: remote, asynchronous, and digital personalized feedback

All trainees had access to the instructional material through the digital platform at any time from anywhere using their electronic devices (mobile phones, tablets, and notebooks) and internet access. As previously stated, the instructional material was divided into seven stages with up to three video tutorials of less than 5 min each, the first two stages were theoretical content and the following five were suturing exercises. Trainees had access to the assessment rating scales.

The course content was available 24 h a day, every day through the digital platform, ensuring time flexibility for both trainees and trainers. To begin the course, trainees had to access the platform with a user created through their institutional e-mail account, proceeding then to the course where they could review the video tutorials and content on demand. Afterward, trainees would practice at home as many times as needed until they felt competent, then they would record themselves with a video recording device (usually a tablet or smartphone) and upload the video to the platform for assessment, and remote and asynchronous digital personalized feedback. Once the video was successfully uploaded, the trainer was notified by email and had to proceed with the evaluation within the next 72 h. If the trainee did not approve the exercise, he/she could upload as many new attempts as necessary to achieve the passing score. If the video recording was considered of poor quality, the trainers could ask trainees to re-upload a new video.

Digital feedback could be provided using four different inputs: onscreen drawings, texts, audio, and common mistakes videos. These inputs were added to the video timeline at the exact time the trainee committed the error, making every assessment specific and personalized. It is important to highlight that different types of feedback inputs could be used at the same time. An example of the feedback applied on the platform can be seen in Fig. 2.

Fig. 2.

Digital platform instructor’s view. On the right side, there is a list of the feedback inputs provided and the exact time they were added to the video, this can also be seen on the video timeline (vertical mark-lines)

Trainers could review the video in just one access or pause their evaluation and save the feedback and assessment provided to finish later. After the trainer completed the assessment, the trainees received a notification announcing that the video had been assessed. Afterward, the trainee could access the video with the feedback provided by the trainer and incorporate it into his/her performance, improving it. It is relevant to mention that trainees did not have access to their peers' videos or feedback, but they could access their assessments and feedback on demand. The platform’s learning cycle can be seen in Fig. 3.

Fig. 3.

Digital platform learning cycle

In case trainees had questions regarding the proper execution of the exercises, the platform enables a forum section where public questions can be asked to all trainers. In addition, when uploading the video, the platform enables a free text section where direct and private questions can be made by trainees to their trainers. This free text section is also available when receiving feedback, and can be replied by trainers to the trainee privately with a maximum of three interactions.

Skills acquisition

To assess skills acquisition, the results obtained from this new methodology were compared to previously published data from our institution; Tejos et al. [12]. In this previous study, the same basic suturing skills curriculum was used with undergraduate medical students being randomized and taught basic suturing skills, but using in-person expert feedback (EF) and video-guided learning without feedback (VGL). The results from that study reported skills acquisition from video recordings of students performing three simple interrupted stitches. This study showed that VGL by itself is not enough to achieve proper suturing skills when compared to EF. These results were compared to those obtained by trainees receiving remote and asynchronous digital during the final assessment stage, where they performed three simple interrupted stitches.

Six-month follow-up

As part of our medical school curriculum, all of the trainees that participated in this study were required to undergo this training course for a second time as part of their surgical rotation during their fifth year of medical school. It is important to mention that they were divided into small groups of 6–8 trainees for these clinical clerkships, rotating between groups every 6 weeks throughout the year. This did not allow them to complete the course for a second time simultaneously. Therefore, for the purposes of this study, to analyze mid-term skills retention, only the trainees that underwent the course for a second time between 6–9 months from the first assessment were included. This analysis was developed by comparing the results achieved in each exercise during the course in the fourth and fifth years.

Statistical analysis

As a primary outcome, final assessment results were compared to previously published work using in-person expert feedback and video-guided learning without feedback [12] using the R Studio program and non-parametric tests [21]. Meanwhile, for skills retention, paired samples with two-tailed t tests were used. Finally, as a complement, descriptive statistics were used for feedback inputs.

Results

Among the 243 trainees that participated in the course throughout March and December 2021, 238 uploaded a video during the final assessment. When comparing the use of this new remote learning method using digital enhanced asynchronous feedback with our previous results (Tejos et al.), we find it to be superior to in-person feedback or video-guided learning without feedback methods, presenting higher results in both GRS [24 vs 20.5 vs 15.5] and SRS score [19 vs 16.5 vs 13.3] (p value: < 0.001). (Table 3).

Table 3.

Summary of results obtained through the different training methods

	Number of trainees	GRS score [median (IQR)]	SRS score [median (IQR)]	p value
Remote and asynchronous feedback (remote)	238	24 (23–25)	19 (17.3–20)
In-person expert feedback (in-person)	42	20.5 (20–21)	16.5 (16–17.5)	< 0.001*
Video-guided learning without feedback (VGL)	44	15.5 (15–16)	13.3 (12.5–14)	< 0.001*

*p values are obtained from comparisons with remote and asynchronous feedback

When it comes to other exercises developed during the course, we found that 190 out of 243 trainees approved all the suturing exercises. The exercise with the highest passing rate was the final assessment, consisting of completing three simple interrupted stitches in less than 3 min, with 95.88% of trainees approving it. Meanwhile, the exercise with the lowest passing rate was interrupted subcuticular suture, with only 86.42% of trainees passing the stage. In further analysis of quantitative results, we observed that the lowest GRS scores were observed in continuous subcuticular and vertical mattress sutures (Table 4).

Table 4.

Results obtained through the course, divided by exercise

	Pass [n (%)]	GRS score*	SRS score*	Videos uploaded per trainee [median (IQR)]	Total videos
Simple interrupted suture	225 (92.59)	22.69	17.96	1 (1–2)	363
Vertical mattress suture	215 (88.48)	22.61	18.26	1 (1–2)	348
Interrupted subcuticular suture	210 (86.42)	22.7	18.16	1 (1–2)	355
Continuous subcuticular suture	217 (89.3)	22.67	16.97	1 (1–1)	291
Final assessment	233 (95.88)	23.75	18.52	1 (1–1)	269

*Mean obtained with passing exercises

Regarding skills retention during fifth year of medical school, 37 trainees were re-assessed. Among this subgroup, we observed they achieved higher GRS scores in their follow-up sessions in three out of five exercises, the exceptions were simple interrupted suture (p value = 0.12) and final assessment (p value = 0.06). Similar results were observed in SRS scores with only the final assessment having no statistically significant differences (p value = 0.07). Results from each exercise can be seen in Table 5.

Table 5.

Results obtained from the comparison between MS4 and MS5 subgroup

	GRS score*		p value	SRS score*		p value
	MS4	MS5		MS4	MS5
Simple interrupted suture	22.08 (19–25)	22.8 (19–25)	0.12	17.44 (14–20)	18.42 (16–20)	< 0.05
Vertical mattress suture	21.47 (12–25)	22.58 (19–25)	< 0.05	17.53 (15–20)	18.44 (15–20)	< 0.05
Interrupted subcuticular suture	22.06 (18–25)	23.03 (19–25)	< 0.05	17.56 (13–20)	18.5 (15–20)	< 0.05
Continuous subcuticular suture	22.33 (20–25)	23.07 (20–25)	< 0.05	17.97 (16–20)	18.7 (17–20)	< 0.05
Final assessment	23.38 (20–25)	24.03 (21–25)	0.06	18.59 (16–20)	19 (16–20)	0.07

MS4 fourth-year medical school students, MS5 fifth-year medical school students

*Results are expressed as mean (range)

Feedback metrics

As previously mentioned, the digital platform allows providing feedback using four tools: texts, drawings, common mistakes, and audios. A total of 10,587 feedback inputs were provided, with a mean of 6.51 inputs per video assessed. The most common inputs used were texts and drawings with 8014 and 2068 inputs, respectively. Further details of the feedback inputs and their distribution per exercise can be found in Table 6.

Table 6.

Feedback inputs distribution per exercise

Feedback input n (%)	Texts	Drawings	Common mistakes	Audios	Mean per video	Total
Simple interrupted suture	1956 (75.2)	507 (19.5)	125 (4.8)	8 (0.3)	7.15	2596
Vertical mattress suture	1913 (73.7)	538 (20.7)	108 (4.2)	35 (1.4)	7.45	2594
Interrupted subcuticular suture	1770 (76.8)	390 (16.9)	132 (5.7)	12 (0.5)	6.49	2304
Continuous subcuticular suture	1330 (75.3)	394 (22.3)	29 (1.6)	14 (0.8)	6.07	1767
Final assessment	1045 (78.8)	239 (18)	38 (2.9)	4 (0.3)	4.93	1326
Total	8014 (75.7)	2,068 (19.5)	432 (4.1)	73 (0.7)	6.51	10,587

Discussion

Simulation-based learning has been part of the surgical curriculum for several years both in undergraduate and postgraduate contexts, showing promising results in the acquisition of skills while maintaining a safe environment. During the past 2 years, our traditional simulation training had to be modified, incorporating remote training and multiple technological resources to ensure quality teaching [13, 14].

Although the implementation of video-based assessments and digital feedback has been going on for a few years, its implementation rose exponentially between 2020 and 2021 due to the COVID-19 pandemic [13, 14, 22–24]. Among these, we found several studies that assessed the effectiveness of remote or distance-based simulation training for basic suturing skills with promising results [25–28]. Even though these studies did answer to the circumstances of the pandemic and allowed to maintain training, there is still a need for trainers to be synchronously present during the tutoring sessions to provide feedback. This can still be hard to achieve since most trainers have scarce time available for education due to their clinical workload [29, 30].

Unlike previously published studies, the digital platform and teaching methodology implemented in this study allowed time flexibility for both trainers and trainees. First, it allowed trainers to assess videos and provide individualized feedback at any time and from anywhere in the world, without the need to find available time away from clinical labor to teach the trainee [25–27, 29, 30]. Second, it provided the trainee the chance to train from any place they want and as many times as they feel comfortable while having unlimited access to the video-enhanced feedback received.

Also, it allowed us as educators to understand the needs of trainees and what exercises may be hard to reach proficiency, while having easy access to the platform data for analysis. For example, we can see in Table 4 that a group of trainees (> 25%) required further training and feedback from experts to achieve proficiency in the simple interrupted suture, vertical mattress suture, and interrupted subcuticular suture. This new methodology enabled trainees to have quality practice, repeat training sessions, and learn at their own pace while receiving personalized expert feedback as needed. In addition, this methodology allowed to implement a course with a large number of trainees (243) with a limited number of expert trainers (5). This could reflect that providing time flexibility to trainers and the possibility for assessing from wherever they prefer, may allow easy scale-up of any course, and help reduce expert tutors' time scarcity for simulated training [29, 30].

Limitations

One of the primary outcomes analyzed for this study was the acquisition of skills in the final assessment, performing three simple interrupted sutures and comparing them to previously published data [12]. Randomization of a single sample could not be implemented since our country underwent some of the most strict lockdowns during the pandemic, setting cities under complete quarantine for over 6 months. This forced our institution to not implement hybrid activities until mid-2021 and in-person activities until March 2022, to prevent students from contracting COVID-19 [31]. Therefore, there were different cohorts of both trainees and trainers. However, both samples are comparable since they share their main characteristics such as institution and basic medical curriculum. When developing these comparisons, it is also important to notice that, for the previous article, the video recordings were performed with standardized settings at our simulation center, meanwhile the current cohort recorded themselves with heterogeneous settings in both hardware (cellphone, tablet, computer, among others) and location. Also, this heterogeneity of video recordings, on occasion, led to trainees having to upload a new video that allowed trainers to properly assess the exercise with a good view. Unfortunately, data on how many videos had to be re-uploaded were not recorded.

On the other hand, there is no information related to how and where the trainees practiced, or if they received help from others (tutors, peers, family, or others). This study opens the field to continue research related to autonomous unsupervised training, making it necessary to assess in further detail the hidden curriculum of trainees. There are several questions that may be interesting to analyze in posterior studies: Did they practice alone or received help? Did they watch additional tutorials online? How much time do trainees invest in practicing at home, and whether there is heterogeneity in these times? How much do these vary from one trainee to another? How confident did they feel performing the exercises? Unfortunately, so far we can only extract from our data that a group of trainees required repeated sessions with expert feedback to achieve proficiency in some basic suturing skills, although the time invested by each of them remains unknown. Then it is not feasible with this data to develop learning curves associated with basic suturing skills learning.

When it comes to trainers’ time requirements, the time to assess each video was not recorded; therefore, it is not feasible to determine the time efficiency of trainers. Although, they estimate that it took them between 5 and 7 min to provide feedback for each video, dedicating 1–2 h per week to assess trainees.

Additionally, since there is also no data available on unsupervised at-home practice when analyzing data from the 6-months follow-up, it remains unknown if trainees practiced the skills before uploading the videos. Did they have any unofficial clinical practice that may add experience? Therefore, the data may not reflect proper skills retention since it could be the result of repeated practice.

Despite the limitations previously mentioned, the current study shows us that it is feasible to teach at least four different types of suturing skills through a remote and asynchronous learning environment, with 78.19% (190 out of 243) of the trainees approving all exercises and an overall passing rate of over 85% in each exercise. In addition, when looking at the results from the subgroup that underwent follow-up, we observed that simple suturing proficiency may have been achieved during the first training course. Meanwhile, a trend for improvement in more complex suturing skills was observed during the follow-up.

Finally, from these results, we can also establish the first data related to the use of multiple feedback inputs, proving what we thought to be true, most trainers use several inputs and provide more than five personalized inputs in almost all the exercises. The use of this new technology enables us as a medical education community to discover more about feedback impact, especially remote and asynchronous; and skills acquisition related to it. Further analysis should be developed to analyze the educational impact of the different types of feedback inputs to determine their effectiveness and optimize their use.

Conclusion

We present the successful adaptation of a traditional suture course to a remote digital methodology, where trainees and trainers can interact at their own pace. Under this digital environment and regarding our suture training program, trainees not only reached the course objectives but also did better than learners trained through the traditional method when looking at assessment scales scores. Although, a further analysis comparing mid-term skills retention of this new methodology with traditional learning is still needed.

We also demonstrated that the knowledge acquired is significant and skills proficiency for simple suturing can be achieved with just one learning course.

We believe that since this new methodology enables repeated training with learning-centered feedback, it might bring us closer to providing trainees with the opportunity for real deliberate practice.

Funding

Dr. Julian Varas is the Founder of Training Competence, an official spinoff startup from the Pontificia Universidad Católica de Chile. Dr. Gabriel Escalona is the chief product manager of this startup. Dr. Francisca Belmar, Dr. María Inés Gaete, and Ignacio Villagrán are consultants of this startup. Training Competence and the Pontificia Universidad Católica de Chile are the owners of the rights and distribution of C1DO1 platform used for the assessment in this study.

Data availability

If any reader is interested in receiving the data associated with this research, the authors are available to provide it while protecting the identity of trainees.

Declarations

Conflict of interest

Slavka Chelebifski, Catalina Ortiz, Cristián Jarry, Valentina Durán, Adnan Alseidi, Elga Zamorano, Fernando Pimentel, and Fernando Crovari have nothing to disclose.