Abstract
Intro
The effects of refraining from practice for different intervals on laparoscopic suturing and mental workload was assessed with a secondary task developed by the authors. We expected the inability to practice to produce a decrease in performance on the suturing, knot tying, and secondary tasks and skills to rebound after a single refresher session.
Methods
22 surgical assistant and premedical students trained to FLS proficiency in intracorporeal suturing and knot tying were assessed on that task using a secondary task. Participants refrained from practicing any FLS laparoscopic tasks for 1 or 5 months. At the time of their return, they were assessed immediately on suturing and knot tying with the secondary task, practiced suturing and knot tying for 40 minutes, and then were reassessed.
Results
The mean suture times from the initial reassessment were greater than the proficiency times but returned to proficiency levels after one practice session, F(2, 40) = 14.5, p < .001, partial η2 = .420. Secondary task scores mirrored the results of suturing time, F(2, 40) = 6.128, p < .005, partial η2 = .235, and were moderated by retention interval.
Conclusions
When participants who reached proficiency in suturing and knot tying were reassessed after either 1or 5 months without practice, their performance times increased by 35% and secondary task scores decreased by 30%. These deficits, however, were nearly reversed after a single refresher session.
Keywords: simulation training, skills retention, mental workload, laparoscopy
Introduction
Laparoscopy has become a standard method for performing a growing number of surgical procedures. Although laparoscopy provides benefits to the patient, it imposes substantial visual-spatial and mental workload challenges on the surgeon to develop and hone the required skills.1,2 Thus, many surgery training programs have turned to simulation to train laparoscopic skills outside of the operating room.3 The American Board of Surgery now requires surgical trainees to demonstrate proficiency in the Fundamentals of Laparoscopic Surgery (FLS) program designed by the Society of American Gastrointestinal and Endoscopic Surgeons to teach and assess surgeon's cognitive and psychomotor skills.
Laparoscopy places substantial demands on visual attention, but the surgeon must still maintain the attentional capacity to monitor ongoing activity in the operating room and plan for the next steps of the operation. At present, there is no standard method to measure the mental demands imposed by laparoscopic surgery. Recently, however, several investigators have begun using the secondary task technique.4,5 Individuals are asked to perform a laparoscopic task and another task simultaneously. According to the multiple resource theory of attention,6 if a secondary task competes for the same visual-spatial attentional resources needed for laparoscopy, then performance on one or both tasks is likely to degrade. Thus, when an individual directs his/her attention to the primary task (i.e., the laparoscopic task), performance on the secondary task should derease as the difficulty of the primary task increases, reflecting changes in spare attentional capacity or mental workload.7 Stefanidis et al.5 used this secondary task method to assess laparoscopic workload and found that the secondary task revealed subtle differences in attentional skill not present in the laparoscopic performance measures.
Our group has developed a new, secondary task specifically for laparoscopic procedures that superimposes a visual-spatial task directly onto the laparoscopic display, producing a combined video image with both tasks falling into the same focal field and requiring the same visual attentional resources. Initial studies have shown that this secondary task is sensitive to differences according to surgical experience, laparoscopic training tasks, and the transition from FLS to fresh cadavers.8,9,10
Several investigators have attempted to measure how well laparoscopic skills can be retained in the absence of practice with different results. Khan et al.11 found no changes in suturing skill over a 6-month interval, while van Bruwaene et al. found some decrease in skill after 10 weeks.12 The goal of the present study was to use the new spatial secondary task to measure potential differences in mental workload associated with laparoscopic suturing over different retention intervals.
We hypothesized that there would be a decrease in performance of suturing and knot tying on the initial reassessment at 1 or 5 months and that this deficit would be accompanied by an increase in mental workload reflected by low secondary task scores. We also assessed the impact of a single refresher session and expected to see an improvement in the performance of suturing and knot tying skills with less mental workload indexed by higher secondary task scores. Last, we expected any decrease in skill to be more pronounced for the 5-month compared to the 1-month interval.
Methods
Participants
Twenty-seven participants were recruited from the Master of Surgical Assisting (SA) program at Eastern Virginia Medical School (n=22) and the premedical classes at UNC-Charlotte (n=5). Participants were required to be unable to practice laparoscopic tasks during the retention intervals.
Primary task
Participants were assessed on the FLS intracorporeal suturing and knot tying task after having been trained previously to proficiency on this task. They were provided with two laparoscopic needle drivers, one endoscopic scissor, and silk sutures cut to 6 inches. The suturing and knot tying task was performed on a basic box trainer. Participants were allowed two trials in immediate succession when assessed on the suturing and knot tying task.
Secondary task
The secondary task was a visual-spatial detection task called the ball-and-tunnel task.13 In this task, four spheres/balls are presented in a representation of a 3D tunnel displayed at 50% transparency on the laparoscopic display. Depth perspective is conveyed in a “tunnel” comprised of small dots that decrease in size and relative distance toward the center of the image. In the standard configuration, balls are located at the 12, 3, 6, and 9 o'clock positions within the tunnel. Participants attend to successive images and respond when they detect a change in the position any of any ball from the standard configuration (i.e. the target). A target is defined as one ball appearing to shift in depth, represented by a change in the diameter and location of the ball. When a ball moves closer, the diameter increases from 26 to 53 mm and shifts 53 mm from the center. When a ball moves farther away, the diameter decreases from 26mm to 11mm and shifts 11mm from the center. Only one ball changes position at any given time. Images were presented for 300 ms every 2 to 4 s with a mean of 3 s. Participants were instructed to respond to targets using a foot switch. There was no importance placed on the color of the ball that moved, or the direction in which it moved. Performance on the ball-and-tunnel task was measured by proportion of correct detections, proportion of false alarms, and response times to correct detections.
Procedure
Participants were introduced to the ball-and-tunnel task and performed this task by itself for 3 min to ensure they could detect the targets. This activity provided a baseline measure of secondary task performance in the absence of competing attentional demands.
All participants were novices and had been trained to proficiency using the FLS tasks. They practiced the peg transfer and cutting tasks from the FLS curriculum until they achieved the expert criteria described by Ritter and Scott14, at which point they were then trained to proficiency on the FLS intracorporeal suturing task. To reach proficiency, they had to complete an accurate and a secure knot within 112 s, as described by Ritter and Scott. The students’ schedules allowed for 4 formal, 1-hour training sessions. At the end of each training session, they were assessed with the ball-and-tunnel task.
In the second phase of the experiment, both groups were assessed for long-term retention after one of two retention intervals. Half of the participants were assessed after 1 month and the other half after 5 months. No student practiced any of the FLS laparoscopic skills during the retention interval. The students were assessed on their initial sutures and the secondary task immediately on arrival for the post-retention interval session. Afterward, they were allowed to practice suturing and knot tying for another 40 min and then were assessed again.
Results
Five participants withdrew from the study leaving 12 participants in the 1-month condition and 10 in the 5-month condition. There were 4 males and 18 females with a mean age of 25 years (SD = 4). All but two participants were right-handed, and all had normal or corrected-to-normal vision. The primary dependent measure of performance for the suturing and knot tying task was completion time. Except for the initial practice session, participants made too few needle placement and knot-tying errors to permit analysis.
Primary task analyses
The results showed that completion times were faster (lower) on the second trial, F(1, 20) = 24.321, p < .001, partial η2 = .549; but neither the effects of time of assessment nor retention interval differed between trial 1 and 2; therefore, the data were averaged across trials for the subsequent analyses. The mean completion times are shown in Table 1 for the participants in the 1 and 5-month retention interval groups. These data were analyzed with a one between, one repeated measure ANOVA and showed a significant effect of assessment, F(2, 40) = 14.5, p < .001, partial η2 = .420. The means were evaluated with a Bonferroni alpha correction (p = .017). Suture times were significantly greater on the initial reassessment compared to both the original proficiency level and the final reassessment. Although the suture times were higher for participants in the 5-month retention group, the mean difference between the retention intervals for the two groups did not reach significance, F(1, 20) = 2.153, p = .158.
Table 1.
Completion Times for Suturing and Knot Tying for each Retention Interval (mean;SDs in parentheses)
| Retention Grp. | Proficiency | Initial Reassessment | Final Reassessment |
|---|---|---|---|
| 1 month | 95.3 (29.6) | 120.1 (39.8) | 93.9 (22.2) |
| 5 month | 102.6 (28.2) | 145.5 (57.8) | 116.3 (41.3) |
| Mean | 99.0 (28.7) | 132.8 (49.5) | 105.1 (33.5)* |
The first column shows the proficiency score from the final practice session. The second column shows data from the initial reassessment after the retention interval. Participants then practiced suturing and were assessed again at the end of that session. Those data appear in the final reassessment column.
P < .017 between initial reassessment and both proficiency and final reassessment.
Secondary task analyses
These means for each assessment are shown in Table 2 separated for the two retention interval groups. The mean for the baseline performance on the ball-and-tunnel task by itself is included for comparison. The assessment data were analyzed with a one between, one repeated measure ANOVA. Significant effects, however, were observed for only the proportion of correct detections. There was an effect of assessment, F(2, 40) = 6.128, p < .005, partial η2 = .235, and the means were evaluated with a Bonferroni alpha correction (p = .017). The proportion of correct detections was significantly less on the initial reassessment compared to both proficiency and the final reassessment.
Table 2.
Proportion Correct Detections, False Alarms, and Response Times for each Retention Interval (mean;SDs in parentheses) on the Secondary Task
| Retention Grp. | Baseline | Proficiency | Initial Reassessment | Final Reassessment |
|---|---|---|---|---|
| P(Correct Detections) | ||||
| 1-Month | .960 (.034) | .485 (.291) | .431 (.215) | .586 (.253)* |
| 5-Month | .970 (.033) | .443 (.278) | .253 (.201) | .333 (.277)** |
| Mean | .464 (.279) | .342 (.223) | .459 (.288)*** | |
| P(False Alarms) | ||||
| 1-Month | .040 (.045) | .100 (.101) | .094 (.091) | .142 (.129) |
| 5-Month | .041 (.044) | .047 (.09) | .064(.132) | .068 (.063) |
| Mean | .074 (.105) | .079 (.115) | .105 (.107) | |
| Response Times (seconds) | ||||
| 1-Month | .713 (.101) | 1.028 (.222) | 1.028 (.211) | 1.070 (.271) |
| 5-Month | .744 (.181) | 1.110 (.492) | 1.114 (.434) | 1.063 (.306) |
| Mean | 1.069 (.382) | 1.086 (.326) | 1.065 (.312) |
P < .017 between the initial and final reassessment
P < .017 between proficiency and the initial reassessment
P < .017 between initial reassessment and both proficiency and final reassessment
The difference between the retention intervals did not reach significance, but there was a significant interaction between retention interval and assessment, F(2, 40) = 3.71, p < .033, partial η2 = .156. A comparison among the means with the Bonferroni alpha correction (p = .017) showed a significant increase in detections between the initial and final reassessments for participants in the 1-month retention group and a significant decrease in detections between proficiency and the initial reassessment for participants in the 5-month retention group. No other comparisons were statistically significant.
Discussion
The goal of this study was to examine how refraining from practice for different intervals would affect laparoscopic suturing performance and mental workload in novices trained to proficiency using a secondary task. Participants were trained to proficiency, ceased practicing for 1 or 5 months, and then returned for a retention test and refresher training session. They were assessed immediately on their return, given 40 minutes to practice, and assessed again. The results showed an increase in suturing and knot tying time over the retention interval indicative of decay in skill level. On average, suture times increased by 35% over the times achieved at proficiency. This difference, however, decreased to about 8% by the end of the 40-minute refresher session and was no longer statistically different from the proficiency times.
The lack of a significant difference between the 1- and 5-month retention interval groups was unexpected and may or may not be related to the small sample size. The literature on psychomotor skill suggests that greater intervals away from practice require more time to get back to levels attained previously.18 This effect, however, is moderated by the degree of automaticity underlying the skill.19,20 Skills that have reached automaticity have short reacquisition periods. Skills that have never reached automaticity often need extensive periods of time to reacquire, and that process demands effortful attention. The trends in our data suggest that those in the 5-month retention interval group may have needed to focus more attention on the suturing task, because they recognized their initial assessment performance was worse than what they had achieved at the end of their proficiency training.
Regarding the secondary task, the proportion of correct detections decreased by about 50% when performed with the laparoscopic task compared to the single task baseline condition. The results for the secondary task mirrored the changes in the primary task. At proficiency, participants were able to detect about 46% of the targets, suggesting they had additional attentional capacity to respond to secondary task targets. Consistent with the primary task results, when assessed immediately on return, the secondary task scores of the participants decreased by a little more than 25% compared to the level at proficiency but increased back to the proficiency level by the end of the 40-minute refresher session. This pattern of results, however, was related to the interval condition. For participants in the 1-month condition, the decrease in detections observed on initial reassessment was not significant, but their detection scores increased significantly by about 35% after 40 minutes of practice. By contrast, the decrease in detections observed on initial reassessment was significant for participants in the 5-month condition. Although these individuals were able to improve their detection performance by about 30% after practice, this increase was not statistically significant.
Collectively, this pattern of results shows that practice is important for maintaining proficiency on laparoscopic skills. The inability to practice over 1 or 5 months led to an increase in the participants’ suturing times significantly beyond proficiency levels. Although the idea that refraining from practice would be detrimental to skills is not surprising, the results from this study show that engaging in a single, 40-minute practice session enabled participants to reacquire their skills to near proficiency levels. Thus, these findings are consistent with those of van Bruwaene et al.12 and others 15,16 who also observed a decrease in skill after different retention intervals and subsequent improvement after retraining.
Perhaps the more important finding from the present study, however, was in the secondary task scores, which showed the source of the performance deficit is related to an increase in mental workload. Refraining from practice resulted not just in longer suturing times but also less attentional capacity to detect targets in the secondary task. Importantly, this decrease in secondary task performance was greater after the greater period of abstinence from practice. Moreover, as participants reacquired their suturing and knot tying skills during the 40-minute refresher session, their attentional capacity also increased as indexed by improved performance on the secondary task; this recovery was, however, slower for the secondary task scores compared to the suturing scores, especially for the participants in the longer retention interval, indicating that the secondary task metric may be more sensitive in detecting subtle changes in performance. Finally, it should be noted that the differences in mental workload were observed in the suturing times but not in accuracy. Participants rarely made any needle placement or knot-tying errors after achieving proficiency.
Our results differ from those of Khan et al.11 who did not find differences in FLS performance over retention intervals of 1, 3, or 6 months. There may be several reasons for this discrepancy. First, we focused solely on intracorproreal suturing and knot tying and not the set of all five FLS tasks studied by Khan et al. Second, we assessed our participants immediately on their return. By contrast, Khan et al. gave their participants two practice or warm-up trials on each FLS task prior to the formal assessment. Thus, it is possible that by the time Khan et al. conducted their formal assessment, any potential decreases in skills had already been reversed prior to the practice session. Further, the frequent reassessments during the retention interval in the study by Khan et al. may have acted as refresher training minimizing skill decay.
There are some limitations to the present study. First, as is often the case in retention studies, there was some attrition; five participants did not finish their training. Thus, the results presented here may be attributable to the more motivated individuals. Along those lines, the attrition may have affected the findings for retention interval. Clearly, the means suggest that 5 months without practice was more detrimental than 1 month, but the statistical power may have been too low with the final sample size to confirm that conclusion.
Another potential limitation is that we studied novices in surgery (surgical assistant students and undergraduates) which may have added additional confounding variables. Also, due to restrictions related to student schedules, the retraining session could not exceed 60 minutes. Participants were allowed 40 minutes to practice with the remaining time reserved for assessments. Our data suggest that 40 minutes of practice was sufficient for those in the 1-month retention interval group to reach their orginal proficiency times accompanied by a significant increase in secondary task scores. Participants in the 5-month retention interval group showed similar but nonsignificant trends suggesting they may have needed more than 40 minutes to hit their previous proficiency levels.
Last, these data are based entirely on skills acquired from an FLS simulator. Although one might expect that extended intervals without practice would also affect laparoscopic skills acquired in the OR, the extent of that potential deficit on primary and secondary task performance is unknown.
Conclusion
Our results showed that when novice participants who had achieved proficiency in suturing and knot tying on the FLS trainer were reassessed after some interval, there was a performance deficit; however, this decrease in both primary and secondary task performance was nearly reversed after 40 minutes of additional practice. Moreover, the results from this experiment show that the ball-and-tunnel secondary task can provide a complementary measure of mental workload and offers a standard index of mental workload allowing investigators to assess the relative differences among a variety of laparoscopic tasks with a common metric. This ability to detect subtle performance differences with the use of this secondary task may allow for a better assessment of when learning is complete during simulator training and thus maximize skill transfer to the clinical environment.17
Acknowledgments
This work was funded in part by a grant from the Agency for Healthcare Quality and Research (1R18HS020386-01) and support of the Modeling and Simulation Graduate Research Fellowship Program at Old Dominion University.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
The authors of this study have no conflict of interest to disclose.
References
- 1.Dekker SWA, Hugh TB. Laparoscopic bile duct injury: Understanding the psychology and heuristics of the error. ANZ J Surg. 2008;78:1109–1114. doi: 10.1111/j.1445-2197.2008.04761.x. [DOI] [PubMed] [Google Scholar]
- 2.Gallagher AG, Cowie R, Crothers I, Jordan-Black JA, Satava RM. An objective test of perceptual skill that predicts laparoscopic technical skill in three initial studies of laparoscopic performance. Surg Endosc. 2003;17:1468–1471. doi: 10.1007/s00464-002-8569-4. [DOI] [PubMed] [Google Scholar]
- 3.Korndorffer JR, Stefanidis D, Scott DJ. Laparoscopic skills laboratories: Current assessments and a call for resident training standards. Am J Surg. 2006;191:17–22. doi: 10.1016/j.amjsurg.2005.05.048. [DOI] [PubMed] [Google Scholar]
- 4.Hsu KE, Man F-Y, Gizicki RA, Feldman LS, Fried GM. Experienced surgeons can do more than one thing at a time: effect of distraction on performance of a simple laparoscopic and cognitive task by experienced and novice surgeons. Surg Endosc. 2008;22:196–201. doi: 10.1007/s00464-007-9452-0. [DOI] [PubMed] [Google Scholar]
- 5.Stefanidis D, Scerbo MW, Korndorffer JR, Scott DJ. Redefining simulator proficiency using automaticity theory. Am J Surg. 2007;193:502–506. doi: 10.1016/j.amjsurg.2006.11.010. [DOI] [PubMed] [Google Scholar]
- 6.Wickens CD. Multiple resources and performance prediction. Theoretical Issues in Ergonomic Science. 2002:159–177. [Google Scholar]
- 7.O'Donnell RD, Eggemeier FT. Workload assessment methodology. In: Boff K, Kaufman L, Thomas J, editors. Handbook of perception and human performance, Vol. 2: Cognitive processes and performance. John Wiley & Sons; New York: 1986. [Google Scholar]
- 8.Scerbo MW, Kennedy RA, Montano M, Britt RC, Davis SS, Stefanidis D. A spatial secondary task for measuring laparoscopic mental workload: Differences in surgical experience. Proceedings of the Human Factors and Ergonomics Society 57th Annual Meeting. 2013:728–732. [Google Scholar]
- 9.Prytz E, Montano M, Kennedy R, Scerbo M, Britt R, Davis S, Stefanidis D. Using a spatial task to measure laparoscopic mental workload: Initial results. Simul Healthc. 2012;7:511. [Google Scholar]
- 10.Britt RC, Scerbo MW, Montano M, Kennedy RA, Prytz E, Stefanidis D. Intracorporeal suturing: Transfer from FLS to cadavers results in substantial increase in mental workload. Surgery. 2015;158:1428–1423. doi: 10.1016/j.surg.2015.03.032. [DOI] [PubMed] [Google Scholar]
- 11.Khan MW, Lin D, Marlow N, Altree M, Babidge W, Field J, Hewett P, Maddern G. Laparoscopic skils maintenance: A randomized trial of virtual reality and box trainer simulators. J Surg Ed. 2014;71:79–84. doi: 10.1016/j.jsurg.2013.05.009. [DOI] [PubMed] [Google Scholar]
- 12.van Bruwaene S, Schijven MP, Miserez M. Maintenance training for laparoscopic suturing: the quest for the perfect timing and training model: a randomized trial. Surg Endosc. 2013;27:3823–3829. doi: 10.1007/s00464-013-2981-9. [DOI] [PubMed] [Google Scholar]
- 13.Scerbo MW, Croll MM, Garcia HM, Stefanidis D, Britt RC, Davis SS. A spatial task for measuring laparoscopic mental workload. Simul Healthc. 2012;7:558. [Google Scholar]
- 14.Ritter M, Scott DJ. Design of a proficiency-based skills training curriculum for the Fundamentals of Laparoscopic Surgery. Surg Innove. 2007;14:107–112. doi: 10.1177/1553350607302329. [DOI] [PubMed] [Google Scholar]
- 15.Castellvi AO, Hollett LA, Minhajuddin A, Hogg DC, Tesfay ST, Scott DJ. Maintaining proficiency after laparoscopic surgery training: A 1-year analysis of skill retention for surgery residents. Surgery. 2009;146:387–393. doi: 10.1016/j.surg.2009.05.009. [DOI] [PubMed] [Google Scholar]
- 16.Mashaud LB, Castellvi AO, Hollett LA, Hogg DC, Tesfay ST, Scott DJ. Two-year skill retention and certification exam performance after fundamentals of laparoscopic skills training and proficiency maintenance. Surgery. 2010;148:194–201. doi: 10.1016/j.surg.2010.05.012. [DOI] [PubMed] [Google Scholar]
- 17.Stefanidis D, Scerbo MW, Montero PN, Acker CE, Smith WD. Simulator training to automaticity leads to improved skill transfer compared with traditional proficiency-based training: a randomized controlled trial. Ann Surg. 2012;255:30–37. doi: 10.1097/SLA.0b013e318220ef31. [DOI] [PubMed] [Google Scholar]
- 18.Schmidt R, Lee TD. Motor control and learning: A behavioral emphasis. 4th Ed. Human Kinetics; Champaign, IL: 2005. [Google Scholar]
- 19.Schneider W, Fisk AD. Degree of consistent training: Improvements in search performance and automatic process development. Perception & Psychophysics. 1982;31:160–168. doi: 10.3758/bf03206216. [DOI] [PubMed] [Google Scholar]
- 20.Stefanidis D, Acker C, Heniford BT. Proficiency-based laparoscopic simulator training leads to improved operating room skill that is resistant to decay. Surg Innov. 2008;15:69–73. doi: 10.1177/1553350608316683. [DOI] [PubMed] [Google Scholar]