Abstract
Background
The purpose of this study was to evaluate face and content validity of a low-cost, low-technology, non-biologic endoscopic sinus surgery (ESS) task trainer and knowledge-based curriculum followed by construct validation of the task trainer.
Methods
A sinus surgery task trainer and curriculum were developed. Attending otolaryngologists were surveyed regarding the utility of the task trainer and curriculum. A cross-sectional construct validation study was performed including medical students, residents, and attending otolaryngologists. Nine tasks were performed on the task trainer and graded using a global rating scale (GRS). Predictors of task trainer performance were evaluated, and spatial performance was measured. Regression analyses were performed to assess main associations.
Results
100% of attendings (n=7) agreed that the task trainer accelerates the learning curve and incorporates essential ESS techniques and that the knowledge-based curriculum describes anatomical and technical knowledge germane to ESS. Twelve medical students, nine otolaryngology residents, and five attending otolaryngologists completed task trainer testing. GRS score varied by level of training after adjusting for potential confounders (slope=2.63; p=0.001). There was evidence of an association between a history of video gaming (slope=1.33; p=0.077), sports experience (slope=2.08; p=0.033), and sinus surgery simulation (slope=2.72; p=0.023) with GRS score, although not statistically significant.
Conclusion
This study demonstrated validity of a knowledge-based sinus surgery curriculum and low-cost, sinus task trainer supporting use in early training. Participants with prior video gaming, sports participation, and sinus surgery simulation experience perform better at baseline on the task trainer.
Keywords: simulation training, residency training in rhinology, sinus surgery, endoscopy, nose models
Introduction
Surgical training is in the midst of a new era underscored by promotion of the value of care and patient safety. Simulation provides a strategic method for increasing resident technical proficiency outside of the operating room, paramount to increasing patient safety. Resident participation in the operating room has been associated with longer operative times and potentially increased complications.1,2 Moreover, data suggest that surgical skill is associated with patient outcomes.3 Low-cost, low-technology simulators boost technical performance and patient safety by providing widely accessible methods that do not depend exclusively on technology and can be used repeatedly.
Simulation increases cognitive skills4 and provides an opportunity for surgical practice. Endoscopic sinus surgery requires unique skills including visualization of three-dimensional movements on a planar computer monitor coupled with bimanual dexterity. A low-technology task trainer developed by Malekzadeh et al. emphasizes movements and tactile feedback with specific sinus surgical procedures including maxillary antrostomy using an egg to mimic the thin bone of the uncinate.5,6 Others have developed models allowing for familiarization with endoscopic visualization7 and practicing destructive maneuvers, although these are limited by re-usability and the scope of training.8
In contrast, cadavers offer a more realistic conduit for teaching specific surgical skills once fundamental instrument handling and endoscopy skills have been acquired. Cadavers are limited by costs, availability, and ethical considerations. A possible solution for high-technology, reusable simulation includes virtual reality trainers.9,10 However, these are limited by cost and maintenance requirements with few institutions having full-time access to virtual reality trainers. A low-cost, low-technology, non-biologic task trainer would offer a widely-accessible, easily reusable solution for acquiring essential endoscopic and instrument handling skills as well as practicing deconstructed sinus surgery movements.
The primary aims of this study were to evaluate the face, content, and construct validity of a low-cost, low-technology, non-biologic sinus surgery task trainer. Secondarily, we evaluated face and content validity of a knowledge-based sinus surgery curriculum. We hypothesized that performance on the task trainer increases with one's level of training. We also sought to evaluate the relationship between potential predictors of endoscopic skill and task trainer performance hypothesizing that a prior history of activities requiring fine motor / spatial skills would be positively associated with task trainer performance.
Methods
Subjects
To evaluate face and content validity, attending otolaryngologists at the University of Washington were surveyed. To evaluate construct validity, subjects enrolled included medical students, R2–R3 level residents prior to their dedicated Rhinology rotation, R3–R5 level residents who had completed their dedicated Rhinology rotation at the time of the study, and attending otolaryngologists. The University of Washington Institutional Review Board approved this study prior to the initiation of data collection (HSD #50376).
Task Trainer Development
Tasks were performed on a model costing on average under $15 USD made from silicone rubber (i.e., Dragon Skin®, Smooth-On, Inc., Macungie, PA). A video illustrating construction of the task trainer is located in the Supplemental Materials (S1). Modeling dough (e.g., Play-Doh®) is formed into a rectangular shape and impressions are made along both sides using a pen creating the reliefs of the middle and inferior turbinates (Figure 1A). The dough mold is placed in a storage container (e.g., Tupperware), and two spherical pieces of modeling dough are suspended adjacent to the main piece of modeling dough to form maxillary sinuses (Figure 1B). Silicone rubber is poured around the relief. Once polymerized, the modeling dough is removed leaving a sinus cavity model. A septum is created using any readily available material (e.g., plastic lid or cardboard stock). The model is stabilized in a Styrofoam block or wig model, and a manikin face (Laerdal Medical Limited, Orpington, Kent, UK) is placed over the model and draped with surgical towels (Figure 1C and 1D). Short push pins are placed in the nasopharyngeal region, middle meatus, maxillary sinus, and in the uncinate region facing posteriorly in preparation for the task set.
FIGURE 1. Creating and setting up sinus surgery task trainer.
(A) A cylindrical object is used to create inferior and middle turbinate impressions in modeling dough. (B) Modeling dough is placed in a small container and dough spheres are suspended adjacent to task trainer. Silicone rubber is poured around the model. (C) The resulting model is placed into a Styrofoam block or wig model. (D) A manikin face cover is placed over the task trainer, and surgical towels are draped over the manikin face to simulate the operative experience.
Knowledge-Based Curriculum Development
The sinus surgery training curriculum was developed following a consensus-driven simulation-based surgical curriculum framework.11 The knowledge-based curriculum includes a written document describing uncinectomy, maxillary antrostomy, ethmoidectomy, and sphenoidotomy similar to that described by Wormald (S2).12 Pictures and descriptions of key instruments, common errors and error mitigation, and complications in sinus surgery are discussed in the document. An instructional video of a maxillary antrostomy, ethmoidectomy, and sphenoidotomy performed by a rhinologist (G.E.D.) was created (S3). Pre- and post-tests evaluate the trainee's sinus surgery knowledge (S4).
Face and Content Validation
Face validity is defined herein as the general appropriateness of an assessment with respect to its intended purpose based on expert review.13 Content validity is defined as a detailed examination of the contents of an assessment with respect to its intended purpose.13 A 14-item survey evaluating the task trainer and a 10-item survey evaluating the knowledge-based curriculum were sent to seven attending otolaryngologists at the University of Washington (S5). Surveys consisted of statements accompanied by answer choices using a 5-point Likert-type rating scale (strongly agree to strongly disagree).
Construct Validation: Population Characteristics
Construct validity is defined as the degree to which an assessment identifies the traits or abilities it was designed to measure.13 Subjects were asked to complete a survey assessing baseline characteristics including gender, level of training, handedness, previous sinus surgery experience, and previous sinus surgery simulation experience. The survey also assessed past and current participation in activities that have been previously associated with endoscopic surgery abilities in the otolaryngologic and general surgery literature including participation in sports, dancing, sewing, musical instruments, video games, and use of tools.14,15
Construct Validation: Task Trainer Protocol
All subjects were asked to perform a series of nine tasks developed by a focus group of attending otolaryngologists and otolaryngology residents. Briefly, a series of increasingly complex tasks deconstructing key actions in endoscopic sinus surgery were chosen to train specific actions. These include two endoscopic visualization tasks using a 0° nasal endoscope, a straight suction task, a sinus injection task, and nasopharyngeal pin removal task using Takahashi forceps (Fig. 2A). To simulate an uncinectomy, we developed a task removing a pin from the simulated uncinate region with a backbiter (Fig. 2B). A maxillary sinus suctioning task is performed with a curved olive tip suction. A 30° endoscope is used to remove a vertically oriented pin in the middle meatus (Fig. 2C) and a pin inside the maxillary sinus using 45° forceps (Fig. 2D).
FIGURE 2. Pin placement for sinus surgery task trainer.
(A) A pin is placed in the posterior region of the task trainer simulating translational movements. (B) A posterior facing pin is placed in the uncinate region simulating movements during uncinectomy. (C) A vertically-oriented pin is placed in the middle meatus region simulating vertically-oriented polyp removal from the middle meatus. (D) A pin is placed in the maxillary sinus simulating maxillary sinus debridement.
Construct Validation: Mental Rotation Test
In order to evaluate the convergent validity of the task trainer assessment, subjects were asked to complete a Mental Rotation Test16 which is a timed assessment of spatial relations consisting of a series of 20 reference cubes each with two matching rotated versions and two non-matching rotated mirror images. Convergent validity is considered a subtype of construct validity defined as agreement among measures that theoretically should be related.17,18
Construct Validation: Video Grading Protocol
Endoscopic video was recorded and de-identified for independent grading by two reviewers (G.E.D. and R.A.H.). A previously validated endoscopic sinus surgery global rating scale (GRS) was used for evaluating video performance.19 Five domains from the GRS were used including Use of Endoscopes, Instrument Handling, Respect for Tissue, Time and Motion, and Overall Surgical Performance. Domains were scored using a 5-point Likert-type rating scale, and mean and cumulative scores were calculated for each subject.
Statistical Analysis
Two-tailed t-tests were used to evaluate face and content validity survey data testing the hypothesis that responses were consistent with agreement or disagreement. Bonferroni corrected α-levels of 0.0036 and 0.005 were used for hypothesis testing for the task trainer and curriculum surveys, respectively. For the construct validity assessment, univariate linear regression was performed to assess associations of interest. Multivariate linear regression was performed to adjust for confounders considered a priori including previous sports, sinus surgery simulation, musical instrument playing, and video gaming experience. Strength of the association is represented by the slope of the regression function (β1). An α-level of 0.05 was used for inferential hypothesis testing with accompanying 95% confidence intervals (CI) presented. To account for multiple comparisons in our secondary analyses evaluating the association between predictors of endoscopic skill and task trainer performance, a Bonferroni adjusted α-level of 0.01 (i.e., 0.05 / 5 comparisons) was used to evaluate statistical significance with accompanying 99% CIs reflecting the adjusted α-level. Partial F-tests were used to evaluate the null hypothesis of no association (i.e., slope = 0) in the regression models.
Our sample size calculation was based on data from Kaplan et al. 2014.20 Seven participants per group (i.e., experienced vs novice) were required to provide 80% power to detect a 10 point difference in mean GRS score between novices and experienced surgeons at a two-sided α = 0.05. Statistical analyses were performed using Stata 14 (StataCorp. 2015. Stata Statistical Software: Release 14. College Station, TX: StataCorp LP) statistical programming software.
Results
Face and Content Validation of the Sinus Surgery Task Trainer Assessment and Knowledge-Based Curriculum
Figure 3 presents response data from seven attending otolaryngologists who completed the task trainer face and content validation survey. Because of the need to adjust for multiple comparisons, an adjusted significance level of 0.0036 was used to test hypotheses for the task trainer face and content validation. All attendings agreed (38.1% agreed, 61.9% strongly agreed) that the task trainer demonstrates face validity for accelerating the learning curve and increasing competency while incorporating essential sinus surgery skills (p < 0.005). There was general consensus with over 70% agreement or strong agreement that the task trainer hones instrument handling skills, injection technique, maneuvering of curved instruments, endoscope orientation, maxillary sinus instrumentation, translational movements, and rotational movements (p < 0.005). There were insufficient data to conclude that the mean response was different than neutral with respect to the vertical pin removal or the posterior bead extraction task (p = 0.015 and p = 0.084, respectively).
FIGURE 3. Task trainer face and content validation survey responses.
A total of seven attending otolaryngologists responded to the survey. Survey question categories are presented on the Y-axis. Responses graded on a 5-point Likert-type scale ranging from Strongly Disagree to Strongly Agree are presented on the X-axis. Max. Sinus Task, maxillary sinus task.
Figure 4 illustrates response data from six attendings who completed the knowledge-based curriculum survey. An adjusted significance level of 0.005 was used to test hypotheses for the knowledge-based curriculum face and content validation. All attendings agreed (17% agreed, 83% strongly agreed) that the pre- and post-tests are valuable assessment tools (p < 0.005). All attendings also agreed (31% agreed, 69% strongly agreed) that the knowledge-based curriculum is a valuable learning tool that appropriately illustrates endoscopic sinus surgery essentials including steps, instruments, common errors, and potential complications of sinus surgery (p < 0.005). There were insufficient data to conclude that the mean response was different than neutral when probed about teaching alternative techniques or additional pitfalls (i.e., errors) (p = 0.041 and p = 0.36, respectively).
FIGURE 4. Cognitive curriculum face and content validation survey responses.
A total of six attending otolaryngologists responded to the survey. Survey question categories are presented on the Y-axis. Responses graded on a 5-point Likert-type scale ranging from Strongly Disagree to Strongly Agree are presented on the X-axis. Addtl. Common Errors, additional common errors. FESS, functional endoscopic sinus surgery.
Population Characteristics of the Construct Validation Assessment Cohort
We enrolled 12 medical students, four R2–R3 level residents prior to their dedicated Rhinology rotation, five R3–R5 level residents who had completed their dedicated Rhinology rotation at the time of the study, and five attending otolaryngologists. Participant characteristics are shown in Table 1. The sample consisted of 10 females and 16 males. None of the medical students had previous sinus surgery experience, although two had previous sinus surgery simulation experience. Half of the participants had no prior experience with any sinus surgery simulation.
Table 1.
Baseline characteristics of medical students, residents and attendings.
| Characteristic | N = 26 No. (%) |
|---|---|
| Sex | |
| Female | 10 (38) |
| Male | 16 (62) |
| Level of training | |
| Medical student / Intern | 12 (46) |
| R2/R2 pre-Rhinology rotation | 4 (16) |
| R3/R5 post-Rhinology rotation | 5 (19) |
| Attending Otolaryngologist | 5 (19) |
| Handednessa | |
| Left | 3 (13) |
| Right | 20 (83) |
| Ambidextrous | 1 (4) |
| Previous experience with sinus surgerya | |
| No experience | 12 (50) |
| <10 cases | 2 (8) |
| 11–20 cases | 1 (4) |
| >20 cases | 9 (38) |
| Previous experience with sinus surgerya simulation |
|
| No experience | 11 (46) |
| 1–5 times | 9 (38) |
| 6–10 times | 1 (4) |
| >10 times | 3 (12) |
There were missing survey data for 2 attendings.
Construct Validation of the Sinus Surgery Task Trainer
Table 2 describes the frequency of participants within each categorical level of training that had any previous and/or current participation in activities requiring hand-eye coordination and fine motor skills. Past experience was noted in all categories of the activities queried, especially a history of sports participation, musical instrument and video/computer game experience, and use of tools. However, current participation levels were substantially lower compared to past participation.
Table 2.
Count and percent of total participants within a categorical level of training with any past and/or current participation in various activities requiring hand-eye coordination.
| Medical Student /Intern n = 12 |
R2/R2 pre sinus-rotation n = 4 |
R3/R5 post- sinus rotation n = 5 |
Attending n = 5 |
||
|---|---|---|---|---|---|
| No. (%) | No. (%) | No. (%) | No. (%) | ||
| Past | |||||
|
aSports participation |
10 (83) | 3 (75) | 5 (100) | b3 (100) | |
| Ballet/dancing | 2 (17) | 1 (25) | 3 (60) | 2 (67) | |
| Sewing/crochet/ weaving |
4 (33) | 1 (25) | 4 (80) | 1 (33) | |
| Video games | 10 (83) | 4 (100) | 4 (80) | 2 (67) | |
| Computer games |
9 (75) | 4 (100) | 3 (60) | 2 (67) | |
| Musical instrument |
11 (92) | 3 (75) | 5 (100) | 3 (100) | |
| Use of tools | 11 (92) | 4 (100) | 2 (40) | 3 (100) | |
| Current | |||||
| Sports participation |
5 (42) | 3 (75) | 2 (40) | 3 (100) | |
| Ballet/dancing | 2 (17) | 0 (0) | 2 (40) | 1 (33) | |
| Sewing/crochet/ weaving |
1 (9) | 0 (0) | 2 (40) | 0 (0) | |
| Video games | 3 (25) | 1 (25) | 1 (20) | 2 (67) | |
| Computer games |
3 (25) | 1 (25) | 0 (0) | 2 (67) | |
| Musical instrument |
5 (42) | 1 (25) | 3 (60) | 2 (67) | |
| Use of tools | 8 (67) | 1 (25) | 2 (40) | 3 (100) | |
Categories of experience with past and current activities include no participation, < 1 hr per week, 1 – 5 hours per week, >5 – 9 hours per week and > 9 hours per week.
Missing survey data for two attendings. Percentages reflect proportion of total respondents.
The relationship between categorical level of training and task trainer performance is illustrated in Figure 5. The unadjusted univariate regression model illustrated evidence of an association between categorical level of training and cumulative GRS score (p < 0.001). For each additional categorical level of training, the estimated cumulative GRS score was 3.00 points higher (95% CI: 1.88, 4.13). When adjusting for potential confounding by past video gaming, sports experience, instrument playing, and sinus surgery simulation, GRS score was 2.63 points higher (95% CI: 1.31, 3.94; p = 0.001) with each additional level of training.
FIGURE 5. Task trainer performance vs level of training.
Global rating scale scores are presented as a function of level of training (r2 = 0.56). MS, medical student. R1, 1st year resident. R2/R3, 2nd – 3rd year residents prior to dedicated Rhinology rotation. R3/R5, 3rd–5th year residents post dedicated Rhinology rotation. Attending, attending otolaryngologist. GRS, Global Rating Scale.
In order to assess consistency in our task trainer performance ratings, interrater agreement was measured using a kappa-statistic for two unique raters. Stratifying GRS scores by competence as in Lin et al.19 where competence is defined as a mean score of greater than or equal to three, we observed a kappa of 0.84. This suggested excellent agreement consistent with Lin et al. where interrater agreement was noted to be 77.36%.19
Table 3 illustrates the relationship between potential spatial predictors of task trainer performance. An adjusted significance level of 0.01 was used to determine statistical significance accounting for five comparisons. 99% confidence intervals (100 – 0.01 = 99%) are presented to reflect the adjusted significance level. Controlling for level of training, GRS score was 1.33 points higher (99% CI: −0.69, 3.35) for each additional level of exposure to past video gaming (p = 0.077). We did not observe an association between past musical instrument playing and GRS score (p = 0.255). GRS score was 1.25 points higher (99% CI: −1.76, 4.25) for each subsequent level of exposure to musical instruments. With increasing levels of prior exposure to sports participation, GRS score increased by 2.08 points (99% CI: −0.50, 4.65; p = 0.033). When adjusting for level of training, GRS score increased by 2.72 points (99% CI: −0.43, 5.87) for each additional level of previous simulation experience (p = 0.023).
Table 3.
Association between potential predictors of endoscopic skill and task trainer performance as measured by GRS score.
| Predictor | Univariate Model β1 (99% CI)a |
pb | Multivariate Model β1 (99% CI)c |
pb |
|---|---|---|---|---|
| Past video gaming | 0.90 (−1.95, 3.74) | 0.38 | 1.33 (−0.69, 3.34) | 0.077 |
| Past musical instrument |
1.25 (−1.76, 4.25) | 0.26 | ||
| Past sports participation |
2.08 (−0.50, 4.65) | 0.033 | ||
| Previous sinus surgery simulation |
4.35 (1.61, 7.10) | < 0.001 | 2.72 (−0.43, 5.87) | 0.023 |
| Spatial performanced |
0.10 (−1.06, 1.26) | 0.80 |
Univariate regression model evaluating association between predictor and GRS score; β1, slope of regression model
Bonferroni adjusted α (significance level) = 0.01
Multivariate regression model adjusting for level of training
Spatial performance measured by Mental Rotation Test
We evaluated the association between Mental Rotation Test scores and task trainer performance. There was insufficient evidence in our data to support the presence of an association between Mental Rotation Test and GRS scores (p = 0.803). The GRS score was 0.10 times lower for each additional unit increase in MRT score (99% CI: −1.18, 1.29).
Discussion
The ultimate goal of deliberate surgical practice using surgical simulators is to increase patient safety and accelerate the learning curve before a novice surgeon begins operating on live patients. The primary aims of this study were to evaluate the face, content, and construct validity of a low-cost, low-technology, non-biologic sinus surgery task trainer and knowledge-based curriculum. Secondarily, we aimed to assess potential predictors of endoscopic surgical skill. Our principal findings were as follows: 1) 100% of attendings agreed that the chosen tasks accelerate the learning curve and increase competency through teaching skills germane to sinus surgery, 2) 100% of attendings agreed that the knowledge-based curriculum teaches appropriate endoscopic sinus surgery techniques including key instruments, common errors, and potential complications, 3) a relationship between level of training and task trainer performance was noted supporting construct validity, and 4) the data suggest an association between task trainer performance and spatial skills predictors including past video game, sports, and sinus surgery simulation experience. There was insufficient evidence to support a relationship between spatial performance on the Mental Rotation Test and task trainer performance.
Evaluation of the face and content validity of the task trainer suggests that the task trainer is valuable for teaching basic endoscopic sinus surgery techniques and providing a platform for acquiring fundamental endoscope and instrument handling skills. Malekzadeh et al. found consensus regarding the utility of a biologic sinus surgery task trainer emphasizing the development of tactile feedback, camera navigation skills, hand-eye coordination, and nasal endoscopic skills.6 Our findings were consistent, although there was a neutral response to the utility of removing a vertically-oriented object from the middle meatus. Future iterations could include pin placement along the “skull base” of the task trainer to expand the breadth of the task set.
Training both cognitive and technical skills in combination has a greater impact on surgical performance than either alone.4 There is currently a paucity of data in the literature evaluating the efficacy of reproducible, low-cost simulation training incorporating knowledge-based curricula in sinus surgery education. The ES3 virtual reality sinus surgery simulator validated by Fried et al. included a cognitive component that was coupled with surgical practice.9 However, virtual reality systems have not reached mainstream use due to maintenance requirements and initial start-up costs. This study presents an initial evaluation of a sinus surgery simulation-based curriculum comprised of a reproducible, low-cost, non-biologic task trainer coupled with a knowledge-based curriculum.
Prior work lends evidence that specific types of simulators may be individualized to trainees based on their previous experience with sinus surgery and a priori fine motor skills. For example, Malekzadeh et al. developed a biologic sinus surgery task trainer allowing for increased tactile feedback that included an egg puncture task deconstructing maxillary antrostomy movements.5 This type of trainer may be beneficial among individuals with adequate fine motor skills who need to develop their sense of tactile feedback and endoscopic abilities for sinus surgery. Similarly, our task trainer may be used to develop a sense of tactile feedback with deconstructed sinus surgery tasks in a non-biologic model. The model can be made a single time, used repeatedly, and requires minimal setup time facilitating use given time constraints of residents and faculty, an advantage over high-technology and biologic sinus surgery simulators.
For trainees seeking additional development of fine motor skills and experience with endoscope handling, low-cost task trainers are a reliable, easily-transportable tool to support this goal. Leung et al. created a non-biologic sinus surgery task trainer using readily available components focusing on increasing hand-eye coordination and endoscope handling skills.8 The task trainer presented here also provides a platform for increasing hand-eye coordination and endoscope and instrument handling skills via deliberate practice of deconstructed sinus surgery movements.
High-technology physical models provide excellent anatomic realism, haptic feedback, and allow use of actual sinus surgical instruments, although similar to cadaver models, they cannot be reused. Ossowski et al. utilized a physical trainer for honing nasal endoscopy skills, although tissue debridement would require replacement of the entire model analogous to the use of cadaver models.7 The Sinus Model Otorhino-Neuro Trainer provides an anatomically-realistic environment with interchangeable pathology that must be replaced with each use.21 These may be better suited for competency assessment and practice of advanced techniques. High initial costs and ongoing costs associated with replacement parts limit routine implementation especially in the early training period.21
Individualized simulation plans can be provided early in training to optimize training and patient safety. Findings from this study support the association between activities requiring fine motor and hand-eye coordination and endoscopic endonasal sinus surgical abilities. A trainee with less a priori fine motor skills may benefit more from task trainer simulations with increased practice maneuvering the endoscope and touching targets; whereas, a trainee with more a priori fine motor skills may benefit from early practice of specific maneuvers associated with sinus surgery prior to their operating room experience. Future work aims to characterize trainee-specific surgical simulations that accelerate the learning curve and optimize patient safety.
We did not find evidence to support an association between Mental Rotation Test score and task trainer performance. Other studies have noted an association between spatial performance and endoscopic skill in both endoscopic sinus and laparoscopic surgery. Stransky et al. noted that practice with the Mental Rotation Test correlated with increased performance on a laparoscopic box trainer.22 One possible reason for our discordant results is that the Mental Rotation Test is biased towards participants with a background in the physical sciences accounting for generally higher scores on the assessment.23 The task trainer incorporates a unique set of skills requiring a priori anatomic and technical knowledge which may be associated with only a small amount of the variance in Mental Rotation Test performance. Additionally, novices may have more experience with video games and spatial performance skills at baseline than in the past.
Our face and content validation surveys are likely biased as a result of a single institution experience. The survey responses did allow us to make iterative improvements to the task trainer and curriculum and illustrates the adaptability of the present simulation-based curriculum. As the task trainer primarily teaches intranasal and maxillary sinus maneuvers, more advanced tasks including ethmoidectomy and sphenoidotomy are not included limiting the utility of the task trainer to early trainees. This is in line with our main goal of helping residents overcome the initial learning curve of endoscopic sinus work. Another limitation is that this task trainer may not be beneficial for all trainees. We noted less of a difference in GRS score when comparing task trainer performance between residents and attendings. This may be a function of discriminating skills specific to the task trainer rather than general endoscopic sinus surgery skills. Two of the first year otolaryngology residents performed above the competence threshold (GRS cumulative score of 15). One had prior sinus surgery simulation experience on a previous iteration of the present task trainer while the other reported no prior simulation experience. This could also suggest that skill retention from a low-cost, low-technology task trainer may be long lasting, although this cannot be concluded as this study was not designed to test this association. Once a trainee has acquired sufficient technical skills to safely operate, anatomic knowledge and cognitive skills associated with endoscopic sinus surgery likely play a larger role in surgical technical success.
Conclusion
The present study provides evidence for face, content, and construct validity of an easily accessible, low-cost, low-technology, non-biologic sinus surgery simulator and sinus surgical curriculum. We demonstrated that level of training, prior video game, sports, and sinus surgery simulation experience are predictive of endoscopic performance on the task trainer. Bolstering sinus surgery specific technical and cognitive skills at an early stage of training is paramount to increasing patient safety and accelerating the learning curve for novice surgeons. Future work will evaluate predictive validity of our task trainer and determine whether deliberate practice improves operative performance.
Supplementary Material
Written Curriculum: This module contains a concise description of the key steps to basic functional endoscopic sinus surgery including uncinectomy, maxillary antrostomy, ethmoidectomy, and sphenoidotomy. A list with pictures and brief descriptions of key instruments used in sinus surgery is included. A discussion of common errors among novice surgeons and prevention strategies is also included along with a list of complications that can occur during sinus surgery and how to avoid them.
Sinus Surgery Task Trainer How-To Video (MPEG-4; 00:01:24): This video demonstrates the materials and techniques for making the sinus surgery task trainer.
Instructive Sinus Surgery Video (MPEG-4; 00:10:34): This video illustrates a rhinologist (G.E.D.) performing an uncinectomy, maxillary antrostomy, ethmoidectomy and sphenoidotomy.
Knowledge-based pre- and post-tests: These assessments evaluate the learners understanding of paranasal sinus anatomy, appropriate instrument handling technique, adverse events and their management, and surgical technique.
Task Trainer and Curriculum Surveys: These surveys were utilized to evaluate expert opinion on the utility of the deconstructed tasks chosen for the task trainer and the utility of the cognitive curriculum including its written component and knowledge-based pre- and post-tests.
Acknowledgments
Support for this research included funding from NIH grant T32DC000018. The authors would like to acknowledge the support of Dr. Ian Humphreys for his advice on design of the tasks. We also acknowledge the University of Washington Department of Biostatistics for their assistance and advice with study and analysis planning. We acknowledge the assistance of our clinic staff and the staff of the University of Washington Institute for Simulation in Healthcare. Lastly, we thank the University of Washington Otolaryngology Outcomes Research Group for feedback pertaining to study design and interpretation of the results.
Footnotes
Financial disclosures: No financial disclosures.
Conflict of Interest: No conflicts to declare.
The findings in this manuscript were presented at the American Rhinologic Society Annual Meeting in San Diego, CA which took place September 16th–17th, 2016.
References
- 1.Krell RW, Birkmeyer NJ, Reames BN, et al. Effects of resident involvement on complication rates after laparoscopic gastric bypass. J Am Coll Surg. 2014;218(2):253–260. doi: 10.1016/j.jamcollsurg.2013.10.014. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Dedhia RC, Lord CA, Pinheiro-Neto CD, et al. Endoscopic endonasal pituitary surgery: impact of surgical education on operation length and patient morbidity. J Neurol Surg B Skull Base. 2012;73(6):405–409. doi: 10.1055/s-0032-1329620. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Birkmeyer JD, Finks JF, O'Reilly A, et al. Surgical skill and complication rates after bariatric surgery. N Engl J. Med. 2013;369(15):1434–1442. doi: 10.1056/NEJMsa1300625. [DOI] [PubMed] [Google Scholar]
- 4.Palter VN, Grantcharov TP. Development and validation of a comprehensive curriculum to teach an advanced minimally invasive procedure: a randomized controlled trial. Ann Surg. 2012;256(1):25–32. doi: 10.1097/SLA.0b013e318258f5aa. [DOI] [PubMed] [Google Scholar]
- 5.Malekzadeh S, Pfisterer MJ, Wilson B, Na H, Steehler MK. A novel low-cost sinus surgery task trainer. Otolaryngol Head Neck Surg. 2011;145(4):530–533. doi: 10.1177/0194599811413373. [DOI] [PubMed] [Google Scholar]
- 6.Steehler MK, Pfisterer MJ, Na H, Hesham HN, Pehlivanova M, Malekzadeh S. Face, content, and construct validity of a low-cost sinus surgery task trainer. Otolaryngol Head Neck Surg. 2012;146(3):504–509. doi: 10.1177/0194599811430187. [DOI] [PubMed] [Google Scholar]
- 7.Ossowski KL, Rhee DC, Rubinstein EN, Ferguson BJ. Efficacy of sinonasal simulator in teaching endoscopic nasal skills. Laryngoscope. 2008;118(8):1482–1485. doi: 10.1097/MLG.0b013e3181772d4f. [DOI] [PubMed] [Google Scholar]
- 8.Leung RM, Leung J, Vescan A, Dubrowski A, Witterick I. Construct validation of a low-fidelity endoscopic sinus surgery simulator. Am J Rhinol. 2008;22(6):642–648. doi: 10.2500/ajr.2008.22.3245. [DOI] [PubMed] [Google Scholar]
- 9.Fried MP, Sadoughi B, Gibber MJ, et al. From virtual reality to the operating room: the endoscopic sinus surgery simulator experiment. Otolaryngol Head Neck Surg. 2010;142(2):202–207. doi: 10.1016/j.otohns.2009.11.023. [DOI] [PubMed] [Google Scholar]
- 10.Varshney R, Frenkiel S, Nguyen LH, et al. The McGill simulator for endoscopic sinus surgery (MSESS): a validation study. J Otolaryngol Head Neck Surg. 2014;43:40. doi: 10.1186/s40463-014-0040-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Zevin B, Levy JS, Satava RM, Grantcharov TP. A consensus-based framework for design, validation, and implementation of simulation-based training curricula in surgery. J Am Coll Surg. 2012;215(4):580–586. e583. doi: 10.1016/j.jamcollsurg.2012.05.035. [DOI] [PubMed] [Google Scholar]
- 12.Wormald PJ. Endoscopic Sinus Surgery: Anatomy, Three-Dimensional Reconstruction, and Surgical Technique. New York: Thieme; 2008. [Google Scholar]
- 13.Gallagher AG, Ritter EM, Satava RM. Fundamental principles of validation, and reliability: rigorous science for the assessment of surgical education and training. Surg Endosc. 2003;17(10):1525–1529. doi: 10.1007/s00464-003-0035-4. [DOI] [PubMed] [Google Scholar]
- 14.Lin D, Pena G, Field J, et al. What are the demographic predictors in laparoscopic simulator performance? ANZ J Surg. 2015 doi: 10.1111/ans.12992. [DOI] [PubMed] [Google Scholar]
- 15.Chole RA, Ogden MA. Predictors of future success in otolaryngology residency applicants. Arch Otolaryngol Head Neck Surg. 2012;138(8):707–712. doi: 10.1001/archoto.2012.1374. [DOI] [PubMed] [Google Scholar]
- 16.Vandenberg SG, Kuse AR. Mental rotations, a group test of three-dimensional spatial visualization. Percept Mot Skills. 1978;47(2):599–604. doi: 10.2466/pms.1978.47.2.599. [DOI] [PubMed] [Google Scholar]
- 17.Trochim W. The Research Methods Knowledge Base. 2nd. Cincinnati, OH: Atomic Dog Publishing; 2000. [Google Scholar]
- 18.Fried MP, Sadoughi B, Weghorst SJ, et al. Construct validity of the endoscopic sinus surgery simulator: II. Assessment of discriminant validity and expert benchmarking. Arch Otolaryngol Head Neck Surg. 2007;133(4):350–357. doi: 10.1001/archotol.133.4.350. [DOI] [PubMed] [Google Scholar]
- 19.Lin SY, Laeeq K, Ishii M, et al. Development and pilot-testing of a feasible, reliable, and valid operative competency assessment tool for endoscopic sinus surgery. Am J Rhinol Allergy. 2009;23(3):354–359. doi: 10.2500/ajra.2009.23.3275. [DOI] [PubMed] [Google Scholar]
- 20.Kaplan DJ, Vaz-Guimaraes F, Fernandez-Miranda JC, Snyderman CH. Validation of a chicken wing training model for endoscopic microsurgical dissection. Laryngoscope. 2014 doi: 10.1002/lary.24977. [DOI] [PubMed] [Google Scholar]
- 21.Nogueira JF, Stamm AC, Lyra M, Balieiro FO, Leao FS. Building a real endoscopic sinus and skull-base surgery simulator. Otolaryngol Head Neck Surg. 2008;139(5):727–728. doi: 10.1016/j.otohns.2008.07.017. [DOI] [PubMed] [Google Scholar]
- 22.Stransky D, Wilcox LM, Dubrowski A. Mental rotation: cross-task training and generalization. J Exp Psychol Appl. 2010;16(4):349–360. doi: 10.1037/a0021702. [DOI] [PubMed] [Google Scholar]
- 23.Peters M, Laeng B, Latham K, Jackson M, Zaiyouna R, Richardson C. A redrawn Vandenberg and Kuse mental rotations test: different versions and factors that affect performance. Brain Cogn. 1995;28(1):39–58. doi: 10.1006/brcg.1995.1032. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Written Curriculum: This module contains a concise description of the key steps to basic functional endoscopic sinus surgery including uncinectomy, maxillary antrostomy, ethmoidectomy, and sphenoidotomy. A list with pictures and brief descriptions of key instruments used in sinus surgery is included. A discussion of common errors among novice surgeons and prevention strategies is also included along with a list of complications that can occur during sinus surgery and how to avoid them.
Sinus Surgery Task Trainer How-To Video (MPEG-4; 00:01:24): This video demonstrates the materials and techniques for making the sinus surgery task trainer.
Instructive Sinus Surgery Video (MPEG-4; 00:10:34): This video illustrates a rhinologist (G.E.D.) performing an uncinectomy, maxillary antrostomy, ethmoidectomy and sphenoidotomy.
Knowledge-based pre- and post-tests: These assessments evaluate the learners understanding of paranasal sinus anatomy, appropriate instrument handling technique, adverse events and their management, and surgical technique.
Task Trainer and Curriculum Surveys: These surveys were utilized to evaluate expert opinion on the utility of the deconstructed tasks chosen for the task trainer and the utility of the cognitive curriculum including its written component and knowledge-based pre- and post-tests.