Validation of an Educational Simulation Model for Vaginal Hysterectomy Training: A Pilot Study

Abstract

Introduction and Hypothesis:

The Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC) was designed as a realistic vaginal surgery simulation model. Our aim is to describe this model and present pilot data on validity and reliability of the model as an assessment tool of vaginal hysterectomy skills.

Methods:

We video recorded 10 obstetrics and gynecology residents (novice group) and 10 practicing gynecologists (expert group) performing vaginal hysterectomy using the Miya model ™ (Miyazaki Enterprises, Winston-Salem, NC). Blood loss and time taken to complete the procedure were documented. Participants evaluated the model using a postsimulation survey. In addition, two experienced gynecologic surgeons independently evaluated video recordings of each participant’s performance using two previously validated global rating scales (Reznick’s Objective Structured Assessment of Technical Skill (OSATS) and Vaginal Surgical Skills Index (VSSI)).

Results:

Most participants (80% of novice and 100% of expert group) rated the model as “effective” or “highly effective” for vaginal hysterectomy training and assessment. Median time to procedure completion was significantly higher in the novice group compared to the experts whereas median estimated blood loss was not different between the two groups. No significant differences were observed in the composite median OSATS or VSSI scores between novice and experts. The inter-rater reliability indices for subscales and composite scores of the OSATS and VSSI were high and ranged from 0.79 to 0.90 and 0.77 to 0.93, respectively.

Conclusion:

With further study, the Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC) may be a useful tool for teaching and assessing vaginal surgical skills.

Brief Summary

This pilot study reports the evidence for face, content, and construct validity and reliability of a high-fidelity simulation model for vaginal hysterectomy training.

Introduction

Vaginal hysterectomy is the preferred route of uterine removal whenever feasible because of demonstrated reductions in morbidity and cost when compared with abdominal, laparoscopic, and robotic routes of hysterectomy [1–3]. However, data from the Nationwide Inpatient Sample revealed that the percentage of hysterectomies performed vaginally has decreased from 24.8% in 1998 to 16.7% in 2010, which may in part be due to the increasing utilization of laparoscopic and robotic approaches to hysterectomies [4]. Fewer vaginal hysterectomies being performed further translates into fewer number of vaginal hysterectomies available for resident training, resulting in a downward cycle of decreasing numbers of graduating residents achieving competence in this procedure [5]. The Accreditation Council for Graduate Medical Education reported that graduating residents in 2002 – 2003 performed a median of 31 vaginal hysterectomies during residency, which decreased to 21 in the 2016–2017 graduating class [6].

Lack of sufficient training and experience with vaginal hysterectomy has been reported as one of the major factors why a surgeon may choose a more invasive or expensive route of hysterectomy [6]. Moreover, the learning curve for vaginal hysterectomy can be steep and remediation of surgical performance following residency training is difficult. Analogous to the use of simulation training to enhance laparoscopic and robotic surgical skill acquisitions, surgical simulation may serve to address similar challenges in vaginal hysterectomy training by providing a standardized environment to learn and practice this procedure.

The use of simulation is not only essential for training but may be equally critical for assessment of proficiency. The need for objective competency-based assessment tools for certification of competency in surgical procedures has been recently addressed by the graduate medical education leadership [7, 8]. In addition, many medical boards are requiring demonstration of skills competency for board certification and maintenance of certification. For example, starting in 2016, the American Board of Obstetrics and Gynecology included participation in a simulation course as an option to fulfill Part IV of the Maintenance of Certification “Improvement in Medical Practice” [9]. The Board also recently added completion of Fundamentals of Laparoscopic Surgery as a requirement for physicians seeking board certification which will start in May 2020.

Teaching and assessment models have the greatest potential impact when the simulation closely approximates reality. However, few vaginal hysterectomy training models have been described, and even fewer high fidelity simulators are available. There is currently no well-studied, high fidelity simulation model for vaginal hysterectomy training and credentialing. The Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC) is a female pelvic anatomy model developed for realistic gynecologic surgical simulation of multiple procedures including vaginal hysterectomy. The aim of this study is to describe the Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC) and present pilot data on the face, content, and construct validity, and inter-rater reliability of the model as an assessment tool of vaginal hysterectomy skills.

Materials and Methods

Miya Model™ (Miyazaki Enterprises, Winston-Salem, NC)

The Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC) is a model of the female pelvis with life-like features that include vagina, vulva, pelvic viscera including an inflatable bladder, and bilateral pressurized vascular system (Figure 1). All the essential steps of a vaginal hysterectomy can be simulated on the model including entry into the anterior and posterior cul-de-sac, ligation of all ligamentous and vascular pedicles, and vaginal cuff closure. Although not the focus of this study, other procedures that can be performed on the model include both retropubic and transobturator midurethral slings, and anterior and posterior colporrhaphies. Each disposable/ non-reusable portion will support the performance of one hysterectomy, midurethral sling, anterior and posterior colporrhaphies. The entire model costs $6,500 U.S. dollars, and the disposable/ non-reusable portions (vagina, uterus/adnexa) costs $440 U.S. dollars with a newer, more simplified version of uterus and vagina costing less than $200 U.S. dollars.

Figure 1.

Front and back view of the Miya Model ™ (Miyazaki Enterprises, Winston- Salem, NC) (i) Front view of the Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC) pelvis showing external female genitalia (ii) View from behind exposing the uterus and bony pelvis with tubing to represent uterine arteries

Specifically, the model weighs 6.5 lbs. and is easily portable. It is comprised of an injection molded bony pelvis according to standardized gynecoid dimensions which attaches to a support bracket that sits on a stand. The pelvis can be rotated 360 degrees, and can be placed in Trendelenburg and reverse Trendelenburg positions. There are eight replaceable parts: vulva, vagina, uterus tubes and ovaries, bladder, obturator membrane, perineum and sacrospinous ligament. Each anatomic piece was designed to mimic real tissue from feel, cutting and suturing. The uterus, tubes and ovaries are one unit. It is made of a polyurethane elastomer with subcutaneous mesh reinforcement in the broad ligament. The broad ligament connects to the pelvic frame via stiff rods for secure attachment and easy replacement into sliding lock grooves in the pelvis. Uterine vessels, 6mm diameter tubing, run through the broad ligament and can be connected externally to standard intravenous tubing which can be connected to an intravenous bag of fluid to simulate vascular fluid. With a blood pressure cuff around the intravenous bag (can be colored red using food coloring), the simulated vascular system can be pressurized and the uterus/ broad ligament will bleed if the vessels (tubing) are not properly ligated. The bladder is made of a Polyurethane elastomer and has a hollow cavity which can be filled with liquid to simulate bladder injury with leakage of fluid if the simulated bladder is injured. There is a oneway valve in the urethra which simulates the urethral sphincter and allows for catheterization by a 12–16fr catheter. The bladder attaches anteriorly to a groove in pubic bone and fits snugly between pubic bone and uterus. Within the bladder, there is a raised ureteric ridge and ureteral dimples which represent the ureteral orifices for simulating performance of a diagnostic cystoscopy. There vulva, vagina, and attached rectum are made of medical grade, dual durometer Room-Temperature-Vulcanizing (RTV) platinum cure silicone.

Pilot Study

This study was exempted by the Institutional Review Board. Twenty surgeons, 10 obstetrics and gynecology residents and 10 experts (attending faculty in obstetrics and gynecology who routinely performed vaginal hysterectomies) were recruited from one institution to participate in a simulated vaginal hysterectomy procedure over the course of four weeks. Participants completed a questionnaire on demographics and medical experience including years in residency/ practice and number of vaginal hysterectomies performed. Prior to the procedure, a 10-minute presentation about the model, its design, and rules of the study was given to the participants and participants were given permission to examine the model. Each surgeon was then equipped with a Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC) Pelvic Surgery Training Model (Figure 1) and an identical set of standard vaginal surgery instruments and suture. Participants had two third-year medical students as assistants who were not allowed to perform any movements other than what was directed by the surgeon. All surgeons were provided with a typed copy of the procedural steps for vaginal hysterectomy according to ACOG Simulations Working Group vaginal hysterectomy toolkit and were given 60 minutes to complete the procedure [10]. An intravenous bag filled with 500 ml’s of water was pressurized to 60 mm of mercury and was attached to the uterine cartridge to simulate a pressurized vascular system. Each simulation was recorded by a videographer, all of whom were college-aged students, with identical cameras and stands that filmed from a position behind the surgeon. The videographers also recorded the time from beginning of hysterectomy to the end of vaginal cuff closure in minutes. If the 60-minute limit was reached, the last completed step was recorded. Estimated blood loss was recorded by measuring the remaining fluid volume in the bag.

Following completion of the simulation, to evaluate face validity, participants were asked to provide overall ratings of the model as a training and assessment tool for performance of vaginal hysterectomy using an anchored Likert scale of 1 – 7 (1, not effective to 7, highly effective). To evaluate content validity, participants were asked to rate the model on performance of each essential vaginal hysterectomy step.

To asses construct validity and interrater reliability, two experienced vaginal surgeons who were experts in surgical education research reviewed each recording independently and rated the performance using two validated assessment tools, the Objective Structured Assessment of Technical Skill (OSATS) global rating scale initially developed for open surgery by Reznick and colleagues and the Vaginal Surgical Skills Index (VSSI) developed and validated to evaluate vaginal hysterectomy skills [11, 12]. The reviewers were blinded to the identities of the participants including their levels of training and experiences with performing vaginal hysterectomy.

Statistical Tests

Categorical variables were compared using Chi-square test. Due to the small sample size, medians and interquartile ranges were calculated for continuous variables and the Wilcoxon Rank-Sum test was used to test for differences in median OSATS and VSSI scores between expert and novice surgeons. To evaluate for inter-rater reliability, weighted Kappa with 95% confidence interval were calculated. To further evaluate the association between individual participant experiences and performances on the simulator, participants’ years of experience and number of hysterectomies performed were correlated with OSATS scores and VSSI scores using the Kendall rank correlation coefficient. Statistical analysis was completed using STATA 14.2 (College Station, TX) with 0.05 level of significance.

Results

As expected, novice surgeons were significantly younger and performed fewer hysterectomies than expert surgeons (1). Four novice surgeons and one expert surgeon failed to complete the vaginal hysterectomy simulation within the one-hour time limit. Among the study participants who completed the simulation, time to procedure completion was significantly higher in the novice group compared to the experts (median (IQR): 58.5 minutes (45 – 60) vs. 40.5 minutes (37 – 44), respectively; p-value = 0.02). Median estimated blood loss was not different between the novice and expert groups (62.5mL (IQR 0 – 200) vs. 35mL (IQR 0 – 100) respectively; p-value = 0.76).

Table 1.

Participant demographics and experiences

Characteristics	Novice (n=10)	Expert (n=10)	P
Age category, years, n (%)			0.004
<30	6 (60%)	0
30–40	4 (40%)	3 (30%)
40–49	0	2 (20%)
50–59	0	5 (50%)
Gender, n (%)			0.65
Male	5 (50%)	6 (60%)
female	5 (50%)	4 (40%)
Years of experience post medical school, median (IQR)	2.5 (2–3)	16 (12–26)	< 0.001
Number of all hysterectomies performed or supervised/ year, median (IQR)	0(0–10)	40 (25–80)	< 0.001
Number of vaginal hysterectomies performed or supervised/ year, median (IQR)	0 (0–2)	12.5 (8–20)	< 0.001

Statistics are reported either as n (%) and compared using Chi-square test or median (interquartile range) and compared using Wilcoxon Rank-Sum Test.

Most novice and expert study participants found that the model was overall “effective” or “highly effective” as a simulation model for vaginal hysterectomy training and assessment (Table 2). Most novices and experts also ranked the model as being “effective” or “highly effective” when assessing the effectiveness of the model for performance of each step of the vaginal hysterectomy procedure (Table 3).

Table 2.

Assessment of effectiveness of Miya Model as a simulation tool between novice and expert surgeons

	Highly Ineffective + Ineffective n (%)	Somewhat Ineffective n (%)	Uncertain n (%)	Somewhat Effective n (%)	Effective + Highly Effective n (%)	Median (IQR)	P
Do you feel the Miya Model is an effective simulation tool for vaginal hysterectomy training?
Novice	0	0	0	2(20%)	8(80%)	7(6–7)	0.71
Expert	0	0	0	0	10(100%)	7(6–7)
Do you feel the Miya Model is an effective simulation tool to objectively assess skill in vaginal hysterectomy in accordance with ACOG guidelines?
Novice	0	0	1(10%)	2(20%)	7(70%)	6(5–7)	0.40
Expert	0	0	0	1(10%)	9(90%)	6(6–7)
Did the Miya Model simulate a vaginal hysterectomy much like a live patient?
Novice(n=8)	0	1 (13%)	0	2(25%)	5(62.5%)	6 (5–6)	0.88
Expert(n=10)	0	0	0	4(40%)	6(60%)	6(5–6)
Do you feel the model is a practical alternative/addition to the current Halsted teaching method of “See one, do one, teach one”?
Novice(n=9)	0	0	0	0	9(100%)	7(6–7)	0.36
Expert(n=10)	0	0	0	2(20%)	8(80%)	6(6–7)
Do you feel model training/simulation is important prior to live surgery for patient safety?
Novice(n=9)	0	0	0	0	9(100%)	7(6–7)	0.89
Expert(n=10)	0	0	0	0	10(100%)	7(6–7)

n =10 in each group, unless otherwise noted; median and interquartile range of scores were calculated in the novice and expert groups and the scores between groups were compared using Wilcoxon Rank-Sum Test.

Table 3.

Assessment of effectiveness of Miya Model for vaginal hysterectomy between novices and experts

Ability to:		Highly Ineffective + Ineffective n (%)	Somewhat Ineffective n (%)	Somewhat Effective n (%)	Effective + Highly Effective n (%)	Uncertain n (%)	Median(IQR)	P
Feel anterior and posterior cul de sac	Novice	0 (0%)	0 (0%)	2 (20%)	7 (70%)	1 (10%)	6 (5–7)	0.96
	Expert	0 (0%)	0 (0%)	1 (10%)	8 (80%)	1 (10%)	6 (6–7)
Enter anterior peritoneal cavity	Novice	0 (0%)	0 (0%)	3 (30%)	7 (70%)	0 (0%)	7 (5–7)	0.53
	Expert	0 (0%)	0 (0%)	2 (20%)	7 (70%)	1 (10%)	6 (5–7)
Enter posterior peritoneal cavity	Novice	0 (0%)	0 (0%)	1 (10%)	9 (90%)	0 (0%)	7 (6–7)	0.35
	Expert	0 (0%)	0 (0%)	0 (0%)	10 (100%)	0 (0%)	7 (6–7)
Clamp uterosacral ligaments	Novice	0 (0%)	0 (0%)	0 (0%)	7 (70%)	3 (30%)	6 (4–7)	0.97
	Expert	0 (0%)	0 (0%)	3 (30%)	6 (60%)	1 (10%)	6 (5–7)
Clamp cardinal ligaments	Novice	0 (0%)	0 (0%)	1 (10%)	7 (70%)	2 (20%)	6 (5–7)	0.14
	Expert	0 (0%)	0 (0%)	2 (20%)	8 (80%)	0 (0%)	7 (6–7)
Clamp and ligate uterine arteries	Novice	0 (0%)	0 (0%)	5 (50%)	4 (40%)	1 (10%)	5 (5–6)	0.18
	Expert	0 (0%)	0 (0%)	2 (20%)	7 (70%)	1 (10%)	6 (5–7)
Properly place clamps	Novice	0 (0%)	1 (10%)	3 (30%)	6 (60%)	0 (0%)	6 (5–7)	0.13
	Expert	0 (0%)	0 (0%)	1 (10%)	9 (90%)	0 (0%)	7 (6–7)
Use proper surgical technique	Novice	0 (0%)	0 (0%)	0 (0%)	9 (90%)	1 (10%)	6 (6–7)	0.34
	Expert	0 (0%)	0 (0%)	0 (0%)	10 (100%)	0 (0%)	7 (6–7)
Evaluate pedicles for homeostasis	Novice	1 (10%)	0 (0%)	1 (10%)	7 (70%)	1 (10%)	6 (5–6)	0.84
	Expert	0 (0%)	0 (0%)	3 (30%)	7 (70%)	0 (0%)	6 (5–6)
Close vaginal cuff	Novice (n=9)	0 (0%)	0 (0%)	0 (0%)	7 (78%)	2 (22%)	7 (6–7)	0.27
	Expert (n=10)	0 (0%)	0 (0%)	1 (10%)	9 (90%)	0 (0%)	6 (6–6)
Perform a culdoplasty	Novice (n=8)	0 (0%)	0 (0%)	0 (0%)	5 (63%)	3 (37%)	6 (4–7)	0.68
	Expert (n=7)	0 (0%)	0 (0%)	1 (14%)	5 (72%)	1 (14%)	6 (5–7)

n =10 in each group, unless otherwise noted; median and interquartile range of scores were calculated in the novice and expert groups and the scores between groups were compared using Wilcoxon Rank-Sum Test.

No significant differences were observed in the composite median OSATS or VSSI scores between novice and expert surgeons (Table 4). Among the individual subscales within each global rating scale, only the “knowledge of instalments” subscale scores in OS ATS were significantly different between novices and experts (median 3.5 (IQR2–4) vs. median 4.5 (IQR4 – 5), respectively; p-value < 0.05).

Table 4.

Inter-rater agreement and OSATS and VSSI scores in novices compared with expert

Variable		Interrater Reliability (95% Cl)		Novice (n=10) median (IQR)	Expert (n=10) median (IQR)	P
Objective structured assessment of technical skills (OSATS)
Respect for tissue		0.83 (0.69, 0.93)		2.5 (2–4)	3 (2 – 4)	0.82
Time and motion		0.88 (0.72, 0.97)		2.5 (1–3)	3.5 (2–4)	0.12
Instrument handling		0.87 (0.78, 0.95)		2.5 (1–3)	3 (2 – 4)	0.26
Knowledge of instruments		0.79 (0.65, 0.91)		3.5 (2–4)	4.5 (4–5)	0.05
Use of assistants		0.84 (0.69, 0.95)		3(1–4)	4(3–5)	0.17
Flow of operation and  forward planning		0.79 (0.62, 0.91)		3 (2 – 4)	3.5 (2–4)	0.73
Knowledge of specific procedure		0.88 (0.77, 0.96)		4(3–5)	4(3–5)	0.87
Composite score		0.90 (0.82, 0.96)		21 (11–27)	25 (19–31)	0.24
Vaginal surgical skills index (VSSI)
Incision		0.83 (0.65, 0.95)		2.5 (2–3)	3 (2 – 4)	0.43
Main visibility		0.83 (0.69, 0.94)		2(1–3)	3 (2 – 4)	0.21
Use of assistants		0.78 (0.61, 0.91)		2 (2 – 3)	2.5 (2–3)	0.47
Knowledge of instruments		0.85 (0.64, 0.96)		4 (2 – 4)	3.5 (3–4)	0.97
Tissue and instrument handling		0.77 (0.5, 0.89)		2.5 (2–3)	2.5 (2– 3)	0.68
Knot tying		0.85 (0.66, 0.94)		3 (2 – 4)	3 (2 – 4)	0.75
Hemostasis		0.81 (0.58, 0.98)		2.5 (1–3)	2.5 (1–4)	0.69
Procedure completion		0.85 (0.70, 0.94)		3 (2 – 4)	3 (2–3)	0.91
Time and motion		0.85 (0.70, 0.95)		2(1–3)	2.5 (1–3)	0.58
Flow of operation/forward planning		0.78 (0.57, 0.91)		2.5 (2–3)	2.5 (2–3)	0.69
Knowledge of procedure		0.90 (0.78, 1.00)		3.5 (3–4)	4 (2 – 4)	0.90
Composite		0.93 (0.89, 0.98)		28.5 (17–37)	30(24–38)	0.50

Interrater reliability between the two raters was calculated using weighted Kappa.

OSATS and VSSI composite scores (sum of subscale scores) and subscale scores of the novice group were compared to the scores of the expert group using Wilcoxon Rank-Sum Test.

Each subscale in the composite OSATS global rating scale are scored on a 5-point anchored Likert- type scale (1 – 5) with higher scores indicating better performance.

Each subscale in the composite VSSI global rating scale are scored on a 4-point anchored Likert- type scale (0 – 4) with higher scores indicating better performance.

VSSI also included “initial inspection” and “electrosurgery” but as participants were not specifically instructed to inspect the operative field before getting started and electrosurgery cannot be used with this model, these parameters were not assessed.

The inter-rater reliability indices for each subscale score of the OSATS scale and the composite OSATS score were high and ranged from 0.79 to 0.90 (Table 4). The inter-rater reliability indices for each subscale score of the VSSI scale as well as the composite VSSI score were similarly high and ranged from 0.77 to 0.93.

To further evaluate the association between individual participant experiences and performances on the simulator, participants’ years of experience (postgraduate year) and number of hysterectomies performed were correlated with OSATS scores and VSSI scores. Median composite OSATS scores as well as subscale scores for some of the individual parameters within OSATS were found to be significantly but weakly correlated with postgraduate year (composite score: r = 0.33, p-value = 0.05) (Table 5). Although some of the individual OSATS subscale scores were also significantly and weakly correlated with number of vaginal hysterectomies performed, the composite OSATS scores were not significantly correlated with this measure of participant experience (composite score: r = 0.26, p-value = 0.14). While composite VSSI scores were also weakly associated with postgraduate years, which bordered on significance (r = 0.32, p-value = 0.07), composite VSSI scores were not significantly correlated with number of vaginal hysterectomies performed (r= 0.22, p-value = 0.21). Within the individual subscales of VSSI, “maintenance of visibility” and “use of assistants” subscale scores were weakly but significantly associated with postgraduate years.

Table 5.

Correlations of OSATS and VSSI scores to postgraduate years of experience and total number of vaginal hysterectomies performed

Variable	Postgraduate Year		Vaginal Hysterectomy Total
	r	P	r	P
Objective structured assessment of technical skills (OSATS)
Respect for tissue	0.19	0.29	0.11	0.55
Time and motion	0.36	0.04	0.35	0.05
Instrument handling	0.32	0.07	0.29	0.10
Knowledge of instruments	0.50	0.005	0.39	0.03
Use of assistants	0.40	0.02	0.37	0.04
Flow of operation and forward planning	0.23	0.19	0.23	0.21
Knowledge of specific procedure	0.22	0.23	0.11	0.56
Composite score	0.33	0.05	0.26	0.14
Vaginal surgical skills index (VSSI)
Incision	0.33	0.06	0.34	0.06
Main visibility	0.44	0.01	0.35	0.06
Use of assistants	0.41	0.02	0.35	0.06
Knowledge of instruments	0.26	0.16	0.17	0.37
Tissue and instrument handling	0.26	0.15	0.21	0.25
Knot tying	0.24	0.20	0.19	0.32
Hemostasis	0.28	0.12	0.12	0.52
Procedure completion	−0.02	0.92	−0.01	1.00
Time and motion	0.33	0.07	0.29	0.11
Flow of operation/ forward planning	0.30	0.09	0.27	0.15
Knowledge of procedure	0.18	0.36	0.15	0.45
Composite score	0.32	0.07	0.22	0.21

Participants’ years of experience following medical school and number of hysterectomies performed were correlated with OSATS and VSSI composite scores (sum of subscale scores) as well as subscale scores using the Kendall rank correlation coefficient.

VSSI also included “initial inspection” and “electrosurgery” but as participants were not specifically instructed to inspect the operative field before getting started and electrosurgery cannot be used with this model, these parameters were not assessed.

Discussion

In this study, we described the Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC) for vaginal surgery and presented pilot data on validity and reliability for the assessment of vaginal hysterectomy skills. Specifically, we demonstrated face and content validity as most expert and novice surgeons rated this model as an “effective” or “highly effective” tool for teaching and assessing vaginal hysterectomy skills as well as for performing each essential vaginal hysterectomy step. We also demonstrated that experts performed vaginal hysterectomy more efficiently than novices with significantly decreased time to procedure completion. Although we were not able to demonstrate that experts were more skillful at performing vaginal hysterectomies than novices as assessed using OSATS and VSSI global rating scales, when we further analyzed the data by correlating the composite OSATS and VSSI scores with participant experiences, median composite OSATS scores were weakly but significantly correlated with postgraduate years, i.e., participants who were more experienced had higher OSATS scores. Similarly, median composite VSSI scores were also weakly correlated with postgraduate years, which bordered on significance. Importantly, we also demonstrated that this model allowed for reliable assessments of vaginal hysterectomy skills as the inter-rater reliability indices were high.

One common criticism of synthetic surgical simulation models is the lack of fidelity, i.e., synthetic models do not feel lifelike and therefore does not simulate live surgery when used for surgical practice [13, 14]. A previously published model for vaginal hysterectomy used a pine stud covered in Velcro and stocking material as the uterus, which was then placed into a commercially manufactured plastic bony pelvis [15]. While the study participants found the model useful for education, the authors did not evaluate the model for construct validity. Another low-fidelity model used a balloon filled with caulk as the uterus that was then inserted into the hand portion of a surgical glove. The thumb and fingers of the glove were used to simulate the various ligaments which was attached into a commercially available birthing simulator pelvis [16]. Seventy-five percent of participants who evaluated that model rated it as effective for teaching vaginal hysterectomy. However, this study also did not assess the validity of model for teaching or assessing surgical skill; instead, it only addressed the efficacy of the model as part of a more comprehensive educational intervention to increase the written knowledge of study participants. Unlike the two previously published low-fidelity vaginal hysterectomy models, the Miya Model ™ (Miyazaki Enterprises, Winston-Salem, NC) exhibits realistic visceral and vascular anatomy as well as bony landmarks and tissue planes which mimics the tissue tension a surgeon would experience operating on biologic tissue. Furthermore, in this study, we were able to demonstrate the face and content validity of this model.

Simulation training is well-suited for complex procedures with steep learning curves such that repetitive practice can allow for acquisition of skills without compromising patient safety. Vaginal hysterectomy is one such procedure given that surgeons must effectively navigate the complex anatomic relationships of vital structures within the relatively restricted confines of the bony pelvis. These limitations may also impact the faculty’s ability to effectively teach this procedure to trainees and assess trainees’ proficiency. Moreover, as medical education and credentialing shifts from the traditional experienced-based model to a demonstration of proficiency model, it is imperative that we develop simulation models that permit the valid and reliable assessment of surgical skills. Although the Miya Model (Miyazaki Enterprises, Winston-Salem, NC) allowed for a reliable assessment of vaginal hysterectomy skill using two global rating scales (OSATS, VSSI) that have previously been shown to be valid and reliable in intraoperative settings, we did not find significant differences in global rating scale scores between novices and experts. This is likely due to our small sample size as when we increased our statistical power by correlating individual global rating scale scores with participant experiences (postgraduate years, number of vaginal hysterectomies performed), we found weak but significant or bordering on significant correlations.

Limitations of this pilot study include the small number of participants, which may contribute to the lack of statistical significance found in this study. Moreover, although both raters were familiar with each global rating scale, a pre-grading training session for each rater focused on building consensus on rating each parameter within both global rating scales may further strengthen the study findings. We are rectifying these limitations in an ongoing NIH funded project to further study this model.

Furthermore, although having a valid and reliable simulation model for assessing vaginal hysterectomy skills is important, it is equally critical to have a valid and reliable simulation model for learning vaginal hysterectomy skills. This was not addressed in this study but will be evaluated in future projects. We also did not evaluate if the surgical skills assessed in this simulator setting reflects actual skills in the operating room or if the skills gained in a simulator setting are transferable to an operating room setting. These are planned areas of future research. Importantly, surgical skill is only one aspect of surgical performance. We also did not evaluate for other aspects of surgical performance including surgical judgement.

The strengths of this study included the use of two experienced vaginal surgeons who reviewed each recording independently and rated each performance while blinded to participant identity and experience. The reviewers were from a different institution than the first author and the study participants, and did not have any commercial ties to this model. Each reviewer evaluated video recordings of each surgical performance using two well-studied global rating scales (OSATS and VSSI) including one specifically developed to assess vaginal surgical skills that was previously shown to be reliable and valid [11, 12].

In conclusion, we developed a high-fidelity simulation model for teaching and assessing vaginal skills. In this pilot study, we used this model to assess vaginal hysterectomy skills and found that the model demonstrated face validity, content validity and high reliability. With further study, the Miya Model ™ (Miyazaki Enterprises, Winston- Salem, NC) may be a useful tool for teaching and assessing vaginal surgical skills and may be adapted by residency programs to establish competency in its trainees and surgical specialty organizations as a requirement for ongoing maintenance of certification.