Development and implementation of a clinical pathway approach to simulation-based training for foregut surgery

Abstract

Introduction

Contemporary demands on resident education call for integration of simulation. We designed and implemented a simulation-based curriculum for PGY1 surgery residents to teach technical and non-technical skills within a clinical pathway approach for a foregut surgical patient, from outpatient visit through surgery and post-op follow-up.

Methods

The three-day curriculum for groups of six residents comprises a combination of standardized patient (SP) encounters, didactic sessions, and hands-on training. The curriculum is underpinned by a summative simulation “pathway” repeated on days 1 and 3. The “pathway” is a series of simulated pre-op, intra-op, and post-op encounters following a single patient through a disease process. The resident sees an SP in clinic presenting with distal gastric cancer, then enters an operating room to perform a gastro-jejunostomy on a porcine tissue model. Finally, the resident engages in a simulated post-operative visit. All encounters are rated by faculty members and the residents themselves, using standardized assessment forms endorsed by the American Board of Surgery.

Results

18 first-year residents underwent this curriculum. Faculty ratings of overall operative performance significantly improved following the three-day module. Ratings of preoperative and postoperative performance were not significantly changed in three days. Resident self-ratings significantly improved for all encounters assessed, as did reported confidence in meeting defined learning objectives.

Conclusions

Conventional surgical simulation training focuses on technical skills in isolation. Our novel “pathway” curriculum targets an important gap in training methodologies by placing both technical and non-technical skills in their clinical context as part of managing a surgical patient. Results indicate consistent improvements in assessments of performance as well as confidence and support its continued usage to educate surgery residents in foregut surgery.

Introduction

Simulation-based training is gaining attention in residency education as a way to address contemporary demands for increased patient safety and accountability. Governing bodies for graduate medical education, as well as various professional societies are requiring surgical residency programs to utilize competency-based methods of evaluation, and encourage integration of simulation into training curricula. However, the specifics of how to design and implement such curricula have yet to be established, and are left to individual institutions.

Foregut surgery is a growing field encompassing diseases of the esophagus and stomach, as well as bariatric surgery. Foregut surgery can be technically challenging, with even classic foregut surgery procedures such as esophagectomy still associated with considerable complication rates.¹ Emerging procedures such as bariatric surgery are also associated with unique technical challenges in perioperative management.² A recent needs assessment of national stakeholders in surgery training has also revealed that non-technical skills of communication with patients and families spanning the perioperative period is perceived as one of the key themes that should be addressed in surgical training.³

Complex issues such as duty hour standards, combined with ever-increasing demand to staff clinical services, creates significant tension between residency programs’ aims to provide rigorous yet well-balanced education, and health systems which seek to remain financially solvent through reimbursement for health care services rendered.

We designed and implemented a simulation-based curriculum for first-year surgery residents to provide integrative training of technical and non-technical skills centered around a clinical pathway approach for a foregut surgical patient. The clinical pathway represents a continuum of care for a foregut surgical patient, and is comprised of a sequence of patient care encounters between a resident and patient, from an outpatient visit through surgery and postoperative follow-up. Using this patient-centric approach allows for balanced delivery of education as well as evaluation of technical and non-technical skills in their appropriate context.

Material and Methods

Setting

This research was conducted as part of the residency program in general surgery at the Hospital of the University of Pennsylvania. Pursuant to institutional standards, we submitted our protocol to the Institutional Review Board and received confirmation of exemption under 45 CFR 46.101, category 1 for human subjects research regarding the effectiveness of instructional curricula in an established educational setting. Written consent was sought from all participating residents regarding the collection of data on their simulated clinical performances for the purpose of ongoing curricular improvement, research and publication, with the understanding that their consent or refusal would not have any impact on the provided educational content or their standing as a resident in the program.

The entire educational curriculum was implemented on site at the Penn Medicine Clinical Simulation Center.⁴ This 22,000 square-foot facility incorporates various classrooms, skills training rooms, as well as simulated operating rooms, inpatient ward, and outpatient clinic environments. Each simulation room is equipped with an electronic audiovisual system (SimCapture, B-Line Medical, Washington DC, USA) that allows for simultaneous live monitoring as well as recording of encounters of multiple camera angles from multiple rooms. These live or recorded video streams may be accessed from a dedicated viewing room, or by logging into a portal site from a browser on any computer on the same network. Recorded videos are only accessible by the system administrators, the study team, and the learners themselves using individual login information and passwords.

Learning Objectives

Foregut surgery was selected as one of six surgical specialty areas in which first-year residents would benefit from simulation-based training (the others areas being acute care, biliary, cardiovascular, colorectal, and trauma/surgical critical care). The surgery simulation program director consulted with surgery faculty to outline learning objectives for the curriculum.

Content from the SCORE (Surgical Council on Resident Education) Portal was referenced as a starting point for the selection of level-appropriate curricular content.⁵ Table 1 shows the topics selected for inclusion in the module content from the relevant SCORE curricular headings: Alimentary Tract – Stomach, and Alimentary Tract – Esophagus, and Endoscopy. Priority was given to reinforce topics and skills that are expected to be a routine part of the residents’ immediate practice. For example, our institution has an active bariatric surgery program, and residents frequently encounter patients on this service. Bariatric surgery is usually reserved for more advanced level trainees, but was considered relevant to our first-year residents’ immediate practice and was included in the module learning objectives.

Table 1.

Selected SCORE Content

SCORE Heading	Alimentary Tract - Stomach	Alimentary Tract - Esophagus	Endoscopy
Topic	Duodenal Ulcer Gastrectomy – Partial/Total Gastric Cancer Gastric Ulcer Morbid Obesity Morbid Obesity – Operation Peptic Ulcer Disease with Bleeding Peptic Ulcer Disease with Obstruction Peptic Ulcer Disease with Perforation	Antireflux Procedure - Laparoscopic Antireflux Procedure - Open Dysphagia Gastroesophageal Reflux/Barrett’s Esophagus	Esophagogastroduodenoscopy

Learning objectives were adjusted through an iterative process based on realistic limitations of scheduling, balance of various learning modalities within the module, feasibility of standardized patient scenario design, as well as faculty availability as discussed below. The final list of 13 learning objectives defined for the module is shown in Table 2.

Table 2.

Defined learning objectives of the module, and resident self-confidence in meeting these defined learning objectives.

#	Learning Objective	Pre Median	Post Median	WSR p-value
1	Knowledge of diagnoses of foregut disease	3	4	0.0015
2	Knowledge of appropriate investigations for a patient with foregut disease	3	4	0.0016
3	Operative consent of patients with foregut disease	3	4	0.0016
4	Knowledge of the surgical anatomy of the foregut	3	4	0.0011
5	Technical skills for basic surgical anastomosis	2	4	0.0008
6	Pre-operative management of patients with foregut disease	3	4	0.0011
7	Intra-operative management of patients with foregut disease	2	4	0.0007
8	Post-operative management of patients with foregut disease	3	4	0.0082
9	Management of post-operative complications of foregut disease	3	4	0.0010
10	Technical skills for basic upper gastrointestinal endoscopy	2	3	0.0012
11	Management of bariatric surgery patients	2	4	0.0071
12	Diagnosis and management of patients with GERD	3	4	0.0063
13	Diagnosis and management of patients with peptic ulcer disease	3	4	0.0036
1–13	Pooled	3	4	0.0000

Foregut surgery is a broad and diverse topic, covering a multitude of disease states and interventions. Teaching faculty in foregut surgery also come from diverse backgrounds, with interests ranging from melanomas to robotic bariatric surgery. While learning objectives should be specific, generic phraseology was used to allow discretion by teaching faculty, who were encouraged to draw from experience and expertise in their respective practices. However, future work will need to balance the need to cover a broad range of potential topics while retaining specific, measurable learning objectives.

Scheduling of Residents

Dedicated educational time was scheduled for first-year residents to report to the simulation center for training. Days equivalent to a one-month rotation (18 days) were allocated to this initiative for the 18 first-year residents in the general surgery program. Groups of up to six residents were scheduled for three-day weekday blocks distributed throughout the year. This modular implementation accommodates residents in small groups to minimize staffing disruptions. Other residents, physician assistants, and non-physician practitioners on the team cover resident duties while they take part in simulation education. These compact three-day modules can be repeated over the course of the year to ensure the entire resident class is given equal access to this educational opportunity.

The resulting curriculum implementation utilized all 18 dedicated training days (6x 3-day modules) for each resident over the course of a year. An admitted reality of clinical training programs is the essential role of trainees in staffing clinical services. Finding a balance between service and training is a recurring issue in contemporary graduate medical education. At the same time, participation in educational programs is an integral part of the duties of a resident and is subject to duty hour regulations. The implementation was subsequently streamlined to 12 training days per year (4x 3-day modules, one per month) to allow for more flexibility in staffing of clinical services to better guarantee a minimum standard of care for patients.

Design of the Module

The three-day module for groups of six residents comprises a combination of standardized patient (SP) encounters, didactic sessions, and hands-on training. A summarized schedule of the three days is shown in Figure 1.

Figure 1.

Foregut Module Schedule Grid

Didactic Sessions

Overviews of basic knowledge of common disease processes in foregut surgery were provided in the format of didactic sessions. Topics selected were gastroesophageal reflux disease (GERD), peptic ulcer disease, and bariatric surgery (two sessions). Gastric cancer replaced one of the bariatric surgery sessions for the second year of implementation.

Case-based discussion questions associated with the relevant online modules of the SCORE curriculum were transcribed to PowerPoint format and provided to faculty. Between 5–25 discussion questions were available for each topic. These standardized materials were generally sufficient to provide topics to serve as a basis for discussions to fill most of the allotted time. To accommodate a range of teaching styles and expertise, faculty were permitted to bring and use their own didactic material and allocate time to various topics at their discretion.

Standardized Patients

While traditional surgical simulation tends to focus on technical skills, we sought to conduct a more comprehensive form of training and evaluation that reflects the reality of clinical practice. To this end, we established a working relationship with our institution’s standardized patient program to implement patient-centric simulated encounters.

SPs that meet the criteria for the case scenarios are cast a few months in advance of each module iteration and provided with detailed case descriptions. The SP undergoes training about a week prior to the actual module run to confirm that they are able to accurately answer questions regarding the relevant history described in the case, and enact the described physical findings. Questions and clarifications regarding case details were fielded by the simulation fellow.

On the day of the module, an SP coordinator and an audiovisual technician are part of the staff in addition to the SPs themselves. The SP coordinator tracks the timing of each encounter, and ensures that the appropriate evaluations are conducted. The audiovisual technician operates the digital video recording system to start and stop recording of encounters, adjust camera angles as appropriate, and ensure the resident name and encounter type are properly associated with the recording.

The Pathway Simulation

The module is underpinned by the summative ‘pathway’ simulations repeated on the first and third (final) day. The ‘pathway’ is a series of simulated preoperative, intraoperative, and postoperative encounters following a single patient through a disease process. The resident sees an SP in clinic presenting with distal gastric cancer, then enters an operating room to perform a gastro-jejunostomy on a porcine tissue model. Finally, the resident engages in a simulated postoperative visit. The nature of each encounter will be described in more detail below.

The SP scenarios were developed by a content expert, who provided detailed case descriptions for the ‘pathway’ sequence, as well as three additional preoperative and postoperative scenarios which are described below. The goal was to develop a case description with sufficient detail for the SP to provide a complete patient history, pertinent positive and negative physical findings, as well as having associated vital signs, laboratory, imaging, and pathology findings available for the resident to reference. SPs were cast to match the desired case description (age, sex, race) as closely as practical, and props and moulage were also added as appropriate to the description.

For all six residents in the group to complete the sequence of encounters within a 4-hour window, the time of each encounter must be limited. Upon consideration of the time required, up to 15 minutes were allocated to the preoperative encounter, 20 minutes for the operative encounter, and 10 minutes for the postoperative encounter.

The same SP sees all the residents, and plays both the preoperative and postoperative roles. Therefore the encounters take place on a cascading basis – the residents proceed one by one in sequence, with the leading resident’s intraoperative encounter and the trailing resident’s preoperative encounter taking place simultaneously. Observing faculty alternate between two screens, listening to one resident’s SP encounter while keeping an eye on another resident’s operative performance. Upon completing all the preoperative encounters, the SP is dressed and moulaged, and postoperative encounters can begin. Now the leading resident’s postoperative encounter and trailing resident’s intraoperative encounter proceed simultaneously. The SP does not break character until the entire sequence of encounters is complete, at which point they provide feedback regarding interpersonal aspects of the encounters to each resident. Finally, the observing faculty rater provides a short debriefing to the residents either individually or as a group depending on their preference and remaining time available.

Preoperative Encounters

Four preoperative scenarios were developed for use in the curriculum: anemia secondary to chronic bleeding from distal gastric cancer, dysphagia due to a tumor in the cardia, Barrett’s esophagus, and malnutrition due to distal esophageal carcinoma. The first case (distal gastric cancer) also serves as the preoperative case for the pathway simulation. Residents are provided with ‘doorway information’ for each case, which describes the setting, chief complaint, and vital signs. Associated diagnostic information appropriate to the case, such as laboratory and imaging results from the referring physician are also provided for the resident to review.

For practical reasons of casting standardized patient actors, the four cases describe patients of similar age range and physical characteristics. For example, all four cases for this module describe patients 60–65 years old, meaning the roles can be enacted by the same person. In addition, an effort was made to make the cases gender neutral to allow for more flexible scheduling of actors/actresses by the standardized patient program.

The Intraoperative Encounter

Twenty minutes are allocated to the intraoperative patient encounter, during which the residents are tasked with the performance of a hand-sewn gastro-jejunostomy as part of an ongoing sub-total gastrectomy. The gastrojenual anastomosis was deemed be a sufficiently relevant technical challenge that is realistically encountered in clinical practice, thus an appropriate task for the residents to learn. It is possible to approach completion within the 20 minute time window if performed efficiently.

Cadaveric porcine tissue was sourced from Animal Technologies Inc. in Tyler, Texas. Porcine stomach with attached esophagus (approximately $14.80 per unit), as well as segments of porcine small bowel (approximately $9.20 per 10 foot segment) were shipped frozen and stored in a dedicated commercial freezer at the simulation center prior to defrosting and use. Disposal of biohazard waste was also arranged through the facility housing the simulation center.

The porcine tissue is arranged inside a torso-shaped open container (BTS300 Torso Simulator, Pharmbotics Ltd, United Kingdom), placed on a surgical table and draped as a patient undergoing open abdominal surgery would be. Body parts such as a head, arms, and legs may be supplemented using various manikin parts to enhance the experience. A draped Mayo stand holds surgical instruments and common suture types that might be used to perform the anastomosis. An anesthesia machine is positioned at the head of the bed as would be for general anesthesia. Surgical lamps are positioned to illuminate the surgical field. Residents are asked to wear a cap, mask, gown, and sterile gloves prior to approaching the field. While it is not feasible to mirror every last detail of an actual intraoperative environment, care is taken to achieve a believable level of fidelity that allows a willing learner to live their role, without being forced to pretend and act at all times.

Confederates were trained to serve as a surgical assistant (a medical student) and anesthesiologist during the intraoperative encounter. While the necessity of these confederate roles may appear minimal, we felt it a vital component of the psychological validity of the immersive simulation experience. Even during this simulated encounter, the learners should be of the mindset that they are caring for a patient as a responsible member of a professional team, with the expectation that appropriate attention to patient safety as well as professional behavior and courtesy should be displayed at all times.

The confederate anesthesiologist role was filled by an actor cast by the standardized patient program. A role description and script was provided for the actor to greet the resident and perform a time out to confirm the patient’s name, procedure, and prophylactic antibiotic administration if requested. The patient’s simulated vital signs are displayed on a computer monitor (Patient Monitor Application, Laerdal Medical, Norway) along with sounds such as pulse tone and intermittent noninvasive blood pressure measurements. The parameters of the scenario are defined such that the patient is stable on a maintenance dose of anesthesia, with vital signs remaining unchanged throughout the procedure regardless of the operative situation. The anesthesiologist may engage in light conversation with the resident, as well as respond to other requests such as adjustment of table height and ambient music.

A student researcher or the simulation fellow served as confederates to play the role of a medical student scrubbed in the operating room to assist the resident. The assistant medical student role calls for a rudimentary understanding of sterile technique and surgical instruments (e.g. scissors, forceps). The assistant uses basic instruments as instructed to handle tissue, but does not perform more invasive or technical tasks such as cutting, knot tying, and suturing unless specifically instructed and coached step by step. The assistant is permitted to point out known limitations of the simulation environment and model (e.g. abbreviations in sterile technique, restriction on use of energy devices), but does not volunteer any help or information to the resident regarding the procedure.

Postoperative Encounters

Ten minutes were allocated to each postoperative patient encounter. Four postoperative cases were developed in line with the preoperative encounters: normal postoperative day 10 subtotal gastrectomy, intra-abdominal hemorrhage 3 hours following a laparoscopic sub-total gastrectomy, duodenal stump leak on postoperative day 4 of a laparoscopic sub-total gastrectomy, and an anastomotic stricture 8 weeks following an open total gastrectomy. Again, the first case is uncomplicated and corresponds to the postoperative segment in the pathway simulation sequence. The other cases need not have matching preoperative counterparts, allowing for greater flexibility in scenario design. However, the four cases are kept within a reasonable age range due to the practical limitations of casting, and are kept gender-neutral where possible.

Peer-engaged Learning

On the second day of the module, standardized patient encounters were used in a formative setting for peer-engaged learning. A single SP enacted the four scenarios, preoperative or postoperative, as previously described. Residents in the group would each be assigned a scenario. While one resident interacted with the SP, the remaining residents, along with attending faculty, observed the interaction on video from a separate viewing room. At the end of the simulation, the resident rejoined his/her colleagues and faculty to present and discuss the specific case, as well as engaging in discussions regarding alternate approaches to the management of the relevant disease process and/or complication. This peer-engaged simulation offers a unique opportunity for residents to directly observe how their peers approach patient management, as well as spend time discussing diverging paths in patient management algorithms in ways that are not usually possible in the traditional bedside teaching round setting. This debriefing session is intended to be a safe, relaxed space for professional development, rather than a formal evaluation.⁶ At the conclusion of the final encounter, the SP provided feedback to the group on interpersonal aspects of the encounter.

Upper GI Endoscopy Training

The American Board of Surgery Flexible Endoscopy Curriculum mandates a basic understanding of gastrointestinal diseases and endoscopic anatomy, as well as “simulation exposure with an emphasis on basic scope manipulation including one-handed wheel deflection, control of suction, irrigation, and insufflation, and passage of instruments through the working channel” as the first level of exposure to endoscopy to be typically completed at the PGY1 or PGY2 level.⁷ The Penn Clinical Simulation Center maintains two virtual reality task trainers - GI Mentor (Simbionix USA, Cleveland, OH) and Endoscopy AccuTouch (CAE Healthcare USA, Sarasota, FL), which are made available to the residents for training. In addition, an Olympus endoscopy tower and gastroscope (GIF-Q160, Olympus America, Center Valley, PA) are available for a tutorial on endoscope components, as well as equipment setup and troubleshooting. Residents attempted a series of simulated diagnostic endoscopy cases under supervision of the simulation fellow to gain familiarity with basic scope manipulation.

Scheduling of Faculty

At least one month prior to each iteration of the module, the surgery simulation program director invited members of the general surgery faculty to participate. Based on their availability and interest, faculty signed up to teach the nine 4-hour teaching blocks that comprise the three-day module. In the first year of implementation, it was possible to fill all faculty teaching blocks for the three iterations of the module. The simulation program director or the simulation fellow would substitute in the event scheduled faculty were unable to attend due to last-minute clinical or personal emergencies. In rare cases (such as a university-wide access emergency due to heavy snowfall) sessions would be rearranged or cancelled, though every effort was made to deliver the prepared educational content in the allotted time.

Evaluation of Resident Performance

The American Board of Surgery requires assessment of operative and clinical (outpatient and/or inpatient) performance during general surgery residency. Program directors are already required to attest to the completion of two operative and two clinical performance assessments for residents completing programs in the 2012–2013 academic year or thereafter, with the requirement increasing to six assessments each from the 2015–2016 academic year.⁸ The trend for requirement of ongoing clinical assessments continues with the ACGME Milestone project, which defines specific levels of achievement that residents are expected to demonstrate at established intervals as they progress through training.⁹ While a variety of measurement tools exist for the evaluation of clinical performance, validity evidence is scarce and not yet conclusive.¹⁰ The ABS strongly suggests the use of the following forms, which represent the current state of the art in assessment of clinical performance by direct observation.

CAMEO

The Clinical Assessment and Management Exam – Outpatient or CAMEO is designed to evaluate surgery residents’ ability to assess and manage a patient in an initial outpatient clinic encounter.¹¹ The assessment is based on five criteria (test ordering and understanding, diagnostic acumen, history taking, physical examination, and communication skills) in addition to overall performance, each scored on a 5 point Likert scale. There is also an indication of the difficulty of the case on a 3 point Likert scale. For our module, attending faculty observe the residents’ simulated outpatient encounter on video and perform a live rating of their performance. Immediately following the encounter, each resident evaluates their own performance using the same form.

OPRS

The Operative Performance Rating System (OPRS) is used to rate the intraoperative technical skills of a surgeon.^{12, 13} The assessment consists of five general criteria, several additional procedure-specific criteria as available, as well as an indication of case difficulty and degree of prompting or direction by attending faculty. Established procedure-specific criteria were not available for gastro-jejunostomy, so only the six general criteria (respect for tissue, time and motion, instrument handling, knowledge of instruments, flow of operation, use of assistants), and ‘knowledge of specific procedure’ as a measure of overall performance were used. Again, each criterion is rated on a 5 point Likert scale. Attending faculty observe the residents’ simulated operative encounter on video and rate their performance. Each resident also evaluates their own performance immediately following the encounter.

Mini-CEX

The Mini Clinical Evaluation Exercise (Mini-CEX) is an tool for assessment of trainees in any setting.¹⁴ It is associated with the strongest available validity evidence.¹⁰ The Mini-CEX assessment consists of six criteria in addition to overall clinical competence. Each criterion is scored on a 9-point scale, grouped into three performance categories (1–3 for unsatisfactory, 4–6 for satisfactory, and 7–9 for superior). Supplemental information such as case complexity and evaluator satisfaction with the rating scale may also be collected. Like the previous encounters, each resident evaluates their own performance immediately following the postoperative encounter.

Evaluation of Resident Confidence and Comments

Residents were surveyed at the beginning and conclusion of the three-day training module on their confidence in meeting the thirteen defined learning objectives (Table 2) on a 5-point Likert scale. Residents were also asked to provide written comments on their educational experience and suggestions for improvements to the curriculum. In addition, a luncheon session was hosted by the department on the final day of each module, which served as an informal forum for the residents to discuss their impressions and concerns regarding the curriculum with the simulation education team.

Statistical Analysis

Response data for all of the above evaluations were collected and entered into the STATA software package for analysis (StataCorp. 2013. Stata Statistical Software: Release 13. College Station, TX: StataCorp LP.). Each pre-training and post-training evaluation was analyzed with the Wilcoxon matched-pairs signed-rank test. Each criterion was tested individually, as were the pooled values for each evaluation scale. Spearman’s rank correlation was used to assess correlation between faculty and resident evaluations of the same encounter.

Cost

The cost associated with implementing each three-day event was estimated at $17,266, or an average cost of $2,877.67 per resident trained. Charges for exclusive use of simulation center facilities and support staff was the most significant cost at approximately $8,000. The cost of compensating faculty surgeons for their time teaching was a close second at an estimated at $7,200 based on a rate of $200.00 per hour. The cost of retaining the services of the standardized patient program was $1,632. Consumables, namely the porcine tissue were valued at $434. The above estimate does not include fixed costs such as capital costs to build and maintain the simulation center facilities with basic supplies (such as gloves and gowns), overhead costs such as employment of residents, faculty, and staff outside these three-day blocks, and the costs associated with the initial development effort invested by various stakeholders.

Results

18 first-year residents were assigned to the simulation curriculum over the 2013–2014 academic year. Due to scheduling considerations, 17 of the 18 residents were able to participate in the foregut surgery training module. The group was comprised of 7 categorical general surgery residents, 7 preliminary general surgery residents, and 3 urology residents in the general surgery program. There were 11 male and 6 female residents. All analyses and conclusions reported here are based on the data set for the 2013–2014 academic year.

Faculty Ratings of Resident Performance

Faculty rating of resident operative performance was significantly improved following the three-day module. Preoperative and postoperative performance did not significantly increase, though there was a trend for improvement. Median preoperative score pre- to post-training was unchanged at 4, p = 0.09. Median operative score was 3 (range 1 to 4) pre-training, and 3 (range 2 to 5) post-training, representing a significant increase at p<0.001. Median postoperative scores were unchanged at 6 out of 9, p = 0.5. Median scores and p-values for the Wilcoxon matched-pairs signed-rank test performed on individual criteria within the rating scales are shown in Tables 3–5.

Table 3.

Rating of resident preoperative performance (using CAMEO)

#	CAMEO Criterion	by Faculty Day 1 Median	by Faculty Day 3 Median	WSR p-value	by Resident Day 1 Median	by Resident Day 3 Median	WSR p-value
1	Case Difficulty	3	3	-	3	3	0.3173
2	Test Ordering and Understanding	3	3	0.0366	3	3	0.0656
3	Diagnostic Acumen	3	3	0.0845	3	3	0.0782
4	History Taking	3	3.5	0.8977	3	3	0.2568
5	Physical Examination	3	4	0.4391	3	3	0.0759
6	Communication Skills	3	3	0.2967	3	3	0.1655
7	Overall Performance	3	4	0.5650	2	3	0.0161
2–7	Pooled	4	4	0.0918	3	4	0.0000

Table 5.

Rating of resident postoperative performance (using Mini-CEX)

#	Question	by Faculty Day 1 Median	by Faculty Day 3 Median	WSR p-value	by Resident Day 1 Median	by Resident Day 3 Median	WSR p-value
1	Medical Interviewing Skills	6	6	0.5014	6	7.5	0.0599
2	Physical Examination Skills	6	5.5	0.4392	5	7	0.0493
3	Humanistic Qualities/Professionalism	7	7	0.4045	7	7.5	0.2270
4	Clinical Judgment	6	6	0.7704	6	7.5	0.2359
5	Counseling Skills	6	6.5	0.7146	7	7	0.3494
6	Organization/Efficiency	6	6	0.7324	6	7.5	0.1104
7	Overall Clinical Competence	6	6	0.9578	6	7	0.1151
1–7	Pooled	6	6	0.5166	6	7	0.0001

Resident Self-Ratings

Residents self-ratings of their own performance was significantly improved for all encounters assessed. Overall median scores for the preoperative encounter improved from 3 to 4, p<0.0001. Operative scores showed a significant (p<0.0001) increase from a median of 3 (range 1 to 3) to 3 (range 2 to 5). Median postoperative scores were similarly increased from 6 to 7, p=0.0001. Median scores and p-values for the Wilcoxon matched-pairs signed-rank test performed on individual criteria within the rating scales are shown in Table 3–5.

Resident Confidence

Residents were significantly more confident in meeting the defined learning objectives of the module. The overall median confidence score increased from 3 (range 1 to 5) pre-training to 4 (range 2 to 5) post-training (p <0.0001). Results for each specific learning objective are shown in Table 2.

Correlation Between Ratings of the Same Encounter

Analysis by Spearman’s rank correlation indicates there was very weak correlation between observing faculty and resident self-ratings. None of the encounters showed a significant level of correlation except for the postoperative ratings on the final day, with Spearman’s rho = 0.24 (p=0.047).

Discussion

Through a systematic design and implementation of a simulation-based curriculum, we were able to successfully provide a class of first-year surgery residents with an introduction to a broad range of topics in foregut surgery. Curricular accountability was maintained by sequential pre-training and post-training evaluation of performance over the course of a sequence of simulated patient encounters conducted in a realistic and immersive clinical setting.

Results from the first year of implementation of this curriculum indicate that statistically significant improvements in performance as assessed by attending faculty can be achieved over the course of a compact three-day educational module, and support its continued usage to educate surgery residents in foregut surgery. Resident self-evaluations of their own performance, as well as self-confidence in meeting learning objectives were significantly improved.

Sample size is a widespread limitation in residency education research. Only 1,210 general surgery residency positions were offered by 249 programs in the 2014 residency match, averaging just under 5 residents per program in the United States.^{15, 16} This places a significant limitation on achievable sample sizes for this type of research at any given institution. The University of Pennsylvania is among the larger programs in the country, with a quota of 7 categorical, 9 surgery preliminary, and 4 surgery/urology preliminary residents for a total first-year class numbering up to 20 residents. This size of resident class makes our program uniquely situated to conduct meaningful education research given the limited availability of subjects.

An important factor in the successful launch of this curriculum was the structure and oversight provided by the simulation program director and full-time simulation fellow. The program director made administrative arrangements in advance to schedule faculty and residents to participate, and ensure coverage of participating residents’ clinical duties.

Teaching at the simulation center is a significant time commitment for faculty. While the priority is to provide residents an engaging learning experience, it is also important to cater to the faculty teaching experience order to make this curriculum sustainable. The full-time fellow made the teaching experience turnkey – all necessary teaching materials and equipment were set up and ready for use at the simulation center, and the fellow was available to answer any questions regarding expectations for teaching.

Feedback from participating residents for the most part has been positive, with particular praise given to the opportunity to have direct learning interactions with attending faculty in an intimate, nonjudgmental small-group setting outside the usual pressures of the patient care environment. We have also received anecdotal accounts from residents who have completed the curriculum attributing their three-day educational experience to greater confidence and ease in the operating room when performing certain operative tasks (like a hand-sewn anastomosis) for the first time on a patient.

A number of challenges remain for broader implementation and sustainability of this curriculum. Feedback from residents is a concern for sustainability, as the reported popularity of fledgling educational initiatives can significantly influence the level of departmental support it receives on an ongoing basis. Qualitative evaluation of residents’ written comments revealed two issues that are consistently voiced as being unsatisfactory.

The first is the time residents spend away from their clinical teams. The residents are ‘pulled’ from their usual clinical duties for the three-day duration of the simulation module, and return to duty afterwards. While the absence is fully sanctioned by the department and supported by other providers who step in to take on the resident’s clinical responsibilities, the residents report anxiety regarding being out of touch with their team and status of patients on their service. This issue has been partially addressed by considerations in scheduling, by having modules run on the tail end of a work week (Wednesday-Friday) where possible to minimize interruptions. Scheduling simulation blocks at the ends of rotations would be another potential method to avoid disrupting continuity of care.

The second and perhaps more fundamental point of dissatisfaction expressed by residents is in regard to the time spent with standardized patients. Entry-level trainees are understandably most passionate about acquiring specific specialty skills in their chosen field. A number of surgery residents reported they feel standardized patient interaction lacks educational value, as they do not directly acquire surgical knowledge and skills by doing so. This may be in part due to residents having extensive prior exposure to standardized patients as part of their medical school training and preparation for licensing examinations. Such experiences may lead trainees to believe they have already achieved an acceptable level of proficiency with regard to this type of encounter and find further training to be redundant.

First-year surgery residents are not expected to be proficient independent operators, but are always expected to be attuned to their patients’ status and advocate for their care and safety. We therefore believe standardized patient interactions are a level-appropriate training tool, and place greater priority and emphasis on their use. While operative skill is a desirable characteristic of surgeons at any level, we firmly believe that these technical skills should not be removed from the context of patient care, especially early in one’s career. Training should address the comprehensive role of the surgeon in the care of a patient beyond the technical management of individual procedural events.³

Conclusions

A successful implementation of a simulation curriculum for general surgery residents was described which incorporated competency-based methods of evaluation. Our implementation begins to address societal expectations regarding accountability and reduction of patient risk in residency education, while balancing the stringent clinical service demands on residents and faculty. Residency programs seeking a practical implementation of simulation-based education within their institution may consider adopting our modular curriculum design. The three-day module template was implemented for other topics in surgery, and the structure should be broadly adaptable to other surgical or medical specialties.

Figure 2.

Faculty rating of resident performance during encounters.

Figure 3.

Self-rating of resident performance during encounters and confidence.

Table 4.

Rating of resident intraoperative performance (using OPRS)

#	Question	by Faculty Day 1 Median	by Faculty Day 3 Median	WSR p-value	by Resident Day 1 Median	by Resident Day 3 Median	WSR p-value
1	Respect for Tissue	3	3	0.7389	3	3	0.0842
2	Time and Motion	3	3	0.2699	3	3	0.1148
3	Instrument Handling	3	3	0.6589	3	3	0.1148
4	Knowledge of Instruments	3	3.5	0.0956	3	3	0.0842
5	Flow of Operation	3	4	0.0099	3	3	0.0788
6	Use of Assistants	3	3	1.0000	3	3	0.0960
7	Knowledge of Specific Procedure	3	4	0.0015	2	3	0.0196
1–7	Pooled	3	3	0.0003	3	3	0.0000