Development and incorporation of hybrid simulation OSCE into in-training examinations to assess multiple CanMEDS competencies in urologic trainees

Abstract

Introduction:

As residency training requirements increasingly emphasize a competency-based approach, novel tools to directly evaluate Canadian Medical Education Directives for Specialists (CanMEDS) competencies must be developed. Incorporating simulation allows residents to demonstrate knowledge and skills in a safe, standardized environment. We describe a novel hybrid simulation station for use in a urology resident in-training Objective Structured Clinical Exam (OSCE) to assess multiple CanMEDS competencies.

Methods:

An OSCE station was developed to assess Communicator, Health Advocate, Manager, and Medical Expert (including technical skills) CanMEDS roles. Residents interviewed a standardized patient, interacted with a nurse, performed flexible cystoscopy and attempted stent removal using a novel bladder/stent model. Communication was assessed using the Calgary-Cambridge Observational Guide, knowledge was assessed using a checklist, and technical skills were assessed using a previously validated global rating scale. Video debriefing allowed residents to review their performance. Face and discriminative validity were assessed, and feasibility was determined through qualitative post-examination interviews and cost analysis.

Results:

All 9 residents (postgraduate years [PGY] 3, 4, 5) completed the OSCE in 15 minutes. Communicator and knowledge scores were similar among all PGYs. Scores in technical skills were higher in PGY-5 compared with PGY-3/4 reside nts (mean score 79% vs. 73%). Residents and exam personnel felt the OSCE station allowed for realistic demonstration of competencies. Equipment cost was $218 for the exam station.

Conclusions:

We developed and implemented a hybrid simulation-based OSCE station to assess multiple CanMEDS roles. This approach was feasible and cost-effective; it also provided a framework for future development of similar OSCE stations to assess resident competencies across multiple domains.

Introduction

In recent years, medical educators and accrediting bodies have been increasingly focused on the principles of competency-based medical education (CMBE) to teach and evaluate residents. The Royal College of Physicians and Surgeons of Canada has developed the CanMEDS framework of competencies that includes 7 roles: Medical Expert, Communicator, Collaborator, Manager, Health Advocate, Scholar, and Professional.¹ Resident trainees must acquire and develop skills in each of these domains, and educators are required to base their evaluation on resident demonstration of competency in each role.

Most traditional evaluation methods focus on the Medical Expert role and the other 6 roles are less widely developed and implemented. In addition, current evaluation methods often involve assessing individual competencies in isolation. This approach does not necessarily reflect performance in the real-world clinical environment, where physicians must demonstrate multiple competencies simultaneously during individual patient encounters. Simulation recreates a realistic clinical environment or circumstance to allow the learner to undertake a specific task in a safe, controlled manner. Simulation has often been used successfully in residency programs for training purposes.^2–6 Simulation can refer to relatively simple scenarios, such as using a low fidelity task trainer to teach suturing or a clinical interview involving a standardized patient. Hybrid simulations involve combining 2 or more forms of simulation (e.g., a high-fidelity computerized simulator with a standardized patient) to recreate real-world complex scenarios that require the trainee to use multiple skills concurrently.

Objective Structured Clinical Exams (OSCEs) provide a reliable and standardized method for assessing resident performance through direct observation, and can focus on specific areas or skills.⁷ Using simulation within OSCE scenarios combines the benefits of each assessment method and allows for assessment of not only Medical Expert and Technical Skills roles, but also the other less easily tested competencies.

We developed a novel hybrid simulation station for use during a urology in-training OSCE that allows us to assess multiple CanMEDS competencies, including Communicator, Health Advocate, Manager and Medical Expert/Technical Skills.

Methods

An OSCE station was developed to assess Communicator, Health Advocate, Manager, and Medical Expert (including Technical Skills) CanMEDS roles. Residents interviewed a standardized patient, interacted with a nurse, performed flexible cystoscopy and attempted stent removal using a novel bladder/stent model. Communication was assessed using the Calgary-Cambridge Observational Guide,⁸ knowledge was assessed using a checklist, and technical skills were assessed using a previously validated global rating scale (GRS).⁹ Video debriefing allowed residents to review their performance. Face and discriminative validity were assessed, and feasibility was determined through qualitative post-examination interviews and cost analysis.

The setting

The hybrid simulation OSCE station was carried out at the University of Ottawa Skills and Simulation Centre (uOSSC). The uOSSC is a state-of-the-art simulation centre that houses 4 high-fidelity simulation rooms, including 2 operating rooms that are built to clinical standards to ensure realism (Fig. 1). The simulation rooms are equipped with multiple cameras and microphones to capture and allow playback for video debriefing of candidates and a 1-way window to allow for direct observation of trainees during simulation scenarios and examinations. Evaluators are able to communicate with participating exam personnel (nursing confederates and standardized patients) via concealed earpieces. Patient vital signs (heart rate and blood pressure) were simulated and manipulated during the OSCE by the evaluators through the use of SimMan (Laerdal Medical Canada Ltd, Toronto, Canada) concealed in the corner of the operating room (Fig. 2). Using an Olympus digital cystoscope (Olympus Canada Inc., Richmond Hill, ON) and video monitoring system allowed us to assess resident technical performance of cystoscopy and stent manipulation.

Fig. 1.

High-fidelity simulation operating room.

Fig. 2.

Monitor displaying operating room, digital cystoscope image and SimMan vital signs.

The scenario

The residents were provided with written instructions outside the operating room, instructing them to assess a female patient who presents to the outpatient cystoscopy suite for removal of a ureteral stent following extracorporeal shock wave lithotripsy (ESWL). Upon entering the room, residents were presented with a draped female standardized patient in lithotomy position, accompanied by a circulating nurse confederate. The patient’s health record was on a stand beside the patient, and included a copy of the most recent outpatient clinic note outlining the course of treatment, an unsigned patient consent form and a post-ESWL radiograph depicting radiopaque encrustation on the proximal coil of the ureteral stent. The standardized patient received the following instructions:

Three months ago you presented with right flank pain and fever. You were found to have a 1.5-cm right kidney stone on the computed tomography (CT) scan. Your initial treatment included intravenous antibiotics and placement of a right ureteral stent. You have had 2 separate ESWL treatments (stone-blasting), 2 months ago and 1 month ago. Recently, you have been experiencing increased blood in the urine and urinary frequency. If asked by the resident, you have just come from the x-ray department where you had an x-ray taken to check on your stone after your last ESWL treatment. As the procedure progresses, the stent will not be able to be removed by the resident. You will become anxious and start complaining of pain in the bladder and right flank as the resident tries to extract the stent.

Hybrid simulation model

After interviewing the standardized patient, residents were required to interact with a nurse and perform a flexible cystoscopy and stent removal using a flexible cystoscope and graspers. An encrusted stent and bladder model was created using a cored-out papaya and standard double-J ureteral stent (Boston Scientific, MA) (Fig. 3a). The stent was secured to the outside of the papaya using staples to mimic encrustation and prevent removal of the stent (Fig. 3b). The bladder model was placed inside a rubber pelvic model (Fig. 3c). The entire unit was then placed under the drapes next to the standardized patient (Fig. 4).

Fig. 3a.

Papaya simulating urethra and bladder.

Fig. 3b.

Ureteral stent secured to outside with staples to prevent removal.

Fig. 3c.

Papaya/stent model in a female pelvic model.

Fig. 4.

Hybrid model consisting of part-task trainer and standardized patient. SimMan is covered in a corner of the operating room.

Evaluation and analysis

Evaluation of resident performance and simultaneous directing of the OSCE (SimMan vitals, instructions to nursing personnel and standardized patient) was done by 2 evaluators concurrently behind the one-way glass. Communication was assessed using the Calgary Cambridge Observation Guide. Technical skills in performing flexible cystoscopy and stent manipulation were evaluated using a previously-validated GRS.⁹ A scoring sheet and checklist were used to assess the CanMEDS roles of (1) Medical Expert (e.g., obtaining pertinent facts in clinical history); (2) Manager (e.g., leading a pre-procedural checklist and ensuring that informed consent was properly obtained); and (3) Health Advocate (e.g., recognizing an encrusted, retained stent and terminating the procedure to prevent further patient distress).

Resident satisfaction with the OSCE was assessed using a brief post-exam qualitative interview and a 9-item questionnaire. Residents met individually with the examiners the week following the exam for a video debriefing session that allowed for further resident insight into their performance. At this session, overall results and technical performance were reviewed, with special attention to communication skills used during the station.

Results

Nine urology residents, postgraduate year- (PGY) 3 through PGY-5, participated in the exam as part of the OSCE portion of the semi-annual in-service examination. The scenario was easily set up and executed in the allocated 15-minute time slot.

Overall scores for the Communicator, Medical Expert (knowledge), Manager and Health Advocate roles were similar between all resident levels. Scores on the GRS used to assess technical skills tended to be higher in PGY-5 residents (mean score 79%) compared to PGY-3/4 residents (mean score 73%).

The papaya bladder model performed well. Flexible cystoscopy was performed without the need for irrigation, and visualization was excellent. The papaya required replacement once during the exam due to degradation of the fruit from repeated cystoscopy attempts. Disposable equipment cost was $218 (2 papayas at $4, 2 double-J stents at $214). There were no facility fees above those for the OSCE, and both the nurse and standardized patient were volunteers.

Based on the post-exam questionnaire, residents felt that the scenario, standardized patient and bladder model were realistic (89%) and agreed that the station allowed them to reasonably demonstrate their CanMEDS competencies (100%) and technical skills (83%). All of the residents (100%) agreed that the simulation station added an important dimension to the traditional oral and written components of the In-Training examination. In addition, residents felt that the video debriefing allowed for further review of, and discussion of, their performance with the evaluators.

Discussion

The initial CanMEDS initiative, formally approved by the Royal College of Physicians and Surgeons of Canada in 1996, introduced a comprehensive framework of competencies designed to improve patient care by enhancing physician training and education.¹ The CanMEDS 2015 project is currently in development and seeks to enhance the existing CanMEDS roles as part of the Competence by Design (CBD) initiative of the Royal College, a major, multi-year project to implement an enhanced model for competency-based medical education (CBME) in residency training and specialty practice in Canada.

Simulation can be an important complement to traditional methods of medical education and may garner greater attention in CBME frameworks.10 Simulation recreates a realistic clinical environment or circumstance to allow the learner to undertake a specific task in a safe, controlled manner, without compromising the safety of real patients. Furthermore, simulation also permits educators to emphasize work-based learning in a way that integrates multiple competencies (e.g., teaches the CanMEDS Roles collectively).¹¹ A recent review suggests that simulation is well-suited to CBME frameworks.¹²

In our study, we successfully developed and implemented a novel hybrid simulation OSCE station to assess multiple CanMEDS roles, including technical skills. The station is unique in that it combined a low-fidelity part-task trainer (papaya bladder model), a high-fidelity, computer driven mannequin to display physiologic parameters (e.g., vital signs) during the procedure, and a live standardized patient who interacted with the resident in real-time. This situation allows residents to use multiple competencies concurrently. Evaluating trainees in this type of realistic simulated environment is desirable and likely leads to a more valid assessment of how they will perform in practice.

For resident evaluation we used several previously validated assessment tools, in addition to a typical checklist-style answer key, which we developed specifically for this scenario. In our study, while scores on most assessment metrics (e.g., communication, medical knowledge) were similar among all residents, the GRS used to assess technical skills demonstrated a trend towards higher scores in more senior residents. This finding may reflect the fact that proficiency in some CanMEDS roles develops earlier in training (e.g., Communicator) as compared to others (e.g., technical skills acquisition), but does begin to lend support the discriminative validity of the station.

In terms of face validity, post-exam debriefing and interviews with all residents and all exam participants (examiners, SPs and nurse confederates) revealed that the scenario was realistic. Residents felt that they were able to adequately demonstrate their knowledge and clinical skills, and that the hybrid cystoscopy model represented a good simulation of a real-world procedure.

We found that the development and implementation of our novel OSCE station was feasible given the resources available at our institution. Financial cost was low ($218 for all equipment) and both papayas and double J stents were readily available. We did however benefit from no-cost access to the simulation centre and volunteer standardized patients/nurse confederates that led to significant cost-savings. This service may not be available at all urology training centres and could add significant financial cost to the running of the station.

Overall, our approach to the development of the simulation-based OSCE station was feasible, of relatively low cost, and appears to represent a valid assessment of resident performance in the particular scenario examined. Although these OSCEs can be resource-intensive, such stations provide a unique opportunity to observe and evaluate residents. Furthermore, feedback from resident trainees demonstrates that such assessments provide valuable insights into their development.

Conclusion

The development and implementation of hybrid simulation-based OSCEs into residency training programs allow for real-life assessment of multiple CanMEDS competencies. As CMBE continues to replace time-based medical training, simulation will fill specific gaps in resident evaluation. Future incorporation of other subspecialties into our scenario (e.g., anesthesiology) may further enhance the realism of such simulation scenarios. Further development of clinical scenarios that can be shared with other urological programs and institutions will be a focus of future research. Lastly, this model of hybrid simulation-based OSCE station can serve as a framework for the development of additional evaluation scenarios for resident trainees both in urology and in other specialties.