Simulation-Based Education with Mastery Learning Improves Paracentesis Skills

Abstract

Background

Paracentesis is a commonly performed bedside procedure that has the potential for serious complications. Therefore, simulation-based education for paracentesis is valuable for clinicians.

Objective

To assess internal medicine residents' procedural skills before and after simulation-based mastery learning on a paracentesis simulator.

Methods

A team with expertise in simulation and procedural skills developed and created a high fidelity, ultrasound-compatible paracentesis simulator. Fifty-eight first-year internal medicine residents completed a mastery learning-based intervention using the paracentesis simulator. Residents underwent baseline skill assessment (pretest) using a 25-item checklist. Residents completed a posttest after a 3-hour education session featuring a demonstration of the procedure, deliberate practice, ultrasound training, and feedback. All residents were expected to meet or exceed a minimum passing score (MPS) at posttest, the key feature of mastery learning. We compared pretest and posttest checklist scores to evaluate the effect of the educational intervention. Residents rated the training sessions.

Results

Residents' paracentesis skills improved from an average pretest score of 33.0% (SD  =  15.2%) to 92.7% (SD  =  5.4%) at posttest (P < .001). After the training intervention, all residents met or exceeded the MPS. The training sessions and realism of the simulation were rated highly by learners.

Conclusion

This study demonstrates the ability of a paracentesis simulator to significantly improve procedural competence.

Editor's Note: The online version of this article contains a physician survey ^{(47.5KB, doc)} about the paracentesis simulator and the 25-item paracentesis skills checklist.

What was known

Paracentesis is a commonly performed bedside procedure with the potential for complications. Simulation-based procedural training is beneficial for learners.

What is new

Internal medicine residents who participated in a 3-hour session with deliberate practice and feedback improved and all achieved a minimum passing score.

Limitations

Small sample, single site, and single specialty.

Bottom line

Residents′ paracentesis skills improved significantly after a simulation and feedback session that is feasible for most programs.

Introduction

Internal medicine residents and gastroenterology fellows commonly perform paracentesis procedures. The American Board of Internal Medicine (ABIM) requires residents to be familiar with the indications, complications, and interpretation of this procedure.¹ Recently, the Accreditation Council for Graduate Medical Education added paracentesis skills as a 2012 program requirement for gastroenterology fellowship training as well as a requirement that fellows have access to simulation training.² Despite these recommendations, many procedures are performed by learners who are not appropriately proficient or confident,^3,4 and there is variable adherence to the gastroenterology core curriculum regarding hepatology in many programs.⁵

Simulation-based mastery learning⁶ featuring deliberate practice⁷ gives residents and fellows the opportunity for procedural skills development and feedback. Mastery learning requires that learners meet or exceed a minimum passing score (MPS) on a simulated examination prior to performing the procedure in actual clinical care.⁶ We previously used simulation-based mastery learning to improve skills of residents and fellows in thoracentesis,⁸ Advanced Cardiac Life Support,⁹ and central venous catheter (CVC) insertion.^10–12

We hypothesized that the mastery model of procedural training could be successfully extended to paracentesis skills on an ultrasound-compatible simulator that was designed and created at our institution.

Methods

Simulator Design and Evaluation

From May to August 2008, various materials and strategies were tested to create an ultrasound-compatible paracentesis model. A prototype was produced and tested to obtain feedback and suggested modifications. Ten identical simulators were pilot tested at a national procedure skills conference held at Northwestern University in October 2009. Twenty-one practicing physicians who participated in the conference tested the simulator and provided feedback about the simulator's realism (provided as online supplemental material, Appendix A).

The paracentesis simulator represents the human abdomen from rib margin to pubis. A foam chassis contains a “cartridge” containing simulated liver, kidneys, pancreas, aorta, intestines, and ascitic fluid. The chassis and cartridge are covered with an ultrasound-compatible silicone-based “skin” which is affixed to the chassis. Realistic ultrasound images can be obtained (figure 1), and a paracentesis can be performed (figure 2). Research and development of the simulator required partial effort of 2 part-time staff to complete. The estimated cost of disposable supplies was $1500 and labor cost $9216. The final cost of materials for each simulator was $57.

FIGURE 1.

Ultrasound Image of the Paracentesis Simulator Demonstrating Anatomically Correct Appearance of Internal Organs

FIGURE 2.

Paracentesis Procedure Using the Simulator

Use of the Paracentesis Simulator in Resident Education

An educational intervention was performed within the internal medicine residency at Northwestern University from July to September 2010. Participants were all 58 postgraduate year-one (PGY-1) internal medicine residents. The Northwestern University Institutional Review Board approved the study, and all participants provided informed consent.

PGY-1 residents received standardized simulation-based education regarding paracentesis. Two faculty members supervised educational sessions for a total of 5 days. Residents underwent baseline skill assessment (pretest) on the simulator, using a 25-item checklist (provided as online supplemental material, Appendix B). The checklist was developed by one author (J.H.B.) using evidence-based guidelines for paracentesis^13,14 and checklist design.^8–12,15,16 The checklist was reviewed for completeness by 2 other authors with expertise and experience in paracentesis, simulation-based education, and checklist design (J.A.V. and D.B.W.).

After undergoing pretesting, all study residents completed a 3-hour educational session featuring a lecture, the New England Journal of Medicine's paracentesis video,¹⁴ ultrasound training, and deliberate practice with the paracentesis simulator with directed feedback.⁷

Following training, residents were retested (posttest) using the 25-item checklist and were expected to meet or exceed the MPS for simulated paracentesis procedures. Residents who failed to meet the MPS the underwent additional deliberate practice until they were able to meet the MPS. The MPS for the paracentesis clinical skills examination was determined by 9 clinical experts (ABIM-certified physicians, 6 physicians from internal medicine, and 1 physician each from gastroenterology, pulmonary/critical care, and infectious disease) using the Angoff and Hofstee standard setting methods.^17–19

Pretest and posttest scores were compared to assess the impact of the educational intervention. Residents responded to a questionnaire about their experience with the simulator and the training sessions after the intervention.

Measurement

Demographic data were obtained from participants including age, sex, United States Medical Licensing Examination (USMLE) Step 1 and 2 scores, and prior clinical experience with paracentesis procedures. All pretests and posttests were graded by a single unblinded instructor (E.R.C.) and were videotaped. A dichotomous scoring scale (done correctly or incorrectly) was used for each item.

A faculty instructor with expertise in scoring clinical skills examinations, who was blinded to pretest and posttest status of each examinee (J.H.B.), rescored a 50% random sample of the videotaped examinations to assess interrater reliability. Course evaluation questionnaires have been previously published,^8,9,12 and answers were recorded on a 5-point Likert scale (where 1  =  strongly disagree, and 5  =  strongly agree).

Statistical Analysis

The Cohen kappa coefficient was used to assess interrater reliability. Pretest and posttest checklist scores were compared using a paired t-test. Multiple linear regression was used to assess the association of posttest score with pretest performance, age, sex, USMLE score, and clinical experience. Analyses were performed using PASW version 18.0 statistical software (SPSS, Inc., Chicago, IL).

Results

All 58 PGY-1 residents consented to participate and completed the entire protocol (table 1).

TABLE 1.

Demographic Data of PGY1 Residents Undergoing Paracentesis Simulation Training

The MPS for the skills checklist was set at 83%. Interobserver agreement for the educational intervention skills checklist was high (κ_n  =  .87). Paracentesis checklist skills improved from an average pretest score of 8.25/25 (33.0%; SD  =  15.2%) to 23.2/25 (92.7%; SD  =  5.4%) at posttest (P < .001) after the training intervention (figure 3).

FIGURE 3.

Pretest and Posttest Paracentesis Skill Scores on the 25-Item Checklist SEM  =  standard error of measurement; MPS  =  minimum passing score

Two residents (3%) failed to meet the MPS at posttest. They subsequently underwent additional deliberate practice and were retested. Both residents reached the MPS within 1 hour of further practice.

Regression analysis showed no associations between posttest scores and pretest performance, age, sex, USMLE Step 1 and 2 scores, or prior clinical experience with paracentesis procedures. Residents rated the training sessions highly (table 2).

TABLE 2.

Residents' Post-training Questionnaire Results

Discussion

To our knowledge, this is the first study to use the mastery learning model for simulation-based paracentesis education. Our data demonstrate significant improvement in paracentesis skills of PGY-1 residents after simulation-based education. The magnitude of skill improvement using the paracentesis simulator was similar to that shown in earlier studies with commercial-grade simulators for thoracentesis,⁸ Advanced Cardiac Life Support,^9,15 and CVC insertion.^10–12 These interventions have also been linked to improved patient care quality.^10,11,20,21 Our results also support the policy of the ABIM¹ which states that simulation should be used to verify trainees′ skill before performing invasive procedures in actual patient care.

This study contributes to what is known about paracentesis procedural skill acquisition among medical learners. When assessed rigorously, there was no association among demographic variables such as age, sex, or USMLE examination scores, and paracentesis clinical skills. Additionally, our study found no relationship between the number of paracentesis procedures performed in actual clinical care and measured procedural skills. However, a difference may have been unlikely, as most residents had previously performed 0–1 procedures. Past research on thoracentesis,⁸ CVC insertion,^10,11 and endoscopic balloon dilation of the esophagus²² also shows no association between the number of procedures performed and procedural skill. This is problematic because a 2003 Core Curriculum developed by the Gastroenterology Leadership Council for gastroenterology fellows recommends that fellows demonstrate competence in paracentesis by performing at least 20 procedures.²³ Clinical experience is not a proxy for skill,²⁴ and our findings are a reminder that competence should not be assumed after a set number of procedures.

This study has several limitations. The tools used to create a “homemade” simulator and the faculty commitment required for simulation training may not be available at other institutions. Training programs should consider the benefits of simulation as well as new training requirements when planning educational interventions. The degree of improvement in posttest scores recorded in our study may be reduced because our paracentesis model was used for both education and testing and we did not use a true “control” group. However, this should not diminish the pronounced impact simulation-based training had on paracentesis skills of first-year residents. Additionally, post-testing was performed immediately after training, thereby enhancing recall of procedural steps. In the service of translational science goals,²⁵ we have not yet linked improved paracentesis skills in the simulated environment to improved health care delivery (higher procedural success rates, lower referral rate to interventional radiology) or better patient outcomes (lower complication rates).

Conclusion

This study demonstrates the ability of simulation-based education to improve paracentesis skills in internal medicine residents. Use of simulation as an adjunct to clinical training helps students meet residency and fellowship accreditation requirements and is feasible and well accepted by learners. Further study should address the effect of training on the long-term retention of paracentesis skills and health care outcomes.