Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2020 Aug 17.
Published in final edited form as: Clin Simul Nurs. 2018 Nov 27;27:39–47.e4. doi: 10.1016/j.ecns.2018.10.005

The Effect of Judge Selection on Standard Setting Using the Mastery Angoff Method during Development of a Ventricular Assist Device Self-Care Curriculum

Jeffrey H Barsuk 1,2, Rebecca S Harap 3, Elaine R Cohen 1, Kenzie A Cameron 1, Kathleen L Grady 1,4, Jane E Wilcox 1,3, Kerry B Shanklin 3, Diane B Wayne 1,2
PMCID: PMC7430939  NIHMSID: NIHMS1509917  PMID: 32818046

Abstract

Background:

Patients and caregivers need to perform ventricular assist device (VAD) self-care safely to help prevent complications (e.g., infection). We developed a VAD self-care simulation-based mastery learning (SBML) curriculum. We determined optimal minimum passing scores (MPSs) and evaluated effects of judge selection.

Methods:

A multidisciplinary team created a VAD self-care SBML curriculum including simulated skills and knowledge examinations. Patients, caregivers, VAD coordinators, and physicians were expert judges who determined MPSs using the Mastery Angoff method.

Results:

MPSs for the skills and knowledge examinations were high (range = 94–99% and 97% correct), respectively. Judges closely agreed on MPSs.

Conclusions:

Stakeholders set stringent MPSs for high-stakes VAD self-care.

Keywords: Simulation, Mastery Learning, Standard Setting, Mastery Angoff, Self-care, Ventricular Assist Device

BACKGROUND

Patients and their caregivers are often required to possess disease- and treatment-specific knowledge and skills to provide self-care to complete their healing process, improve outcomes, and remain healthy. For example, patients or their caregiver may need to perform wound or central line dressing changes, administer home intravenous antibiotics or total parenteral nutrition, perform peritoneal dialysis, or care for a ventricular assist device (VAD). VADs are mechanical heart pumps that are implanted into a patient’s left and/or right ventricle. An electrical cord (driveline) exits from the pump through the abdomen and is attached to a small computerized controller (that controls the actions of the pump), which is connected to a power source (i.e., batteries or AC power; Figure 1). VAD self-care, in particular, requires a high level of knowledge and meticulous skill performance by patients and caregivers. Adverse events related to VADs include driveline infections (1.31 per 100 patient months during the first 3 months post-implant) and strokes which lead to a high readmission rate (O’Horo et al., 2018; Kirklin et al., 2017). The overall mortality of patients with a VAD is 18% at one year (Kirklin et al., 2017). Patients and their caregivers must be able to change the dressing at the driveline exit site using sterile technique, change the controller if it malfunctions, change power sources (e.g., changing from batteries to wall power and back, and charging and changing batteries), learn new medications, troubleshoot controller alarms, and overall adjust to a new daily lifestyle to help prevent these adverse events (Goldstein et al., 2012; Kirklin et al., 2017; Yarboro et al., 2014).

Figure 1.

Figure 1.

Open in a new tab

Food and Drug Administration Approved Ventricular Assist Devices. A) HeartWare™ VAD Reproduced with Permission of Medtronic from https://www.heartware.com/resources. B) HeartMate II™ VAD Reproduced with Permission of Abbott Vascular from http://www.thoratec.com/about-us/media-room/library.aspx.

Traditionally, this information and skill set is taught by nurses or advanced practice providers during a patient’s admission for VAD implant surgery. However, this training may be hurried and sporadic due to time and availability constraints of both trainees (patient and caregiver) and trainers. Furthermore, information and skills learned are not standardized across institutions and rigorous measurement of patient and caregiver knowledge and skill is usually not performed. Therefore, when a patient is discharged from the hospital, it is largely unknown whether the patient and caregiver have acquired the appropriate knowledge and skills for a safe transition home.

A recent article in the Journal of the American Medical Association called for use of simulation in patient self-management training, noting that simulation involves “learning by doing through practice, rehearsal, and role playing” (Coleman, 2014). Despite this clear preference for simulation-based education, we found only two articles using simulation to train chronically ill patients/caregivers in self-care skills. In the first study, caregivers reported being more confident after they were trained to perform home ventilator tasks using simulators (Tofil et al., 2013). In the second study, patients about to undergo coronary artery bypass surgery received either a simulation-based educational intervention or usual educational techniques about what to expect post-operatively (Haupt & Beauvais, 2014). Patients who completed simulation-based education had greater knowledge and lower anxiety than patients who received usual care.

Simulation-based mastery learning (SBML) is an educational model that uses deliberate practice with simulators to train healthcare providers to a high standard and improve patient safety outcomes (Barsuk, Cohen, Feinglass, McGaghie, & Wayne, 2009, 2013; Barsuk et al., 2018b; Barsuk, McGaghie, Cohen, O’Leary, & Wayne, 2009; Schwab, Teitelbaum, Barsuk, Soper, & Hungness, 2018). In SBML, all trainees must meet or exceed a minimum passing score (MPS) before completion of training. Those who do not meet this standard participate in further deliberate practice and retesting until they meet the MPS. In a meta-analysis of 82 studies, educational outcomes from simulation with mastery learning were superior to those from non-mastery simulation training (Cook, Brydges, Zendejas, Hamstra & Hatala, 2013). SBML has been used to train nurses, medical students, residents, fellows, and attending physicians (Barsuk, Ahya, Cohen, McGaghie, & Wayne, 2009; Barsuk et al., 2015; Barsuk et al., 2016; Barsuk et al., 2012b; Butter, McGaghie, Cohen, Kaye, & Wayne, 2010). However, SBML has never been used to train patients or caregivers, despite a wide body of literature on the benefits of SBML on healthcare professional practice and patient outcomes.

Setting an MPS is a critical component of creating an SBML curriculum (Downing, Tekian, & Yudkowsky, 2006; Yudkowsky, Park, Lineberry, Knox, & Ritter, 2015). Traditionally, the average of the Angoff and Hofstee standard setting methods have been used to set MPSs for SBML (Barsuk et al., 2012a; Barsuk et al., 2012b; Barsuk, McGaghie, Cohen, Balachandran, & Wayne, 2009; Wayne, Barsuk, O’Leary, Fudala, & McGaghie, 2008; Wayne et al., 2006). In the Angoff method, expert judges are asked to review each test item and estimate the percentage of borderline or minimally competent trainees (i.e., one who has a 50–50 chance of success) who would perform each item correctly after training. The Hofstee method is whole-test based and asks judges about the minimum and maximum acceptable passing scores and failure rates. However, these methods may not be appropriate when patient safety is a major concern. Recent studies argue that in a mastery learning environment, judges should not be asked about “borderline” trainees but instead should consider the performance of a trainee who is well prepared to perform care safely and independently (Barsuk, Cohen, Wayne, McGaghie, & Yudkowsky, 2018; Yudkowsky et al., 2015; Yudkowsky, Tumuluru, Casey, Herlich, & Ledonne, 2014). Therefore, the Mastery Angoff method was developed, in which judges rate each test item based on the percentage of “well-prepared” (i.e., one who can perform the task independently and safely) trainees who would perform each item correctly after training.

Although patients and caregivers are important partners in patient care, SBML has not been used to either teach or boost skills among patients or caregivers, and use of patients or caregivers as expert judges has only occurred on a limited basis in the medical literature (Wayne, Cohen, Makoul, & McGaghie, 2008). Therefore, given the critical nature of adherence to a VAD care regimen after discharge from the hospital, we developed and tested an SBML intervention to improve VAD self-care. In this manuscript, we (a) describe the development of a VAD self-care SBML curriculum for patients and caregivers; (b) set MPSs needed for SBML using the Mastery Angoff technique including patients, caregivers, VAD coordinators (nurses and advanced practice providers) and physicians as expert judges, and (c) evaluate and compare the stringency of MPSs set by each group of expert judges.

METHODS

We created a VAD self-care SBML curriculum at Northwestern Memorial Hospital (NMH), a tertiary care academic medical center in downtown Chicago, Illinois, US. Advanced heart failure providers at NMH implant approximately 60 VADs and care for about 140 patients with VADs annually. Curriculum development and standard setting exercises engaged groups of VAD patients and caregivers, VAD coordinators, and physicians. We compared standard setting results between groups. The Northwestern University Institutional Review Board approved this study (STU00203567 and STU00203530), and all participants provided written informed consent.

Curriculum Development

We created a VAD self-care SBML curriculum for two FDA-approved VADs, HeartMate™ and HeartWare™, using the Kern and Thomas framework (Thomas, Kern, Hughes, & Chen, 2016) which describes a method for developing medical educational curricula. We conducted 46 semi-structured interviews with 16 patients, 12 caregivers, seven VAD coordinators, and 11 physicians to identify important factors in VAD self-care. Factors identified for inclusion in the curriculum were: critical learning content (e.g., dressing and controller changes, power source changes, medications, and controller alarms), training modalities such as wearable simulators, educational videos, reminder cards (e.g., depictions of known controller alarms, and brief descriptions of what the alarms mean, and what to do in an emergency), and refresher training sessions. We created simulated skills assessments for dressing, controller, and power source changes while written examinations were created to assess knowledge about topics such as medications, controller alarms, and what to do in emergencies.

The objective of our mastery learning curriculum was that all patients and caregivers would meet or exceed the MPSs on the simulated skills and written examinations before completing training. We created educational videos for dressing and controller changes. We did not create a video on changing power sources because interviewees stated that this task was straightforward. Reminder cards were created to help reinforce knowledge of controller alarms and VAD-related items needed for travel as well as important phone numbers and what to do in an emergency. We created assessment checklists for each of dressing, controller, and power source change for scoring pre- and post-skills exams. The checklists included dichotomous scoring, in which raters select one option if a skill was performed correctly, or the second option if the skill was performed incorrectly/not done. These assessments were developed using the checklist development guidelines of Stufflebeam (Stufflebeam, 2000) as well as NMH policies, manufacturers’ published instructions for use, and relevant literature (Menon et al., 2015; Pereda & Conte, 2011). Six VAD coordinators, an advanced heart failure physician and an expert in checklist design reviewed the checklists. Final versions were created using the modified Delphi method. Therefore, there were three checklists to assess skills for the HeartMate™ device, and three checklists for the HeartWare™ device. Dressing change skills assessments were identical for both HeartMate™ and HeartWare™. However, controller and power source change assessments were unique to each device to reflect their unique characteristics. The checklists for the HeartMate™ and HeartWare™ devices can be found in the Appendix.

We also developed two unique post-training written examinations (HeartMate™ and HeartWare™) with key knowledge components including medication management, controller alarms, and what to do in an emergency. The post-training written examinations were adapted from the manufacturers’ published examination materials which can be found on their respective websites (HeartWare, 2012; Thoratec, n.d.). The written examinations were amended to include more questions on medication management. All exam questions were written in language at an eighth grade reading level (Paniagua & Swygert, 2016). Six VAD coordinators, an advanced heart failure physician, an expert in written examination design, and an expert in communication skills created and reviewed the written examinations. Final versions were created using the modified Delphi method. The written examinations include multiple choice, fill in the blank, true/false, and matching questions, and are administered as an “open book” exam.

The first component of the VAD self-care SBML curriculum is a simulation-based pretest on the three self-management skills assessed using the developed checklists for dressing changes (caregiver only), controller changes, and power source changes. At the beginning of training and before pretesting, the instructor reads a “briefing” to patients and caregivers. The briefing explains the intent to create a safe learning environment to maximize education outcomes. The briefing also orients patients and caregivers to equipment including the simulator. Instructors provide debriefing and feedback to all learners immediately after pretesting using the methods described by Rudolph (Rudolph, Simon, Rivard, Dufresne, & Raemer, 2007). After completion of pretesting, patients and caregivers watch the dressing and controller change videos. A trained instructor reviews medications, controller alarm functions, and what to do in emergency situations with learners. Patients and caregivers also complete deliberate practice (repetitive, focused skills practice with expert feedback) of dressing, controller, and power source changes using a simulator (Ericsson, 2015; Ericsson, 2016). Total time needed for deliberate practice is determined by both the instructor and patient or caregiver, based on progress, but each skill needs to be fully practiced at least three times. After 2 to 3 hours of deliberate practice and education, participants complete skills posttests and the written knowledge examination. An instructor again reads a briefing to patients and caregivers before completing posttests and debriefing with feedback is performed after testing.

Patients and their caregivers use a wearable simulator (Figure 2) made for this study by Simulab Co, Seattle, Washington for all skills testing and training sessions. Patients and caregivers are trained together when possible. Training may occur separately due to patients’ clinical status and need for testing. All training and testing sessions are standardized and performed by two VAD coordinators (RH and KS) who were trained in debriefing and calibrated as raters. These coordinators use the same simulator and supplies for all sessions which occur either at the patient’s bedside or in a nearby conference room.

Figure 2.

Figure 2.

Open in a new tab

Wearable Ventricular Assist Device (VAD) Simulator with Connected HeartWare™ VAD.

In the mastery model, participants must meet or exceed the predetermined MPS on each skill assessment as well as the written examination. Participants who do not meet or exceed the MPS complete additional deliberate practice of the simulated skill and retest until the MPS is achieved. Reassessment/retraining is a Joint Commission requirement for VAD self-care tasks within the year post implantation (The Joint Commission, 2018). To comply with this requirement, the three skills, dressing, controller and power source changes, are reassessed at one and three months after initial posttest, and retraining occurs if learners are unable to meet or exceed the MPSs.

Measurement

An expert panel of 36 judges including 12 VAD patients, 12 caregivers, six VAD coordinators, and six physicians reviewed the skills checklists and written examinations and used the Mastery Angoff standard setting method to set MPSs. Patients and caregivers were divided into equal groups for each of the VADs (i.e., HeartMate™ or HeartWare™), depending on which VAD the patient received at implant. VAD coordinators and physicians served as expert judges for both VAD types due to their clinical expertise caring for each device. Additionally, the six VAD coordinators and six physicians who served as expert judges also participated in the semi-structured interviews. However, the patient and caregiver judges consisted of a new cohort who did not participate in the interviews. We collected demographic and clinical information about the judges, including education for patients and caregivers, and clinical experience and training for physicians and nurses. Expert judges were trained immediately before the standard setting sessions using the methods described by Norcini and Guille (Kane, 1994; Norcini & Guille, 2002). This training involved: (a) defining the qualities of the assessments and who are the examinees; (b) educating the judges about the consequences of their decisions in terms of patient safety; (c) discussing the purpose of the skills and written evaluations and what represents appropriate performance; (d) defining the learner well prepared to succeed; and (e) group practice, feedback, and discussion. An author (JHB) who created the checklists and written examinations and with extensive expertise in standard setting exercises led the sessions. We held a total of 14 standard setting sessions: one for each patient and caregiver pair (total 12), one for VAD coordinators, and one for physicians. Groups discussed the first three assessment items on the dressing change checklist, and further collaborative discussion was held as needed during the remainder of the standard setting sessions. We compared and contrasted responses from patients, caregivers, VAD coordinators, and physicians.

Analysis

We created a final MPS by averaging scores from all of the judges for each skills assessment and written examination as required by the Mastery Angoff technique. We used the intraclass correlation coefficient (ICC) to measure interrater agreement between judges for each of the three skills examinations and the written examinations, and overall items on all 4 tests combined. Statistical analyses were performed using SPSS version 25.0 (IBM Corp., Armonk, NY).

RESULTS

Demographic and clinical information for the 36 expert judges can be found in Tables 1 and 2. When averaged across groups, the MPSs for the skills examinations ranged from 94–99% items correct, while the written examinations were 97% for each of the FDA approved VADs (Table 3). These rigorous standards allowed patients and caregivers to miss no more than 2 of 32 items on the dressing change skills checklist, 0 of 12 items on the controller change skills checklist, 0 of 12 items on the power source change checklist, and 1of 54 items on the written examination for the HeartMate™ device. MPSs for the HeartWare™ device were similar because examinees could miss no more than 1 of 32 items on the dressing change skills checklist, 0 of 12 items on the controller change skills checklist, 0 of 9 items on the power source change checklist, and 1 of 35 items on the written examination.

Table 1.

Demographics of Patients and Caregivers who Participated in the HeartMate™ and HeartWare™ Standard Setting Exercises.

Characteristic HeartMate™ HeartWare™
Patients
n=6		 Caregivers
n=6		 Patients
n=6		 Caregivers
n=6
Age, mean (SD) 63.5 (15.3) 61.5 (11.9) 47.8 (19.4) 62.2 (13.1)
Male, no (%) 5 (83) 6 (100) 6 (100) 1 (17)
Race, no (%)
 African-American 2 (33) 2 (33) 3 (50) 3 (50)
 Caucasian 4 (67) 4 (67) 3 (50) 3 (50)
Married, no (%) 5 (83) 5 (83) 4 (67) 6 (100)
Education (highest degree), no (%)
 High school graduate 3 (50) 2 (33) 1 (17) 1 (17)
 Technical school, some college or associate’s degree 2 (33) 2 (33) 4 (67) 2 (33)
 Bachelor’s degree 0 1 (17) 1 (17) 1 (17)
 Postgraduate degree 0 1 (17) 0 1 (17)
 Unknown 1 (17) 0 0 1 (17)
Employment Status, no (%)
 Not employed 6 (100) 3 (50) 5 (83) 3 (50)
 Part-time employment 0 2 (33) 1 (17) 2 (33)
 Full-time employment 0 1 (17) 0 1 (17)
Open in a new tab

Table 2.

Demographics of Ventricular Assist Device (VAD) Coordinators and Attending Physicians who Participated in the HeartMate™ and HeartWare™ Standard Setting Exercises.

Characteristic VAD Coordinators
n=6		 Attending Physicians
n=6
Age (years), mean (SD) 35.7 (7.3) 44.0 (9.7)
Male, no (%) 0 4 (67)
Race, no (%)
 Asian 1 (17) 0
 Caucasian 5 (83) 6 (100)
Education (highest degree)*, no (%)
 BSN 1 (17)
 RN 1 (17)
 MSN 2 (35)
 MS, PA 2 (35)
 MD 6 (100)
Years in practice, mean (SD) [min,max] 6.9 (4.0) [3, 14.5] 10.8 (10.5) [2, 26]
Years of VAD experience, mean (SD) [min,max] 3.4 (1.9) [1,4] 10.5 (8.5) [3, 21]
Open in a new tab

Table 3.

Mean (SD) Minimum Passing Scores (MPS)s and Intraclass Correlation Coefficient Comparisons for Patients, Caregivers, Ventricular Assist Device (VAD) Coordinators, and Physicians as well as Final MPSs for the HeartMate™ and HeartWare™ Skills and Written Examinations.

Clinical Skills
Examination		 Patients
n=6		 Caregivers
n=6		 VAD
coordinators
n=6		 Physicians
n=6		 ICC Overall
ICC		 Final
MPS
HeartMate™
Dressing change (32 items) 93.4% (7.5%) 91.7% (7.5%) 93.3% (5.6%) 96.1% (3.1%) 0.68 0.80 93.6%
Controller change (12 items) 94.7% (7.0%) 96.3% (4.1%) 97.5% (2.1%) 99.6% (0.6%) 0.27 97.0%
Battery and power source changes (13 items) 98.6% (1.6%) 98.2% (2.6%) 99.4% (0.9%) 99.7% (0.4%) 0.35 99.0%
Written examination (54 items) 97.9% (1.9%) 95.2% (6.4%) 97.0% (1.3%) 96.9%(2.2%) 0.62 96.8%
HeartWare™
Dressing change (32 items) 92.2% (3.5%) 95.0% (4.6%) 93.3% (5.6%) 96.1% (3.1%) 0.71 0.74 94.1%
Controller change (14 items) 90.1% (6.1%) 94.9% (6.2%) 98.2% (2.0%) 99.9% (0.1%) 0.48 95.8%
Battery and power source changes (9 items) 94.3% (3.9%) 94.4% (7.2%) 99.3% (1.8%) 100% (0%) 0.47 97.0%
Written Examination (35 items) 95.1% (1.9%) 95.5% (2.0%) 98.2% (1.3%) 96.6% (2.2%) 0.82 96.5%
Open in a new tab

ICC=Intraclass Correlation Coefficient

The variation in individuals’ MPSs set for all assessments within each judge group ranged from 79% to 100% for patients, 79% to 100% for caregivers, 88% to 100% for VAD coordinators, and 91% to 100% for physicians. Intraclass correlations between patient, caregivers, VAD coordinators and physicians were high overall (0.80 and 0.74) for HeartMate™ and HeartWare™, respectively. However, lower interrater reliability was observed for the controller, and battery and power source change skills checklists (Table 3).

DISCUSSION

We designed a rigorous VAD self-care SBML curriculum for patients and caregivers. The curriculum features education best practices such as deliberate practice with feedback, use of a defensible MPS, and opportunities for additional practice until the MPS is reached (Cook, et al., 2013). We then used panels of patients, caregivers, VAD coordinators and physicians and the Mastery Angoff approach to set minimum standards for patients’ and caregivers’ VAD self-care skills and knowledge. Finally, once the panels’ judgments were assessed, we compared and contrasted each group’s MPSs and demonstrated that they were similar for each of the clinical skills.

This study adds to the SBML literature in several ways. First, to our knowledge, this is the first SBML curriculum for patients and/or caregivers. Second, we demonstrated that the Mastery Angoff standard setting method can be performed using patients and caregivers, as well as clinicians, and that this method results in uniformly high MPSs across different judge groups. Finally, we demonstrated that patients and caregivers set the same high standards for themselves as healthcare providers. Using patients and caregivers to help develop the curriculum and set standards is extremely important because it promotes patient-centered care.

Our study takes the use of simulation to train patients and caregivers one step further by using the mastery model, which ensures a high level of competency before completion of training. SBML has been shown to improve patient care and outcomes for skills performed by healthcare providers (Barsuk, et al., 2009b,c,d; Barsuk et al., 2013; Barsuk, et al., 2018b; Schwab et al., 2018; Zendejas et al., 2011). However, SBML has never been used to train patients or caregivers. In fact, little is known about the use of simulation-based education as a modality to improve the skills of patients and caregivers. Therefore, we believe the next step for simulation educators and practitioners is to consider training patients and their caregivers in additional self-care skills and to document competency among patients and caregivers. Based on prior work, we believe our SBML training model will improve both education and clinical outcomes. Specifically, we hope VAD self-care SBML will reduce outcomes such as driveline related infections and hospital readmission. Further study is needed to evaluate the impact of the VAD self-care SBML curriculum on learning and healthcare related outcomes for patients and their caregivers.

In our study, standard setting exercises were performed by a multidisciplinary team. A multidisciplinary team was previously used to set standards for resident physicians’ communication skills (Wayne et al., 2008b). In the earlier study, judges consisted of patients, communication experts, residency and fellowship program directors, and postgraduate medical trainees. Patients and communication experts set more stringent standards than program directors and trainees. This difference may have been due to the nature of the traditional Angoff method which asks judges to consider performance of “borderline” examinees, and is subject to individual interpretation. However, the Mastery Angoff method considers the “well prepared” learner who can perform tasks or demonstrate knowledge that is safe for patient care. In our study, patients, caregivers, VAD coordinators, and physicians largely agreed on the standards set using the Mastery Angoff technique, despite wide variation in background and experience. We believe this agreement occurred because perceptions of patient safety and high skill performance are similar among patients, caregivers, and healthcare providers. Our findings support other published research showing that the Mastery Angoff technique is feasible and easy to use in a variety of settings. (Prenner et al., 2018; Barsuk et al., 2018a).

The final MPSs set by the judges in our study were very high, requiring almost perfect performance. This standard suggests that all judges recognized the critical nature of the examinations because patients can suffer severe morbidity and mortality if VAD self-care tasks are performed incorrectly. Although some of the patient and caregiver MPSs were lower than VAD coordinators and physicians, the differences only represented up to one additional item on the assessments. There was also moderate variation between individuals’ MPSs within judge groups. The Mastery Angoff requires using an overall average of all judges’ scores to account for variations. Despite universally high standards, we found less interrater agreement for controller and battery and power source changes than dressing changes and the written examination. This difference likely resulted from the small number of test items on each of these checklists. In practice, the identified differences between groups were few and only resulted in a small change in the number of test items that could be missed (one item) while still achieving the MPSs.

Our study has some limitations. First, it was performed with a relatively small number of judges at a single medical center. Second, the curriculum is particular to VADs and no data is available about other patient education tasks. Third, some of our checklists had a low number of items with small variation between judges, which made intraclass correlation coefficients less reliable. However, the overall intraclass correlation coefficients between judges were very good, indicating a strong level of agreement for the standards set across all test items.

CONCLUSIONS

We created the first patient-centered SBML curriculum using feedback from a multidisciplinary panel of patients, caregivers, and clinicians. MPSs set using the Mastery Angoff technique by patients, caregivers, VAD coordinators and physicians were similar across groups. Our findings demonstrate that patients and caregivers, as well as clinicians, recognize the need for excellence in performance of high stakes self-care skills. Future studies will compare educational and healthcare outcomes in patients and caregivers randomized to our SMBL curriculum versus usual care. This innovative curriculum has the potential to not only improve patient and caregiver self-confidence and skills but also help to prevent serious adverse events.

Highlights.

  • Developing a simulation-based mastery learning self-care curriculum is feasible

  • Judges set high standards for ventricular assist device self-care assessments

  • Judges largely agreed upon the standards set using the Mastery Angoff Method

Key Points.

  • We developed a ventricular assist device self-care simulation-based mastery learning (SBML) curriculum for patients and caregivers.

  • A multidisciplinary team of patients, caregivers, nurses, advanced practice providers and physicians largely agreed on the minimum passing standards set using the Mastery Angoff method.

  • The Mastery Angoff method is ideal for setting standards due to agreement between judges on what constitutes a “well prepared” learner who can perform high stakes patient care tasks safely and independently.

ACKNOWLEDGEMENTS

We would like to acknowledge Drs. Douglas Vaughan, Kevin O’Leary, Duc Pham, and Clyde Yancy for their support and encouragement of this work. We thank the patients, caregivers, VAD coordinators and physicians at Northwestern Memorial Hospital for their dedication to education and patient safety.

Funding: This work was supported by the National Institutes of Health, National Institute of Nursing Research (grant number 1R21NR016745-01).

APPENDIX

72 Hour Dressing Change for HeartMate™ and HeartWare™

Copyright © 2017 by Northwestern University. All Rights Reserved.

graphic file with name nihms-1509917-f0001.jpg

Key: A = Done Correctly B = Done Incorrectly/Not Done

Checklist Skill Item
1. Have patient lie down and remove their shirt. May ask: “how you want to position the patient? A B
2. Use an antibacterial wipe to clean off the surface that is being used for set-up. A B
3. Wash your hands for 30 seconds with soap and water using antibacterial soap (removing jewelry). May ask: “Does your ring come off?” A B
4. Dries hands with a clean towel. A B
5. Properly opens the dressing change kit. A B
6. Takes out the sterile white sheet and places on surface for disposal pile. A B
7. Ensures everyone in the room has on a mask and places it on patient and self. (This must occur before the old dressing is removed.) A B
8. Puts on the gloves A B
9. Checks the skin anchor. Must demonstrate knowledge that it needs to be changed every seven days or if it does not look proper/on the patient. May ask: “what determines when the skin anchor needs to be changed?” A B
10. Removes the driveline from the tabs of the skin anchor and peels off the anchor. A B
11. Removes the pair of pants (small dressing). A B
12. Peels up the white part of the large dressing holding onto the driveline and pulling towards the driveline exit site. A B
13. Uses the alcohol prep pads in the kit to help remove extra material. If no extra material on patient can ask “How would you remove extra material (adhesive) from the skin?” A B
14. Removes the gloves. A B
15. Washes hands for 30 seconds. Soap and water or alcohol gel or foam is ok. May ask: “Does your ring come off?” A B
16. Puts on one sterile glove (wrong if violate sterile technique while doing so). A B
17. Puts on the other sterile glove (wrong if violate sterile technique while doing so). A B
18. Opens the Biopatch®, takes it out of its packaging and places it in the sterile kit. A B
19. Using one hand, gently grabs the driveline with the gauze. A B
20. Using the other hand, cleans a four to five inch diameter around driveline exit site by scrubbing back and forth for at least 30 seconds with the chlorhexidine wand. A B
21. Using one hand, gently grabs the driveline with a NEW piece of gauze. A B
22. Using the other hand, beginning at the point where the driveline exits the skin, scrubs the driveline with a back and forth motion for 15 seconds, on each of the four sides with a NEW chlorhexidine wand. (Total of 60 seconds) A B
23. Waits 90 seconds to dry. A B
24. Properly places the Biopatch®. A B
25. Properly places the large dressing. A B
26. Properly places the small dressing (pants). A B A B
27. Secures the new anchor to the skin. A B
28. Properly places the driveline in the tabs of the anchor. A B
29. Disposes of used materials. A B
30. Washes hands. A B
31. Did not have a potentially damaging pull on driveline exit site (one that puts pressure on the skin at the entry site). A B
32. Maintains sterile technique. A B
Open in a new tab

HeartMate™ Performing a System Controller Change

Copyright © 2017 by Northwestern University. All Rights Reserved.

graphic file with name nihms-1509917-f0002.jpg

Key: A = Done Correctly B = Done Incorrectly/Not Done

Checklist Skill Item
1. Have patient lie down. A B
2. Get the backup controller and place within easy reach (the battery clips should be attached already). A B
3. Place batteries in each of the two battery clips of the back-up pocket controller by lining up red arrows. A B
4. Slide open the safety door on the back of the primary pocket controller . A B
5. Press the red button to release the driveline. A B
6. Hold the metal part of the driveline to pull it out of the primary pocket controller. A B
7. Hold the metal part of the driveline while inserting the driveline into the backup controller. A B
8. Must somehow demonstrate understanding yellow arrow to yellow arrow (might be black) as well as need to feel/hear a click. Can ask: “how do you know you are putting in the driveline correctly?” A B
9. Close the safety door on the back of the new controller. A B
10. Disconnect the two batteries (or the MPU power) from the old controller. A B
11. Press and hold the battery button on the old controller for five seconds. A B
12. Perform a self-test on new controller. A B
Open in a new tab

HeartMate™ Performing a System Controller Change

Copyright © 2017 by Northwestern University. All Rights Reserved.

graphic file with name nihms-1509917-f0003.jpg

Key: A = Done Correctly B = Done Incorrectly/Not Done

Checklist Skill Item
1. Have patient lie down. A B
2. Get the backup controller and place within easy reach. A B
3. Remove the cord connecting to the power source (battery or AC) from number 1. A B
4. Connects this power cord to number 1 on the backup controller. A B
5. Pull back the white driveline cover that sits near the entry point at number 2. A B
6. Grasp the ribbed area of the driveline and pull the driveline from the bad controller. A B
7. Push the driveline into the new controller at number 2. A B
8. Must somehow demonstrate understanding that the notch or black line must be lined up with the orange dot on the controller. Can ask: “how do you know you are putting in the driveline correctly?” A B
9. Removes the red alarm adaptor at number 3 from the backup controller. A B
10. Remove the black cover from the port in the same spot on the bad controller (number 3). A B
11. Put the red alarm adaptor into the port or number 3 on the bad controller. A B
12. Remove the cord connecting to the other power source from number 4 on the bad controller. A B
13. Connect the power source to number 4 on the backup controller. A B
14. Slide the white driveline cover forward to protect the connection. A B
Open in a new tab

HeartWare™ Battery and Power Source Changes

Copyright © 2017 by Northwestern University. All Rights Reserved.

graphic file with name nihms-1509917-f0004.jpg

Key: A = Done Correctly B = Done Incorrectly/Not Done

Checklist Skill Item
Hooking up to MPU (wall power)
1. Ensure that the MPU is plugged in and on. A B
2. Unscrew the white connector from the battery clip and connect the white MPU cable to the white system controller connector. A B
3. Unscrew the black connector from the battery clip and connect the black MPU cable to the black system controller connector. A B
4. Press the battery release button on each battery clip, releasing the batteries. A B
5. Place depleted batteries in battery charger for recharging. A B
Hooking back up to battery A B
6. Gather appropriate equipment: two battery clips, two fully charged batteries, and battery holster or bag (if available). A B
7. Place one fully charged battery into each battery clip by aligning the arrows on the battery and battery clip and pushing the battery into the battery clip until a gentle “click” is felt. A B
8. Unscrew the white controller cable/connector from the white connector on the MPU. A B
9. Connect the white controller connector to the first battery clip. A B
10. Unscrew the black controller cable/connector from the black connector on the MPU. A B
11. Connect the black connector on the system controller to the second battery clip. A B
12. Check the battery “fuel gauge” by depressing the battery button on the controller to ensure the battery is full. A B
13. Place the battery clips and batteries in the patient’s holster or bag. A B
Open in a new tab

HeartWare™ Battery and Power Source Changes

Copyright © 2017 by Northwestern University. All Rights Reserved.

graphic file with name nihms-1509917-f0005.jpg

Key: A = Done Correctly B = Done Incorrectly/Not Done

Checklist Skill Item
Hooking up to wall power
1. Ensure the AC adapter is plugged in and the green light is on. A B
2. Remove one battery by turning the battery hub counterclockwise towards the release arrow and gently pull. A B
3. Connect the white arrow on the AC adapter cord to the red dot on power source connection located in the empty power slot on the system controller (Power source 1 or 2). A B
4. Ensure that patient’s system controller is always connected to one fully charged battery at all times. A B
5. Look at display screen on system controller to confirm patient is now connected to AC Power with one back up battery power source. May ask: “How do you know you are on AC power?” A B
Hooking back up to battery
6. Get second battery. A B
7. Disconnect from AC power by turning the AC hub counterclockwise towards the release arrow and gently pull. A B
8. Connect the white line on the battery cord to the red dot on the empty power source connection located in the empty power slot on the system controller. A B
9. Look at display screen on system controller to confirm patient is now connected to two battery power sources. May ask: “How do you know you are on battery power?” A B
Open in a new tab

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Conflict of Interest Disclosures: The authors have no relevant financial disclosures to report.

REFERENCES

  1. Barsuk JH, Ahya SN, Cohen ER, McGaghie WC, & Wayne DB (2009). Mastery learning of temporary hemodialysis catheter insertion by nephrology fellows using simulation technology and deliberate practice. American Journal of Kidney Diseases, 54(1), 70–76. 10.1053/j.ajkd.2008.12.041 [DOI] [PubMed] [Google Scholar]
  2. Barsuk JH, Cohen ER, Caprio T, McGaghie WC, Simuni T, & Wayne DB (2012). Simulation-based education with mastery learning improves residents’ lumbar puncture skills. Neurology, 79(2), 132–137. 10.1212/WNL.0b013e31825dd39d [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, & Wayne DB (2009). Use of simulation-based education to reduce catheter-related bloodstream infections. Archives of Internal Medicine, 169(15), 1420–1423. 10.1001/archinternmed.2009.215 [DOI] [PubMed] [Google Scholar]
  4. Barsuk JH, Cohen ER, Feinglass J, McGaghie WC, & Wayne DB (2013). Clinical outcomes after bedside and interventional radiology paracentesis procedures. American Journal of Medicine, 126(4), 349–356. 10.1016/j.amjmed.2012.09.016 [DOI] [PubMed] [Google Scholar]
  5. Barsuk JH, Cohen ER, Mikolajczak A, Seburn S, Slade M, & Wayne DB (2015). Simulation-Based Mastery Learning Improves Central Line Maintenance Skills of ICU Nurses. Journal of Nursing Administration, 45(10), 511–517. 10.1097/NNA.0000000000000243 [DOI] [PubMed] [Google Scholar]
  6. Barsuk JH, Cohen ER, Nguyen D, Mitra D, O’Hara K, Okuda Y, … Wayne DB (2016). Attending Physician Adherence to a 29-Component Central Venous Catheter Bundle Checklist During Simulated Procedures. Critical Care Medicine, 44(10), 1871–1881. 10.1097/CCM.0000000000001831 [DOI] [PubMed] [Google Scholar]
  7. Barsuk JH, Cohen ER, Vozenilek JA, O’Connor LM, McGaghie WC, & Wayne DB (2012). Simulation-based education with mastery learning improves paracentesis skills. Journal of Graduate Medical Education, 4(1), 23–27. 10.4300/JGME-D-11-00161.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Barsuk JH, Cohen ER, Wayne DB, McGaghie WC, & Yudkowsky R (2018). A Comparison of Approaches for Mastery Learning Standard Setting. Academic Medicine, 93(7), 1079–1084. 10.1097/ACM.0000000000002182 [DOI] [PubMed] [Google Scholar]
  9. Barsuk JH, Cohen ER, Williams MV, Scher J, Jones SF, Feinglass J, … Wayne DB (2018). Simulation-Based Mastery Learning for Thoracentesis Skills Improves Patient Outcomes: A Randomized Trial. Academic Medicine, 93(5), 729–735. 10.1097/ACM.0000000000001965 [DOI] [PubMed] [Google Scholar]
  10. Barsuk JH, McGaghie WC, Cohen ER, Balachandran JS, & Wayne DB (2009). Use of simulation-based mastery learning to improve the quality of central venous catheter placement in a medical intensive care unit. Journal of Hospital Medicine, 4(7), 397–403. 10.1002/jhm.468 [DOI] [PubMed] [Google Scholar]
  11. Barsuk JH, McGaghie WC, Cohen ER, O’Leary KJ, & Wayne DB (2009). Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Critical Care Medicine, 37(10), 2697–2701. [PubMed] [Google Scholar]
  12. Butter J, McGaghie WC, Cohen ER, Kaye M, & Wayne DB (2010). Simulation-based mastery learning improves cardiac auscultation skills in medical students. Journal of General Internal Medicine, 25(8), 780–785. 10.1007/s11606-010-1309-x [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Coleman EA (2014). Extending simulation learning experiences to patients with chronic health conditions. Journal of the American Medical Association, 311(3), 243–244. 10.1001/jama.2013.283057 [DOI] [PubMed] [Google Scholar]
  14. Cook DA, Brydges R, Zendejas B, Hamstra SJ, & Hatala R (2013). Mastery learning for health professionals using technology-enhanced simulation: a systematic review and meta-analysis. Academic Medicine 88(8), 1178–1186. 10.1097/ACM.0b013e31829a365d [DOI] [PubMed] [Google Scholar]
  15. Downing SM, Tekian A, & Yudkowsky R (2006). Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teaching and Learning in Medicine, 18(1), 50–57. 10.1207/s15328015tlm1801_11 [DOI] [PubMed] [Google Scholar]
  16. Ericsson KA (2015). Acquisition and maintenance of medical expertise: a perspective from the expert-performance approach with deliberate practice. Academic Medicine, 90(11), 1471–1486. 10.1097/ACM.0000000000000939 [DOI] [PubMed] [Google Scholar]
  17. Ericsson KA (2016). Peak : secrets from the new science of expertise. Boston: Houghton Mifflin Harcourt. [Google Scholar]
  18. Goldstein DJ, Naftel D, Holman W, Bellumkonda L, Pamboukian SV, Pagani FD, & Kirklin J (2012). Continuous-flow devices and percutaneous site infections: clinical outcomes. Journal of Heart and Lung Transplantation, 31(11), 1151–1157. 10.1016/j.healun.2012.05.004 [DOI] [PubMed] [Google Scholar]
  19. Haupt BA, & Beauvais A (2014). Veterans receive high-fidelity simulation education preoperatively. Clinical Simulation in Nursing, 10(11), 538–545. [Google Scholar]
  20. HeartWare. (2012). Written Exam for Patients and Caregivers. Retrieved from https://www.heartware.com/clinicians/resources/program-development
  21. Kane M (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425–461. [Google Scholar]
  22. Kirklin JK, Pagani FD, Kormos RL, Stevenson LW, Blume ED, Myers SL.. Naftel DC (2017). Eighth annual INTERMACS report: special focus on framing the impact of adverse events. Journal of Heart and Lung Transplantion, 36(10), 1080–1086. 10.1016/j.healun.2017.07.005 [DOI] [PubMed] [Google Scholar]
  23. Menon AK, Baranski SK, Unterkofler J, Autschbach R, Moza AK, Goetzenich A, & Tewarie L (2015). Special Treatment and Wound Care of the Driveline Exit Site after Left Ventricular Assist Device Implantation. Thoracic Cardiovascular Surgery, 63(8), 670–674. 10.1055/s-0035-1554961 [DOI] [PubMed] [Google Scholar]
  24. Norcini J, & Guille R (2002). Combining tests and setting standards. Dordrecht, the Netherlands: Kluwer Academic Publishers. [Google Scholar]
  25. O’Horo JC, Abu Saleh OM, Stulak JM, Wilhelm MP, Baddour LM & Sohail MR (2018). Ventricular assist device infections: a systematic review. ASAIO Journal. 64, 287–294. 10.1097/MAT.0000000000000684 [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Paniagua MA, & Swygert KA (2016). Constructing written test questions for the basic and clinical sciences. Philadelphia, PA: National Board of Medical Examiners. [Google Scholar]
  27. Pereda D, & Conte JV (2011). Left ventricular assist device driveline infections. Cardioliology Clinics, 29(4), 515–527. 10.1016/j.ccl.2011.08.004 [DOI] [PubMed] [Google Scholar]
  28. Prenner SB, McGaghie WC, Chuzi S, Cantey E, Didwania A, & Barsuk JH (2018). Effect of trainee performance data on standard setting judgments using the Mastery Angoff method. Journal of Graduate Medicate Eduacation, 10(3), 301–305. 10.4300/JGME-D-17-00781.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Rudolph JW, Simon R, Rivard P, Dufresne RL & Raemer DB (2007). Debriefing with good judgment: combining rigorous feedback with genuine inquiry. Anesthesiology Clinics, 25(2), 361–376. 10.1016/j.anclin.2007.03.007 [DOI] [PubMed] [Google Scholar]
  30. Schwab B, Teitelbaum EN, Barsuk JH, Soper NJ, & Hungness ES (2018). Single-stage laparoscopic management of choledocholithiasis: An analysis after implementation of a mastery learning resident curriculum. Surgery, 163(3), 503–508. 10.1016/j.surg.2017.10.006 [DOI] [PubMed] [Google Scholar]
  31. Stufflebeam DL (2000). The Checklists Development Checklist. Retrieved from https://wmich.edu/sites/default/files/attachments/u350/2014/guidelines_cdc.pdf
  32. The Joint Commissions. (2018). Required Advance Certification Ventricular Assist Device. Retrieved from https://www.jointcommission.org/required_advanced_certification_ventricular_assist_device/. [Google Scholar]
  33. Thomas PA, Kern DE, Hughes MT, & Chen BY (2016). Curriculum development for medical education : a six-step approach (Third edition. ed.). Baltimore: Johns Hopkins University Press. [Google Scholar]
  34. Thoratec. (n.d.). Patient Education. Retrieved from http://www.thoratec.com/medical-professionals/patient-education/index.aspx
  35. Tofil NM, Rutledge C, Zinkan JL, Youngblood AQ, Stone J, Peterson DT, … White ML (2013). Ventilator caregiver education through the use of high-fidelity pediatric simulators: a pilot study. Clinical Pediatrics, 52(11), 1038–1043. 10.1177/0009922813505901 [DOI] [PubMed] [Google Scholar]
  36. Wayne DB, Barsuk JH, O’Leary KJ, Fudala MJ, & McGaghie WC (2008). Mastery learning of thoracentesis skills by internal medicine residents using simulation technology and deliberate practice. Journal of Hospital Medicine, 3(1), 48–54. 10.1002/jhm.268 [DOI] [PubMed] [Google Scholar]
  37. Wayne DB, Butter J, Siddall VJ, Fudala MJ, Wade LD, Feinglass J, & McGaghie WC (2006). Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice. Journal of General Internal Medicine, 21(3), 251–256. 10.1111/j.1525-1497.2006.00341.x [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Wayne DB, Cohen E, Makoul G, & McGaghie WC (2008). The impact of judge selection on standard setting for a patient survey of physician communication skills. Academic Medicine, 83(10 Suppl), S17–20. 10.1097/ACM.0b013e318183e7bd [DOI] [PubMed] [Google Scholar]
  39. Yarboro LT, Bergin JD, Kennedy JL, Ballew CC, Benton EM, Ailawadi G, & Kern JA (2014). Technique for minimizing and treating driveline infections. Annals of Cardiothoracic Surgery, 3(6), 557–562. 10.3978/j.issn.2225-319X.2014.09.08 [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Yudkowsky R, Park YS, Lineberry M, Knox A, & Ritter EM (2015). Setting mastery learning standards. Academic Medicine, 90(11), 1495–1500. 10.1097/ACM.0000000000000887 [DOI] [PubMed] [Google Scholar]
  41. Yudkowsky R, Tumuluru S, Casey P, Herlich N, & Ledonne C (2014). A patient safety approach to setting pass/fail standards for basic procedural skills checklists. Simulation in Healthcare, 9(5), 277–282. 10.1097/SIH.0000000000000044 [DOI] [PubMed] [Google Scholar]
  42. Zendejas B, Cook DA, Bingener J, Huebner M, Dunn WF, Sarr MG, & Farley DR (2011). Simulation-based mastery learning improves patient outcomes in laparoscopic inguinal hernia repair: a randomized controlled trial. Annals of Surgery, 254(3), 502–509; discussion 509–511. 10.1097/SLA.0b013e31822c6994 [DOI] [PubMed] [Google Scholar]

RESOURCES