Hand-off Education and Evaluation: Piloting the Observed Simulated Hand-off Experience (OSHE)

Abstract

Aim

The Observed Simulated Hand-off Experience (OSHE) was created to evaluate medical students’ sign-out skills using a real-time assessment tool, the Hand-off CEX.

Setting

Thirty-two 4th year medical students participated as part of an elective course.

Program description

One week following an interactive workshop where students learned effective hand-off strategies, students participated in an experience in which they performed a hand-off of a mock patient using simulated history and physical examination data and a brief video.

Program evaluation

Internal medicine residents served as standardized hand-off receivers and were trained on expectations. Students were provided feedback using a newly developed Hand-off CEX, based on the “Mini-CEX,” which rates overall hand-off performance and its components on a 9-point Likert-type scale. Outcomes included performance ratings and pre- and post-student self-assessments of hand-off preparedness. Data were analyzed using Wilcoxon signed-rank tests and descriptive statistics. Resident receivers rated overall student performance with a mean score of 6.75 (range 4–9, maximum 9). Statistically significant improvement was observed in self-perceived preparedness for performing an effective hand-off (67% post- vs. 27% pre-reporting ‘well-prepared,’ p < 0.009).

Discussion

This brief, standardized hand-off training exercise improved students’ confidence and was rated highly by trained observers. Future work focuses on formal validation of the Hand-off CEX instrument.

Electronic supplementary material

The online version of this article (doi:10.1007/s11606-009-1170-y) contains supplementary material, which is available to authorized users.

AIM

In July 2003, the Accreditation Council for Graduate Medical Education (ACGME) instituted resident duty hour limitations in an effort to reduce sleep deprivation and improve patient safety¹. An unintended consequence of this change in practice was an increase in the number of patient care hand-offs², a vulnerable gap in patient care that is often variable and error prone^3–10. The 2008 Institute of Medicine (IOM) report on the impact of the duty hour regulations acknowledges this issue and also recommends structured training in hand-off communication for all trainees¹¹. Previous literature has revealed that few trainees receive formal training in effective hand-off communication^2,12 despite the ubiquitous nature of this task performed by interns in every specialty. The lack of trainee preparation on how to perform hand-offs impacts a number of stakeholders, including residency program directors¹³. This argues for incorporation of hand-off skills training into the senior year of medical school.

To teach hand-off communication, one promising method to consider is the use of standardized patients (SP). Standardized patients and the observed structured clinical examination (OSCE) were first developed to teach and assess clinical skills¹⁴ and have been extensively validated^15,16. More recently, educators have adapted the OSCE to teach “non-traditional” curricular content to medical students, including quality improvement¹⁷ and teaching skills¹⁸. Based on this successful model of training medical students using standardized patients, we created the first Observed Simulated Hand-off Experience, or OSHE, using standardized resident receivers to evaluate hand-off skills of 4th-year medical students. Since validated tools to assess performance during hand-offs are lacking, we devised a real-time assessment tool modeled after the validated Mini-CEX (Clinical Examination)¹⁹, called the Hand-off CEX.

SETTING

The OSHE was a module in a month-long skills-based “Transitions to Internship” elective course offered to graduating students in April of their final year of medical school. Thirty-four students who had matched into nine different specialties participated in the course. This elective focuses specifically on “orphan” topics not addressed elsewhere in the curriculum. Topics include procedural skills, preparedness for life as a resident, advanced communication skills (e.g., obtaining informed consent and disclosing medical error) and personal life preparedness (e.g., debt management and maintaining work/life balance).

PROGRAM DESCRIPTION

Students participated in a 90-min interactive workshop on communication skills necessary to perform effective hand-offs (see online Appendix A). This module, delivered by one of the authors (VA), discussed communication theory as it relates to effective hand-off communication. Key points communicated during this workshop included: the importance of face-to-face communication, encouraging questions from the hand-off receiver, providing detailed “to-do” lists with rationale and anticipatory guidance. The instructor included case-based examples on the importance of effective hand-offs both in medicine and other industries²⁰. The components of a written hand-off were discussed during the 90-min hand-off training, and students were given a representative example of a high-quality written sign-out²¹.

Finally, students were provided with electronic access to the workshop materials and also received a pocket card with helpful tips on performing an oral and written sign-out^22,23.

One week following the workshop, students participated in the standardized hand-off experience during a 2-h afternoon session. A mock patient history and physical (see online Appendix B) examination transcript of a patient with pneumonia was created by the teaching faculty and pilot-tested with a group of four PGY-2 and PGY-3 residents, and modified based upon their input. The history and physical documentation included routine data, such as the history of present illness, past history, physical examination, hospital medications, identity of patient’s primary care doctor, code status and contact information for family (see online Appendix B).

Students received a 10-min introduction to the OSHE experience (see online Appendix C) and then received a copy of the mock history and physical examination transcript and were given 10 min to review the information. Students then viewed a 5-min video depicting “interval patient events” (view video at http://web.me.com/johnparo/MedSchoolRock/oshe.html). The video conveyed the dynamic nature of patient care by providing important clinical updates intended to trigger anticipatory guidance and “to-do” items. Examples of anticipatory guidance and “to-do” items include following up on pending laboratory results and/or changes in clinical status that could require escalation of care or communication with the patient’s family. For example, during the video an intern physician completes a radiology requisition form, and the resident’s voice is heard stating “Can you make sure your cross-cover follows-up on the chest x-ray?” During the video, the patient becomes hypotensive and tachypneic, and develops an increased requirement for supplemental oxygen. This event is meant to trigger a need for the primary team intern to document a possible need for escalation of care. Students received a blank sign-out template without any prompts or labels as to what information to include (see online Appendix D) and were given 15 min to incorporate information they considered relevant (identifying information, hospital course, medications, history and physical data, interval events). Students were instructed to produce a written sign-out and given 10 min to “hand-off” the patient to a standardized resident receiver and receive feedback on their performance. Students performed their “hand-off” in a private room in a Clinical Performance Center at the Pritzker School of Medicine.

PGY-2 and -3 internal medicine residents served as standardized resident “receivers” of the hand-off. Prior to evaluating the students, all residents received 1 h of training on hand-off expectations using the mock history and physical and anticipated “trigger” events displayed in the video (see online Appendix E). The residents' feedback was facilitated using the Hand-off CEX (see Appendix 2). Drawing upon the investigators’ preliminary work done in this area and relevant strategies utilized in other industries^24,25, we developed a tool to evaluate hand-offs in the clinical setting. This instrument is based upon a previously validated, widely used, real-time educational evaluation tool, the Mini-CEX²⁰. The Hand-off CEX instrument includes assessment of five domains, including: organization, communication skills, content, clinical judgment, humanistic qualities and overall hand-off competence. Each domain is scored on a 9-point scale that includes unique role-based anchors for the highest and lowest scores of each domain. For example, the unsatisfactory anchor for the communication domain included “understanding not confirmed, no time for questions,” whereas the superior anchor included “understanding confirmed, questions elicited.” Setting is a part of the original Hand-off CEX tool, but was not assessed in this study given the standardized conditions of Clinical Performance Center. Finally, space was provided for standardized resident receivers to provide open-ended responses on student performance.

Written sign-outs were prepared by students after the review of the mock chart and interval patient events video. Students’ written sign-outs were evaluated by one investigator (JF) using a scoring system generated by the investigators (JF, VA, JJ) that assigned 20 total points for correct information for the specific case utilized in the exercise in each of the following categories (see Appendix 2): identifying information (6 points), problem list (4 points), medication list (5 points), anticipatory guidance (2 points) and “to-do” tasks (3 points).

To assess self-perception of hand-off competence, students completed an anonymous, paper questionnaire before and after the intervention. The questionnaire included an item for students to rate, using a Likert-type scale, their ability to perform hand-offs (not at all prepared, somewhat unprepared, somewhat prepared, well prepared and very well prepared).

PROGRAM EVALUATION

The University of Chicago Institutional Review Board deemed this study exempt from review. Aggregate unpaired data were analyzed using descriptive statistics and, where appropriate for paired data, Wilcoxon sign-rank tests were used.

Thirty-two of 34 students (94%) enrolled in the course over 2 years participated in the OSHE. Two students were unable to participate in the experience because of personal scheduling conflicts. All students who participated were rated by standardized resident “receivers.” Students received a mean overall sign-out competence score of 6.75 (range 4–9, maximum 9) (see Table 1).

Table 1.

Observed Simulated Hand-off Experience (OSHE) Performance by Hand-off CEX^a Domain (n = 32 Students)

Domain	Mean student performance	SD/range (max 9)
Organization/efficiency	6.62	1.00; 5–8
Communication skills	7.15	0.84; 5–9
Content	6.62	1.43; 3–9
Clinical judgment	6.64	1.10; 3–8
Humanistic qualities/professionalism	7.62	0.83; 6–9
Overall hand-off competence	6.75	1.07; 4–9

^aHand-off CEX rates domains on a scale of 1–9 (unsatisfactory to superior) with role-based anchors (see Appendix 1)

Faculty review of the content quality of all 32 written sign-outs revealed a mean score of 16.2 (range 9–20, maximum 20). The most frequent omission noted was a lack of anticipatory guidance provided for the covering physician. For example, few students included the anticipation for need of escalation of care for a patient with an increasing oxygen requirement. After the exercise, de-brief discussion with standardized resident receivers commented on their satisfaction with the instrument and ease of use.

All 32 students completed pre- and post-surveys assessing preparedness for aspects of internship training, including hand-off performance. Evaluation of pre- and post-survey data revealed statistically significant improvement in preparedness for performing effective hand-off (27% pre- vs. 67% post-reporting ‘well prepared’ or ‘very well prepared;’ p < 0.009). Students also expressed unanimously positive comments on the experience (“a must have, a great experience!” and “probably the MOST USEFUL of all topics”). Students also remarked on the realism of the exercise because of the use of the standardized resident receivers. The majority of students (66%) felt that this experience should be required of all graduating 4th year students, and all (100%) agreed that their OSHE experience would impact their practice as interns. Finally, students also expressed the desire to experience additional scenarios, specifically “receiving” a hand-off as well as “sending” one.

DISCUSSION

The implementation of the duty hour regulations for resident physicians has had the unintended consequence of increasing discontinuity in patient care and frequency of patient hand-offs. As a result, improving hand-off communication has become increasingly important. Previous research has illustrated that current hand-off practices are often incomplete and that these poor hand-offs can contribute to adverse events^5,26. The OSHE is a feasible, interactive mechanism to provide students with the ability to learn and practice hand-off communication.

Education on effective hand-off communication is vital to clinical practice, and the evaluation of hand-off skills is essential in enhancing quality care. Currently, there are no validated instruments with which to evaluate hand-off skills, and many institutions rely upon antiquated and ineffective methods of patient sign-out². The Hand-off CEX instrument may fill this void, as it was well received by both evaluators and students for its ease of use. Our preliminary data suggest that this instrument is able to measure key aspects of hand-off performance.

The use of a modified OSCE, a well-developed and validated training method, for the purpose of teaching hand-off communication skills is promising. Many medical schools already employ OSCE-type training into their clinical skills curricula, and its wide-spread acceptance is manifest by its inclusion into the US medical licensing examination. Preparation for the OSHE involved 1 h of rater training and required a 3-h block in our Clinical Performance Center for this sample of students. Faculty and resident time was provided on a voluntary basis, and use of the Clinical Performance Center was provided by the Academy of Distinguished Medical Educators at the University of Chicago Pritzker School of Medicine. Despite the monetary fee for the space rental, we believe that the modification of the OSCE format to teach and assess communication skills is both logistically and financially feasible.

Although students demonstrated statistically significant improvement in self-perceived preparedness for performing an effective hand-off, only slightly more than half of students reported feeling “well-prepared” even after the session. This suggests that effective communication skills training throughout post-graduate clinical education will be needed. Some institutions are already beginning such education during residency training²², and the recent IOM Report encouraging continued graduate medical education (GME) training in hand-off communication begs for the development of validated tools and curricula.

There are several limitations to this innovation. Our data reflect a single-institution experience, and only one hand-off scenario was evaluated during the OSHE. In addition, the use of housestaff as evaluators, and not formally trained standardized patients, could be viewed as a limitation. However, residents received structured training on effective hand-off communication and on providing feedback utilizing the Hand-off CEX instrument. Also, their familiarity with the process of handing off patients makes them more suited for the role than standardized patients. An additional limitation is the use of a self-perceived preparedness instrument by students rather than a pre- and post-skills assessment. The current findings do suggest using a modified-OSCE to teach hand-off communication is effective as a training method for students, although future work will focus on a more objective measure of hand-off performance. Finally, our innovation was performed using a small number of students.

The OSHE may be a novel and effective method for teaching effective hand-off skills to medical student trainees. The experience was well-received by both students and resident participants. In addition, the Hand-off CEX shows promise as an evaluative tool for both training purposes and in-hospital hand-offs. Future work aims to disseminate the OSHE to other institutions and residency training programs and also to formally establish the validity and reliability of the Hand-off CEX instrument.

OSHE video can be viewed at

http://web.me.com/johnparo/MedSchoolRock/oshe.html

]

Appendix 1: Hand-off CEX instrument

Appendix 2: Standardized Hand-off answer key

Table 2

Table 2.

Standardized Hand-off Answer Key

Identifying information	Clinical scenario	Medication	If…,then…	To do:
Patient namea	One-line summary of clinical scenarioa	Lisinopril 5 mg po daily, insulin glargine 20 units SQ qHS, Novolog 8 units TID c meals, Albuterol/atrovent nebs q4h, sliding scale insulin	1. Anticipation of need for escalation of careb	[] Follow-up on CXRb
Patient MRNa	1. Active medical issues (i.e., COPD exacerbation, fever)			[] Follow-up on BMP/K+b
Rm ba	2. Chronic stable medical issues( i.e., HTN, DM2)
Allergiesa		Moxifloxacin 400 mg po q24h, Prednisone 60 mg po daily SHOULD INCLUDE: NAME [GENERIC] DOSE ROUTE COMPLIANT ABBREVIATIONS	2. Anticipation of hyperglycemia given diabetic and on steroidsa	[] Family meeting to discuss goals of careb
Code statusab	3. Prophylaxis
PCPa	4. Code status

^aInformation derived from mock chart

^bInformation obtained from video

SCORING FOR WRITTEN SIGN-OUT:

Six-point identification data =1 point for each element in identification data

• One for each element missed or incorrect

Four-point problem list

One point for one-line summary statement with why patient is here now

Three points for problem list, 1 point for each active problem with statement of management

Five points for complete medication list (all medications, all doses, all frequencies including PRN medications)

• One point if missing over 50% doses

• One point if missing over 50% of frequencies

• One point if no PRN medications listed

Three points total for To Do:

• One point for each To Do

Two points for If/Then Statement

Twenty points total