The Development of a Web-based Resident Sign-out Training Program

Abstract

Patient sign-out is a mechanism for transferring information, responsibility, and/or authority from one set of caregivers to another. Little research has addressed what information should be communicated during sign-out and how sign-out should be conducted and evaluated. As hospital residents conduct many sign-outs and have limited time in general, targeted web-based training and evaluation have the potential to enhance Graduate Medical Education. However there are no web-based training systems for this very important skill. This paper presents the operational concept and system requirements for a web-based sign-out training system. It discusses an initial functional prototype. Results of a heuristic evaluation and an assessment of areas for improvement are presented.

I. Introduction

Patient sign-out is a mechanism for transferring information, responsibility, and/or authority from one set of caregivers to another. Physicians not involved in the patient’s initial assessment and plan of care can later provide care. Thus during sign-out, physicians assuming patient care need the necessary information, including “just in case” (or contingency) information.

Sign-out is typically informal and unstructured with little standardization [1–6]. Ineffective and/or inefficient sign-out can lead to omissions or misunderstandings of critical information that limit patient care [2, 7–10]. In 2003, the Accreditation Council for Graduate Medical Education instituted duty hour restrictions for all residency programs in the United States [11]. As a result, the frequency of sign-out and associated problems have been leading to a decrease in continuity of care, longer patient hospital stays, an increase in the number of laboratory tests ordered and performed, and an increase in the number of self-reported preventable adverse events [12–16].

Sign-out as a method for maintaining continuity of operations at shift change has been studied in many domains- e.g., [8, 17, 18]. Maintaining continuity includes: 1) seamless transfer of physical presence/activity (as if the same person had retained responsibility across shifts), and 2) shared understanding of the current situation (between those handing over responsibility and those accepting it) [18]. Reaching a shared situation awareness or “co-orientation” [19,20] includes communicating the perception and assessment of a system (i.e. patient) state and plan (i.e. plan of care) as well as “just in case” (or contingency) information [4,8] to support events that may occur during the upcoming shift. Collaboration is as a key factor in mitigating errors in team related communications. In particular, the ability of individuals to evaluate the accuracy of each other’s beliefs or behaviors called collaborative cross-checking [21] has been shown to be a significant factor in error detection during team-based communications [22].

What information resident physicians need for patient care over a shift and the strategies such as developing shared situation awareness or collaborative cross-checking, are not well-defined. Rather, the information exchanged and the strategies employed during sign-out are based on consensus [4], tradition, and subjective opinion [23]. Furthermore, the ability to give and receive sign-out is generally learned informally by observing more senior providers giving sign-out [24–26]. Effective sign-out strategies such as collaborative cross-checking may not be observed in practice [22].

Little is known about how sign-out related interventions, particularly training, improve the quality of sign-out, or how these interventions influence patient safety and the quality of patient care, especially in cognitive-based medical specialties where physicians spend most of their time examining and treating patients as opposed to performing procedures [27]. Thus educational programs must be developed to train residents how to communicate effectively at sign-out [4, 7, 8, 10, 25]. Unfortunately, there is little evidence to guide the development of such educational programs and even less evidence on their effectiveness [4, 28]. A small number of training programs have been designed for general medical team training which address communication issues between members of medical teams using simulations as well as classroom training [29]. However, few of these training programs formally teach resident physicians effective communication strategies for sign-out (or the information that should be communicated) and even fewer programs assess residents’ abilities to apply these strategies [4, 6, 30]. Measures of assessment have focused primarily on residents’ opinion and general observation, and have not involved assessment of learning or behavioral changes that result from training specific sign-out strategies.

Using an Instructional Systems Design (ISD) methodology [31], we are developing, delivering, and evaluating a resident sign-out training curriculum. The goal of the curriculum is to improve the sign-out process from one that is informal and unstructured (prone to variability, lacking standardization in the type, extent, and order of information exchanged, largely driven by the giver, and deficient in summarization, collaborative problem solving and collaborative cross checking), to a more formal and structured, consistent process (emphasizing teamwork and shared situational awareness between participants, and promoting collaborative problem solving and collaborative cross checking).

The first phase (Phase 1) of the sign-out training curriculum is based on the five key content components that we have identified as integral to effective patient sign-out (patient identification, the “big sentence”, current condition, plan of care, and contingency plans), and the process of conducting sign-out. Later additions to the sign-out training curriculum will focus on communication and teamwork for creating shared situation awareness (co-orientation) and communication for collaboration (collaborative cross-checking skills). Herein we describe our sign-out training strategy for Phase 1 of the curriculum and the requirements for the training delivery system based on the training strategy. We present our design and implementation of a functional prototype. Constraining the design was that our stakeholders wanted to receive the training online via an interactive webpage and should require no more than 45 to 60 minutes to complete. We present the results of a heuristic evaluation of the prototype, the initial results of our evaluation of the training’s overall effectiveness, and discuss areas for improvement in future system designs.

II. Training Strategy

In ISD, the training strategy involves definition of the performance objectives, the organization of the information content, the definition of the assessment strategy, and the strategy for how the training is delivered to learners.

A. Performance Objectives

Performance objectives provide a detailed description of what the learner will be able to do after completion of the training. They are:

1. To identify and define the five key information components for patient sign-out.

2. To recognize effective sign-out by others.

3. To communicate all five components of sign-out without prompting.

B. Organization of Information Content

The content includes five major sections (three training ones, discussed next, and two assessment sections, discussed in Section C.) The three training sections include an overview of sign-out, the five information components of sign-out and the sign-out process. The overview of sign-out is designed to motivate the learner and includes: a history of sign-out, definitions of sign-out, sources of error in sign-out, and an outline of the training goals. The five information components of sign-out is based on a literature review and our own prior work[32–34] It consists of six subsections: an overview, identifying the patient, the “big sentence”, the current condition, the plan of care, contingency plans, and a summary. The sign-out process consists of five subsections: an overview, the importance of process, logistical guidelines, methodological guidelines, and a summary.

The pedagogy includes case studies as a means of modeling situations where poor sign-out led to critical errors in patient care. Three are included in the initial training curriculum. We also developed a simulated History and Physical document to accompany the “big sentence” section as certain data elements from a patient’s History and Physical may be used in constructing a patient’s “big sentence” during sign-out. The pedagogy as includes items to motivate the learner including a series of interesting facts from the literature and resident testimonials related to sign-out. These are called “Did You Know Facts.” While the learner is not required to view them, the idea is to have them available for the interested learner.

C. Assessment Strategy

The assessment strategy includes objective-referenced tests (criterion-referenced tests) designed to measure the set of performance objectives and the learners’ knowledge of the related training content. For assessment we employ a pre-test and post-test assessment paradigm as well review questions at particular points within the training sections. Learners receive response specific feedback to enhance learning. Assessment questions are in the multiple choice or true/false format. Seven review questions and thirteen questions for both the pre-training and post-training assessments (total of thirty-three questions) are included in the training. The pre-training and post-training questions were aligned to mirror each other.

D. Training Delivery Strategy

Because residents have limited time for training, learners should be able to go through training when it is convenient for them and should not require the presence of an instructor. The training delivery strategy includes the use of multimedia in the content and the use of multimedia as well as feedback in the training assessments.

III. Training System Requirements

The training system requirements involve the representation of the training content including the information to be presented as well as learner assessments, navigation though the system modules, and the reporting of results to the learner and the instructor.

A. Representation of Content

The training content involves information for the learner to learn as well as assessment materials. As part of the training strategy, the content on a page could be in text, image, audio or video format, or a combination of each. The system had to be able to support all of these formats.

The assessment questions (pre-assessment test, survey questions, and post-assessment test) are aimed to evaluate knowledge gained immediately through the training. The training system must provide the means for developing objective-referenced assessment questions in both multiple choice and true/false formats.

Some of the training materials were supplemental. Therefore the system needed to support optional materials. Specifically our “Did You Know Facts” (DYKF) represent information associated with sign-out (interesting factual data related to sign-out and resident testimonials) that is supplementary to the training content, but not required.

B. Navigation

This self-paced training system must support learner-centered operation and navigation through the training content without skipping required content. The system should allow the learner to progress forward or backward one page at a time or to restart the current section. The system should allow the user to track his/her progress through the training. The system should also allow the user to exit the system at any time.

C. Reporting of Results

The reporting requirements were two-fold: reporting to the learner and reporting to the instructor. Specifically, the system was required to provide feedback to the learner based on submitted assessment answers with teaching points for any assessment questions completed incorrectly. In addition the system should provide a report with the overall assessment score and scores on each assessment question to the instructor.

IV. Prototype Design

A. Content Implementation

The initial version of the sign-out training system was developed with Adobe Captivate (version 4) and Adobe ColdFusion. The design view of Captivate supports slide-based implementation of the training. Individual slides can be organized into sections using the Table of Contents editor.

In the slide design mode, slides may be populated with text, images, video, audioclips, and action buttons. The text for each slide is inserted using text boxes and images imported as GIFs or JPEGs. In Captivate, audio and video files must be imported as flash video (FLV) files if the designer wishes to include playback controls with the file. If not already in FLV format audio and video files must be converted to FLV format prior to inserting them into the project in order to utilize the playback controls offered by Captivate. In total we developed and converted five instructional videos and ten audio examples of sign-out for the training content. We employed a design template used to give the interface of the project an overall professional and consistent look and feel (Fig. 1).

Figure 1.

Training welcome screen with table of contents

As “Did You Know Facts” were considered supplementary information within the training system it was necessary to determine a method by which learners could optionally access them. Each DYKF fact was implemented as a separate HTML web page which could be accessed via a button in the Captivate project (Fig. 2). By pressing the button in the Captivate project, each “Did You Know Fact” would open as a pop-up window. The brief vignettes, or case studies, were also presented as pop-up windows although a different button design was used.

Figure 2.

Button to access “Did You Know Facts”

B. Delivery Method

Once the Captivate design portion of the project had been completed, it was published as a web-page with an embedded Shockwave Flash object to play the training content. When published as a web page, the table of contents allowed learners to track training progress and to return to any section of the training previously completed (other than an assessment question slide). Completed sections are denoted by a green check mark next to the section title on the table of contents. These check marks also serve as a progress tracker for the user.

C. Implementation of Assessments and Feedback

The quizzing functionality within Adobe Captivate supports a structure for the system’s pre-training and post-training assessments and review questions. The quizzing functionality and options found within the quiz preferences menu of Adobe Captivate allowed each question to be either graded or ungraded in the case of the review questions.

In addition, assessment questions could be developed in many different formats such as multiple choice with one answer, multiple choice with multiple answers, and true/false. Each of the three forms listed above were utilized in the sign-out training system. In particular, the pre-training assessment and the post-training assessment were developed as graded questions, and the review questions were developed as ungraded questions. An example of a graded pre-training assessment question is shown in Fig. 3.

Figure 3.

Example of a graded assessment question

Adobe Captivate supports indicating whether the correct or incorrect answer was provided following the submission of an answer. It supports providing customized feedback based on the learner’s particular answer choice. If the learner attempted to submit a response to a question without actually selecting an answer, a caption instructs the learner to select an answer before continuing.

For additional feedback, an assessment review slide was created and presented to the learner at the end of the training. This slide informs a learner of his or her overall pre-training and post-training assessment scores. The assessment review slide also instructs the learner to submit his or her results in order to complete the training. Assessment results are stored in a single variable by the published Captivate project which contains the results of each graded assessment question as well as the overall assessment results. The “results” variable can be accessed and emailed to an instructor for analysis; however, in order to implement this functionality it was necessary to utilize a separate software package called Adobe ColdFusion. ColdFusion is a server-side scripting language that is designed to access particular variables such as the “results” variable and then perform various actions on them such as emailing the variable contents. Our particular implementation of the ColdFusion script accessed the “results” variable and emailed its contents to a research assistant for analysis.

V. Training Evaluation

We conducted an initial heuristic evaluation of the prototype in order to ensure that the design adhered to the requirements and did not exhibit any unforeseen content or design issues. Once the heuristic evaluation was complete, we evaluated the effectiveness of the training by conducting learning and behavioral assessments of 12 first-year pediatric residents after receiving the training.

A. Method

The heuristic evaluation questionnaire was designed to focus on content and human factors issues as well as general comments. The content section consisted of six questions, each covering a separate aspect audio examples, history and physical example, “did you know” facts, case studies, conceptual organization, and training length. Three human factors questions addressed navigation, the use of pop-up windows, font size and colors, and the overall “look and feel” of the training. The general comments section requested the evaluators to provide the three most useful and least useful aspects of the training, one aspect they would change, and one aspect they would emphasize in future designs.

Our strategy was to employ both medical and human factors evaluators. The three evaluators included two third year pediatric residents and one undergraduate engineering student. The two pediatric residents had previously been identified by their peers as “good sign-out givers”, and thus could provide the most relevant feedback with respect to the training content. The undergraduate had successfully completed a human factors course that focuses on user-centered design. The evaluators were contacted by email for their participation in the study.

At the start of the evaluation session, an analyst described the purpose of the training system and the purpose of the evaluation. Each evaluator was instructed to begin the training using the web-based prototype and was encouraged to stop at any point to discuss important issues or to ask questions. The analyst recorded the comments separately from the questionnaire. After finishing the training, the analyst stepped though questionnaire answers with the evaluator.

For each evaluator, the analyst recorded the amount of time required to take the training. If an evaluator stopped to ask a question or make a comment during the training, the time was not included in the total training completion time.

For the learning and behavioral evaluation of the training, twelve first-year pediatric residents were observed and recorded giving sign-out twice before and twice after receiving the training. Our methodology for behavioral observation and data analysis is described in detail in [35]. Scores on the pre-training and post-training learning assessments were recorded for each of the twelve first-year pediatric residents. Training completion times were also recorded for each of the residents.

B. Heuristic Evaluation Results

All three heuristic evaluators were able to complete the training within the required time frame of 45–60 minutes (minimum = 45 min., maximum = 60 min.).

With respect to the training content, both pediatric resident heuristic evaluators commended the use of multimedia, particularly the audio examples of sign-out. Both recommended the use of additional audio examples in future versions of the training system. However, one suggested that the use of the History and Physical was “not necessarily helpful”. That is, the History and Physical document did not help her grasp the concept of a “big sentence” as it relates to sign-out. In terms of the “Did You Know Facts”, one of the residents commented that they “were an aspect of the system that provided entertaining, yet informative supplementary information.” In discussing the case studies, one resident found them “helpful”, while the other thought they could be improved by placing the learner in one of the roles from the case study, the goal being to make the case study “more personal and engaging”. In terms of the conceptual organization and length of the training, both residents suggested that the content was “well organized” and could be completed within a “reasonable amount of time”.

With respect to human factors concerns, one resident commented that there was a lag time when opening pop-up windows. None of the evaluators found the navigation of the system to be a problem and all three were satisfied with the general look and feel of the interface.

All of the evaluators provided suggestions in the general comments. Both pediatric resident evaluators listed the audio examples as one of the most useful aspects of the training. Other useful aspects given by the resident evaluators were the organization of the training concepts, and “Did You Know Facts” illustrating what can go wrong in certain situations.

Only the two resident evaluators provided feedback on the least useful aspects of the training. The first suggested that use of generic sign-out statistics in the “Did You Know Facts” was “boring” while the second suggested that there were too many audio examples in the “big sentence” section.

With regard to changing one aspect of the training, the first medical evaluator suggested making the training more “personal” by including additional “pediatric related examples” or by relating the case studies to more “personal experiences” of the residents. The engineering student evaluator suggested minimizing the amount of text on certain screens by including audio-based narration. In terms of emphasizing an existing aspect of the training, both resident evaluators said they would emphasize the training content while the engineering student said he would emphasize the overall user-friendliness of the system.

C. Learning and Behavioral Evaluation Results

Mean training completion time for the first-year pediatric residents was 49 minutes 9 seconds. Only two of the pediatric residents exhibited completion times longer than 60 minutes. In terms of the pre- and post-training learning assessments, overall, residents scored well on both the pre- and post-training assessments (μ_pre = 83% correct, μ_post = 82% correct).

As our training system was not designed to measure behavioral changes influenced by the training, we measured behavioral changes through observational study of residents giving sign-out. Results of our observational sign-out data for the twelve first-year pediatric residents are presented in [35].

VI. Discussion

The initial design and implementation of the web-based sign-out training curriculum has been completed. Concepts are grouped into modules with representations including multimedia content in the form of text, images, audio, and video. Supplemental content is available through pop-up windows, which adds an element of interactivity. In terms of the supplemental content (“Did You Know Facts”), personally relevant facts and data seem to be the most interesting to learners so future versions will focus supplemental content in those areas. By including simple navigation controls we were able to make the training self-paced, and by adding a “table of contents”, achieve basic progress tracking, two elements that will help residents maintain focus when taking the training. Finally, the web-based delivery method provides a way for time- and location-constrained learners, such as residents, to access and take the training from any desired location when it is convenient for them to do so.

The pre-training and post-training assessments include multiple choice and true/false questions, a subset of which contain audio or video content, as well as a basic function for reporting results. The results of these assessments provide a direct way of measuring the acquisition of knowledge due to the training.

The heuristic evaluation identified that in some cases, realistic examples were not necessary and will likely be removed to save time in future versions. For example, some audio files in the “big sentence” section as well as the History and Physical example can be trimmed or made optional. With this content removed, it is likely that the requirement for training length could be shortened- a good idea when the target population is busy residents.

With respect to the implementation, Adobe Captivate provided an acceptable platform for developing the prototype sign-out training system. However, there were limitations. The current version of Adobe Captivate does not provide an internal method for capturing and reporting assessment results. Our workaround solution included sending email to an analyst account but in the long-term, this method is not acceptable. With regard to the assessments, another limitation is the lack of an individualized feedback mechanism. In the current prototype, learners receive feedback only for incorrect answer choices and are not provided with a detailed explanation for a specific response. Adobe Captivate is not capable of creating this critical type of feedback. Captivate also does support user login and tracking. As residents are busy, a better system would allow them to start training, logout and return later to complete the training. The system should allow completion of the training in multiple sittings as necessary. To enable the reporting of results through Adobe Captivate and the log-in and tracking aspects, Captivate training files must interface with a separate learning management system (LMS) which can be a costly and time consuming solution for training designers. Without a LMS, it is likely that a more robust development environment will be necessary to achieve these advanced functions.

Overall, results from the heuristic evaluation of the prototype suggested that our initial design adequately implemented the requirements for the system. While the time to complete the training may have been too long for some busy residents, residents were generally able to complete the training in an acceptable amount of time. While some of the content needed to be redesigned or shortened, heuristic evaluators provided mostly positive feedback. Given the positive feedback from the first round of evaluation, the prototype sign-out training system was implemented in an initial sign-out training study in the Department of Pediatrics at the University of Virginia Health System.

Results of the learning assessments showed that residents generally scored high on both the pre- and post-training assessments. Although surprising, this result shows that the concepts in Phase 1 of the training may be more suited for less experienced learners, particularly students nearing completion of medical school. However, the results of the observational assessment of the residents will help clarify whether residents’ knowledge of the concepts from the training (as shown by their scores on the pre- and post-training learning assessments) is reflected in their sign-out behaviors [35].