Assessment of internal medicine trainee sign-out quality and utilization habits

Abstract

Transfers of care have been associated with adverse events. High quality sign-out may help mitigate this risk. The authors sought to characterize the clinical questions asked of physicians covering patients overnight and to determine the adequacy of current sign-out practice to anticipate inquiries. The authors conducted a prospective, self-report study of interns’ overnight experience at two hospitals. We collected data from novice interns (July 7–August 3, 2010) and experienced interns (March 2–March 29, 2011) in an Internal Medicine residency program. Interns recorded information about overnight inquiries regarding cross-covered patients. For each inquiry about a patient, the intern was asked to record what the situation was about, who initiated the contact, where the intern found the desired information, whether all required data was located, whether the call could have been anticipated by the primary team, if so, whether the call was anticipated, whether the sign-out was sufficient, the time required to address the question, and whether the patient was physically visited. Twenty-one interns (13 novice, 8 experienced) reported 167 overnight inquiries. Most were from nursing staff (87 %) about a wide range of topics, with orders (25 %) and plan of care (20 %) being most common. Trainees used the oral or written sign-out to answer 56 % of inquiries. The proportion of inquiries successfully anticipated (47 % overall) significantly decreased as the academic year progressed (AOR = 0.4, 95 % CI 0.2, 0.8). Trainees rely on sign-out to answer nearly half of overnight inquiries, but the quality of sign-out may decrease over the course of the academic year. The deterioration of sign-out quality from novice to experienced interns and the common use of sign-out as a reference by covering interns suggest continued education, support and oversight by supervising physicians may be beneficial.

Introduction

Transfers of care among physicians in training are frequent—on average at least two a day—and have increased since the advent of work hour regulations in 2003 [1]. Patients now spend more than half of their hospitalization being cared for by covering physicians who are not their primary hospital caregivers [1]. Furthermore, the number and complexity of hospital admissions continue to increase, creating a higher-risk pool of patients [2].

Studies suggest that patients who are cared for by covering physicians have higher adverse event rates [3] and increased delays to treatment [4]. In one study, 15 % of adverse events recalled by residents were directly related to physician transfer of care [5]. Studies at our institution revealed 7.1 sign-out-related problems per 100 patient days [6]. These adverse events can occur when information is lost in transition, when a change in clinical condition is not recognized by a physician unfamiliar with the patient, or when plans of care are made that do not appropriately reflect the patient's history and treatment course. In each case, it is possible that improved communication at the time of transfer would reduce error rates. Furthermore, the Accreditation Council for Graduate Medical Education (ACGME) has adopted standards surrounding transition of care as part of the Common Program Requirements for residency programs [7].

Previous work has shown that information is inadequately conveyed at times of shift-to-shift transfers [8–10]. For example, the clinical condition of the patient was conveyed in only 50 % of the oral sign-outs and 38 % of the written sign-outs at our institution [10]. Similarly, other studies have found sign-out information to be incomplete [9] or medication lists to be inaccurate [8]. However, studies have not investigated what information a covering physician requires when presented with a clinical situation, and where that information is gathered. Consequently, recommendations about sign-out content have been primarily based on expert opinion or adverse event reports [11–17], and the importance of sign-out as a source of overnight information relative to other information sources remains uncertain. Furthermore, there is little evidence about how sign-out skills evolve over the course of an academic year as interns gain clinical experience. With increased clinical experience may come improved ability to provide superior sign-outs including event anticipation. On the other hand, increased clinical experience may lessen reliance on sign-out as a source of guidance and information. We hypothesized that sign-outs later in the year would have more anticipatory guidance and more successfully anticipate inquiries, but that interns might less often rely on them.

In order to create an evidence-based, standardized method to ensure that covering physicians have the information they require, enable better decision-making and potentially decrease adverse events related to transfers of care, it is necessary to understand interns’ overnight experience. Moreover, this information would be useful to the guide development of a formal curriculum for sign-out training. Consequently, we conducted a prospective study of novice and experienced interns during overnight call to characterize the clinical questions interns encountered overnight, where they sought information to address problems, and how often the sign-out anticipated these inquiries.

Methods

Study setting

The study took place on inpatient units in two hospitals, Yale-New Haven Hospital, a 946-bed, urban, academic medical center, and the West Haven Campus of the VA Connecticut Healthcare System, a university-affiliated, 259-bed VA Medical Center in West Haven, CT, USA. Formal sign-out education to incoming interns is provided at both hospitals during the first 3 months of intern year [18]. The same group of house staff rotates between the two hospitals.

Sign-out procedure

Sign-out occurs between 4 pm and 7 pm on inpatient services at both medical centers. The process is similar at both facilities. The team leaving the facility prepares a written document including the patient's name, location, medical record number, and code status. These items are automatically populated from the electronic medical record (EMR) at each institution, Sunrise Clinical Manager (version 5.5 Allscripts, Chicago, IL, USA) at Yale-New Haven Hospital and CPRS (US Department of Veterans Affairs, Washington, DC, USA) at the West Haven VA Medical Center [10, 19]. A medication list, also directly populated from the EMR, is included in the VA document, and can be included in the Yale-New Haven Hospital document at the discretion of the person printing the document. In addition, a free text narrative summation of the patient's medical course and condition is included. A list of ‘To Do’ items is created in a free text box, separate from the narrative summation. Typically, sign-out documentation is prepared by the intern; however, residents do occasionally participate.

After the written document is prepared, the intern leaving the facility has a face-to-face discussion with the in-house physicians covering the patients overnight. At a minimum, two interns meet, supervised by at least one second- or third-year resident. The written document is handed to the covering intern at the start of the discussion and hand-written notes may be placed directly on this document during the discussion.

The next morning, the overnight intern meets in person with the day intern, or at minimum, a phone conversation occurs. The written document is sometimes returned to the day team.

Participants

Interns on general medical services as well as the geriatrics and liver/gastroenterology service were eligible for participation. These services were selected as they all include general medical patient cross-coverage responsibilities.

Throughout the residency program, on-call interns could be responsible for cross coverage of up to 30 patients, non-inclusive of their own panel (up to 10) and concurrent admissions. Cross coverage is assigned based on a paired-team model. Two teams, each consisting of two residents and two interns with one to two attending physicians, share coverage responsibilities amongst the group.

At Yale-New Haven Hospital, an intern-resident pair from one of the two teams is on call each night in a four-night cycle and covers patients from their own team and the paired team. The overnight intern's resident remains in house and involved in the cross coverage and admission activities.

At the VA facility, the same paired-team structure is employed; however, the overnight intern only covers patients from her own team. A night float intern arrives nightly at 7 o'clock to cover as many as 50 patients from the overnight teams’ paired team as well as other, non-overnight teams. The night float intern's duties are limited to cross coverage only. There is a second- or third-year resident in house for each intern. The on-call intern is supervised by the on-call resident and the night float intern is supervised by a night float resident, who is also admitting new patients to the service.

Formal instruction on sign-out is provided to all interns in a 1-h didactic session that occurs multiple times; however, each intern only need to attend once [18]. The sessions all occurred prior to data collection.

Data collection

Two rounds of data collection took place, the first was from July 7 to August 3, 2010 (novice interns) and the second was from March 2 to March 29, 2011 (experienced interns). Each intern participated once during a round of data collection. Interns participating during the early period were allowed to participate in the late period as well. There were more eligible interns in the first data collection period than in the second due to an unequal distribution of interns assigned to 2 versus 4 week blocks. During each data collection period, all participating interns were asked to prospectively record data about their cross cover experience overnight. Each intern was provided with a pocket card and asked to record the data directly on the card, which was collected the next day (Fig. 1).

Fig. 1.

Portion of data collection pocket card

This study was approved by the Human Investigation Committee of Yale University and each intern provided a written consent to participate.

Main measures

For each inquiry about a patient, the intern was asked to record what the situation was about (medications, test results, IV access, NPO status, orders, discharge plan, other), who initiated the contact (nurse, patient, family, consultant, ancillary staff, attending of record, other), where the intern found the desired information (written sign-out, EMR verbal sign-out, paper chart, other), whether all required data was located, whether the call could have been anticipated by the primary team, if so, whether the call was anticipated, whether the sign-out was sufficient, the time required to address the question, and whether the patient was physically visited. Interns were permitted to record multiple answers for the question about whether information was found. We defined inquiries as possible to anticipate if the respondent indicated it was anticipatable, or if it was in fact anticipated. For analysis, one event was defined as a single inquiry directed towards the intern. If multiple inquires occurred during a single conversation, each was treated as a separate event. Illegible data was treated as missing.

Our primary outcome measure for sign-out quality was the successful anticipation of inquiries identified as possible to anticipate. Our main predictor was the time of year. Independent variables included the inquiry topic and the hospital in which the intern was working.

Statistical analysis

Data analysis was performed with SPSS v.19 (Armonk, NY, USA) and SAS 9.2 (SAS Institute, Cary, NC, USA). All analyses were two-tailed and significance was set at p > 0.05.

We characterized the data needs and referencing habits of covering interns by describing the distribution of question topics and the information sources interns used to answer questions, for the data set as a whole and also for each data collection period. A Pearson Chi-square was calculated to assess differences in inquiry reporting and a Student's t test to assess differences in time burden across the data collection periods.

Pearson Chi-square test was used to assess bivariate relationships of predictors with the two main quality outcomes. A separate multivariable logistic regression model was then constructed for each quality outcome including all predictors regardless of significance, based on consensus clinical judgment of the authors. We used the rule of at least ten events per predictor to avoid overfitting our models [20]. Because of small numbers we collapsed questions about results, intravenous access, diet, discharge parameters and other miscellaneous questions into an “other” category; similarly, we dichotomized the question originator variable into nurses and others. As a sensitivity analysis, we repeated the analysis using generalized estimating equations to account for the clustering of inquiries within individual respondents. The results were substantively unchanged and for simplicity we present the logistic regression results.

Results

A total of 13/24 (54 %) novice interns participated in the early round, and 8/14 (57 %) experienced interns participated in the late round. An additional intern consented in the early round but did not complete the study. A total of 167 inquiries were reported, 96 by novice interns, and 71 by experienced interns. There were no differences in inquiries reported per intern, inquiries reported at each site, and time spent per inquiry between the two data collection periods (Table 1).

Table 1.

Characteristics of study cohorts

	1st data collection	2nd data collection	p value
Total intern participants	13	8
Night float participants	2	1
Participation rate	54 %	57 %	0.16
Total inquiries reported	96	71
Mean inquiries per intern (SD)	7.4 (5.5)	8.9 (4.5)	0.53
Mean time per inquiry, min (range)	6.5 (0.5, 40)	6.8 (0.5, 45)	0.86
Inquiries at VA hospital (%)	54 (56.3)	28 (39.4)	0.03

VA veterans administration

Questions asked of the covering physicians were most commonly about orders (25 %) and were most commonly from the nursing staff (87 %). Interns were most likely to reference the written sign-out in answering questions (48 %); they used the written or verbal sign-out to answer 56 % of inquiries (Table 2). Inquiry subjects, the source of the inquiry, and use of resources did not change between the two time periods.

Table 2.

Characteristics of overnight inquiries

	Overall N (%)	1st data collection N (%)	2nd data collection N (%)	p value
Inquiry subject				0.24
Orders	41 (25)	21 (22)	20 (29)
Plan of care	34 (20)	16 (17)	18 (26)
Medication questions	30 (18)	19 (20)	11 (16)
Test/lab results	15 (9)	40(42)	21(30)
Discharge	4 (2)
Diet	3 (2)
Intravenous access	2 (1)
Other	37 (22)
Inquiry originator				0.64
Nurse	145 (87)	82 (86)	63 (89)
Family	7 (4)	13(14)	8(11)
Patient	3 (2)
Consultant	4 (2)
Ancillary staff	4 (2)
Other	3 (2)
Resource referenceda,b
Written sign-out	67 (48)	36 (48)	31 (48)	0.97
Electronic medical record	44 (31)	20 (27)	24 (37)	0.19
Paper chart	4 (3)	2 (3)	2 (3)	0.88
Verbal sign-out	31 (22)	18 (24)	13 (20)	0.57
Other	23 (16)	10 (13)	13 (20)	0.29
Written or Verbal sign-out	78 (56)	46 (61)	32 (49)	0.15
Able to locate all informationc	130 (84)	77 (90)	53 (78)	0.05
Possible to anticipate	117 (71)	66 (70)	51 (73)	0.71
Inquiry was anticipated (n = 117)	55 (47)	38 (58)	17 (33)	0.009
Sufficient sign-outd	101 (69)	60 (76)	41 (60)	0.04
Patient visitede	63 (41)	35 (42)	28 (40)	0.79
Mean total time in min (SD)	6.6 (8.3)	6.5 (8.7)	6.8 (7.9)	0.86

SD standard deviation

^aMultiple answers per inquiry allowed

^b27 missing

^c13 missing

^d20 missing

^e14 missing

A total of 71 % of inquiries were considered possible to anticipate; of these, 47 % were anticipated by the primary team. The proportion of inquiries considered possible to anticipate was similar in both time periods (70 vs. 73 %, p = 0.71), but the proportion of inquiries that were successfully anticipated decreased from the first to the second data collection period, from 38/66 (58 %) to 17/51 (33 %), p = 0.009. This relationship persisted after adjustment for inquiry type and hospital location (OR 0.4, 95 % CI 0.2–0.8, p = 0.01) (Table 3).

Table 3.

Multivariable analysis of anticipated inquiries

	Adjusted OR (95 % CI)	p value
Late vs. early data collection period	0.4 (0.2, 0.8)	0.01
Question topic		0.74
Medications vs. orders	0.7 (0.2, 2.2)
Other vs. orders	0.6 (0.2, 1.8)
Plan of care vs. orders	0.5 (0.2, 1.6)
Academic medical center vs. VA hospital	0.9 (0.4, 2.0)	0.81

OR odds ratio, CI confidence interval, VA Veterans Administration

Discussion

This study underscores the central importance of sign-out to overnight care. Despite a robust EMR presence at both facilities, the written sign-out was still the most-used reference source by the covering intern. Either the written or verbal sign-out was referenced in 56 % of inquiries; by contrast, the advanced EMR was used to respond to only 31 % of inquiries. Nonetheless, interns were only able to locate all desired information for 84 % of inquiries, and sign-outs anticipated only 47 % of inquiries interns felt were predictable. Most surprising, the frequency with which sign-outs predicted overnight inquiries declined over the course of the academic year.

Our findings are similar to those of Borowitz and colleagues [9] who previously found that interns often encounter inquiries overnight for which they feel unprepared, most of which they felt sign-out could have anticipated. Yet interns commonly overestimate the effectiveness of their sign-out [21] and there remains a high degree of variability in the sign-out process [22]. Existing methodology [23] and curricula [12, 15, 18] to standardize the sign-out process are based on expert opinion rather than what interns actually encounter. Nurses may provide additional insight into the topics they feel covering physicians are least able to address.

There are several possible explanations for the findings that sign-out quality decreased over the course of the academic year. Data collection for novice interns began 1 week after sign-out training occurred, and better performance by novice interns may have reflected an early effect of skills training. Conversely, attenuation of training, fatigue and burnout on the part of the daytime interns, a loss of attention to detail, and decreased supervision by senior residents are all possible explanations for a decrease in perceived sign-out quality later in the year. The sophistication of the overnight intern may also explain the findings. Experienced interns may have had more sophisticated expectations for a thorough sign-out, therefore holding sign-out to a higher standard later in the year. This explanation is not, however, fully consistent with the concomitant decrease in inquiries that were anticipated by the day team. If interns gain a better understanding of what ought to be included in sign-out during the course of the year, they should also become better at anticipating overnight inquiries. Yet we found the inverse—overnight inquiries were less likely to be anticipated by the day team. It therefore seems most likely that quality of sign-out did in fact decline over the course of the study. Interestingly, there is some evidence in the psychology literature that may support our findings. Epley and colleagues [16] found that perspective-taking, or understanding the position of the receiving person in a communication, is affected by increasing time pressures. As the academic year progresses, more tasks and more outputs are typically expected of the intern. These additional pressures may be adversely affecting interns’ ability to conduct high quality sign-out.

There are several potential limitations of the study. We had a small number of individual participants, but they reported a large number of inquiries, yielding an overall sample size that was sufficient to draw conclusions about significant differences over time. Assessments were subjective and may have been subject to bias. There was a large amount of missing data, but we have no reason to believe they were not missing at random. The single-center nature and low response rate limit generalizability.

A long-held belief of medical training is that practical experience begets improvement. This tenet has been applied to all aspects of medical practice for generations. Sign-out is now an integral and widely accepted practice that directly influences patient care. This study suggests that, with respect to handoffs, the central assumption linking experience to improvement may be flawed. Sustaining high quality handoffs among trainees may require evaluation, feedback, and reinforcement throughout the year and nursing may be a resource for continuous sign-out improvement. Lastly, these data can be used to better simulate real-life cross-coverage situations and may be beneficial to the educators in residency programs to help improve the quality of sign-out.