The Effect of Reducing Maximum Shift Lengths to 16 Hours on Internal Medicine Interns’ Educational Opportunities

Abstract

Purpose

To evaluate educational experiences of internal medicine interns before and after maximum shift lengths were decreased from 30 hours to 16 hours.

Method

The authors compared educational experiences of internal medicine interns at Vanderbilt University Medical Center before (2010, 47 interns) and after (2011, 50 interns) duty hour restrictions were implemented in July 2011. The authors compared number of inpatient encounters, breadth of concepts in notes, exposure to five common presenting problems, procedural experience, and attendance at teaching conferences.

Results

Following the duty hour restrictions, interns cared for more unique patients (mean 118 versus 140 patients per intern, P = .005) and wrote more history and physicals (mean 73 versus 88, P = .005). Documentation included more total concepts after the 16-hour maximum shift implementation, with a 14% increase for history and physicals (338 versus 387, P < .001) and a 10% increase for progress notes (316 versus 349, P < .001). There was no difference in the median number of selected procedures performed (6 versus 6, P = .94). Attendance was higher at the weekly chief resident conference (60% versus 68% of expected attendees, P < .001) but unchanged at morning report conferences (79% versus 78%, P = .49).

Conclusions

Intern clinical exposure did not decrease after implementation of the 16-hour shift length restriction. In fact, interns saw more patients, produced more detailed notes, and attended more conferences following duty hour restrictions.

Medical resident duty hours have been increasingly scrutinized over the past two decades. Following the implementation of Code 405 by the New York Department of Health in 1989, the Accreditation Council for Graduate Medical Education (ACGME) first placed limitations on resident duty hours in July 2003.¹ The Institute of Medicine recommended further restrictions in 2008, citing concern that extended shifts could contribute both to reduced resident well-being and also increased medical errors.² The ACGME further revised duty hour limits to include a maximum of 16 hours per shift for postgraduate year one residents (i.e., interns), effective July 2011.³ These updated limitations generated substantial debate, as they effectively eliminated all overnight calls for interns.

Since the beginning of duty hour reform in the late 1980s, more than 100 studies have evaluated the impact of these progressive restrictions on both patient care and resident outcomes across the spectrum of medical specialties. Several recent systematic reviews demonstrated that these investigations varied widely in methodology but were primarily single-institution, pre–post, or cross-sectional studies.^4–6 While many focused on patient safety outcomes and resident or faculty satisfaction, several focused on resident education metrics — a growing concern in a rapidly changing educational environment.

To date, the methods used to assess the effect of previous duty hour restrictions on resident education have included procedural/operative volume,^7–13 standardized medical knowledge testing,^{7,9,11,14–16} didactic lecture attendance,^17,18 and time spent reading.^19–23 Many of these studies largely relied on self-reported data, which is subject to recall bias, and manual chart review. Most studies did not demonstrate a significant impact on procedural volume or didactic participation. Standardized test scores were largely unchanged or slightly improved after duty hour limitations, and residents generally reported an improvement in time available for reading. Studies using objective data have demonstrated no difference in the number of patients admitted, mean census,²⁴ standardized medical knowledge test scores,²⁵ or didactic lecture attendance. ^26,27

Unintended consequences of duty hour changes, particularly the impact on resident education, are a major concern at academic medical centers (i.e., teaching hospitals). Rigorous evaluation of these changes is critical. A 2005 systematic review noted that there are “no studies that measured actual experience of residents in internal medicine, pediatrics, family medicine.”⁵ In the interim, several studies^4,6,9–15 from different specialties evaluated some factors that affect resident education; however, objective evaluation of internal medicine resident education has lagged relative to other specialties.

Prior to implementation of the 2011 ACGME duty hour restrictions, a survey by the Association of Program Directors in Internal Medicine demonstrated that 79.2% of residency program directors perceived that the quality of the learning environment would somewhat or strongly decrease.²⁸ Another study evaluating the cost of implementing the ACGME rules suggested that educational opportunities may decline due to a shift in the service–learning balance.²⁹ Prior studies have largely relied on resident and faculty perceptions of educational experience, but have not directly measured patient case mix, clinical documentation, or procedure logs. Few studies have evaluated the effects of the comprehensive changes implemented in July 2011.

We utilized an informatics-based approach, allowing near-complete objective capture of resident educational exposures, to evaluate the impact of the new 16-hour limitation on five key areas of intern education: (1) patient volume, (2) note characteristics, (3) exposure to common presenting problems, (4) procedural experience, and (5) structured didactic experiences. We hypothesized that, despite substantial structural changes, there would be no significant difference in any of the five areas.

Method

Setting

We conducted the study at Vanderbilt University Medical Center (VUMC), a 900-bed quaternary care academic medical center located in Nashville, Tennessee. Approval for this study was obtained from the institutional review board.

In April 2011, Vanderbilt transitioned all inpatient internal medicine services from an overnight call system (experienced by the 2010–2011 interns) to a night float system (experienced by the 2011–2012 interns). In the overnight call system, both residents and interns worked up to 30 consecutive hours every fourth day/night; in the night float system residents and interns were scheduled to work 13-hour shifts (16-hour maximum), with night coverage provided by an independent team (see Figure 1). We compared interns’ work and educational exposures during the first six 4-week rotations of 2010 (July 1, 2010 to December 18, 2010) and of 2011 (July 1, 2011 to December 18, 2011). During the 2010 study period, the internal medicine residency program had 121 total residents and 47 interns; during the 2011 study period, the residency had 130 total residents and 50 interns. All data for the 2010 intern group were derived from their work both on primary teams and in cross-coverage; likewise, all data for the 2011 intern group were derived from their work on both day and night teams, as shown in Figure 1. The 2011 interns each completed two weeks, or 12 shifts, of night float during the study period. For patient volume, note characteristics, common presenting problems, and procedural experience, we evaluated only inpatient experiences at Vanderbilt University Medical Center. Although residents included in this study also rotate through the affiliated Veterans Administration (VA) Medical Center, experiences at the VA hospital were excluded due to incomplete capture of interns’ VA experiences. Inpatient rotations at VUMC accounted for approximately two thirds of the time spent in intern rotations for both cohorts.

Figure 1.

Staffing of inpatient medical services at Vanderbilt University Medical Center before and after implementation of the 16-hour shift restriction. The traditional overnight call model utilized prior to July 2011 is depicted in the top panel. The new shift structure utilized after July 1, 2011 is depicted in the bottom panel. Horizontal lines represent shift duration and vertical lines represent patient handoffs.

Evaluating patient volume and note characteristics

VUMC tracks resident clinical exposures via a web-based portfolio system called Learning Portfolio (LP), which is linked to the electronic medical record (EMR) system. As students and residents write clinical notes, these notes are automatically captured by LP and indexed both by patient and trainee. LP uses these notes to create a log of patient experiences.

We evaluated the broad clinical experiences recorded in intern-generated inpatient clinical notes (specifically, admission history and physical and daily progress notes). Typically, these notes are written by only one author. We evaluated all notes written by interns in the two study periods. In 2009, VUMC developed a set of standardized electronic forms to be used for all internal medicine resident notes. In these forms, the history of present illness, subjective, physical exam, and assessment and plan sections have placeholders but do not contain any auto-generated text (i.e., all content must be entered by the note’s author). We restricted our analysis to these notes, which automatically capture all daily notes written by internal medicine trainees.

Natural language processing (NLP) tools were used to process each clinical note to identify biomedical concepts and their corresponding section locations (e.g., “history of present illness,” “cardiovascular exam”) in an automated fashion. These tools, SecTag³⁰ and the KnowledgeMap concept indexer,³¹ use rigorous, score-based methods to clarify ambiguous document phrases using contextual clues (e.g., “CHF” is correctly defined as either “congestive heart failure” or “congenital hepatic fibrosis” based on the context). Concepts are matched to standard terminologies in the Unified Medical Language System, which currently contains > 8 million terms grouped into about 2 million concepts.³² Each concept is also identified by semantic type (e.g., “disease or syndrome,” “clinical drug,” “sign or symptom”). These NLP systems were shown to identify narrow^31,33 and broad³⁴ concepts, including tracking broad themes in trainee notes.³⁵ We evaluated the counts of concepts covered in each note by section (e.g., “history of present illness) as well as by common semantic types (e.g., diseases).

Exposure to common presenting problems

We evaluated intern exposure to five common presenting problems: fever, shortness of breath, chest pain, dysuria, and abdominal pain. These presenting problems have been previously described as measures of clinical competence in work done by Gotterer et al.³⁶ These represent five of twenty-five presenting problems, selected by master clinical teachers and clerkship directors, in which Vanderbilt medical students are expected to demonstrate competence. When compared to the Healthcare Cost and Utilization Project Nationwide Inpatient Sample, these five presenting problems encompassed 16 of the top 25 reasons for hospital admission to internal medicine services.³⁷ To identify these concepts, we created models based on logistic regression analysis that score documents based on specific sets of clinical concepts in certain sections of the clinical notes (e.g., “cough” in a chief complaint), adapted from algorithms evaluated previously.³⁵ Algorithms used in this study were validated by manual physician review; receiver operator curve analysis demonstrated areas that were > 0.80, and each algorithm was tuned to achieve a positive predictive value of 85%.

Procedural experience

We studied five common procedures of sufficient frequency in internal medicine to warrant comparison: arterial line placement, central line placement, thoracentesis, paracentesis, and lumbar puncture. Creation of procedure notes in the EMR is a mandatory part of clinical documentation, and entry of any procedure note automatically creates an entry in the procedure log for the trainee(s) involved.

Didactic opportunities

We evaluated conference attendance with a nametag barcode swipe system at three didactic conferences: morning report conferences at Vanderbilt University Medical Center or the Nashville Veterans Administration Medical Center (each occurring three times weekly) and a weekly noon conference led by the chief residents. Residents are expected to attend all conferences and have an opportunity to register attendance until five minutes after the conference starts. Attendance rates were calculated as the number of actual attendees (as recorded by ID card swipe data) divided by the number of residents expected to attend conference (based on staffing assignments).

Resident duty hours

We also compared individual resident duty hour logs before and after implementation of the 16-hour maximum shift. Duty hours were self-logged by interns in a web-based program separate from the EMR. The average number of hours worked per day was calculated for each intern, and subsequently multiplied by six to calculate weekly work hours (consistent with ACGME regulations requiring one day off in seven). In this way missing entries were not assumed to represent a day off. The available records were assumed to provide a representative but not comprehensive sample of duty hours.

Statistical analysis

We evaluated intern patient volume (including notes written and number of unique patients seen) using a two-sample t-test with equal variances for normally distributed data. Intern procedure counts and NLP-derived exposure data were compared using the Wilcox Rank-Sum test and the binomial test of proportions. We evaluated each procedure as an outcome variable independently and as a sum of total procedures performed by each intern. Teaching conference attendance rates before and after the change in duty hours were analyzed using a binomial test of proportions. Estimated average hours worked per week were analyzed using a two-sample t-test. All analyses were performed using two-tailed tests of association. We performed statistical analyses using STATA 10.1 statistical software (StataCorp, College Station, TX).

Results

Study group characteristics

Baseline intern characteristics from both study groups are summarized in Table 1. All 47 interns from 2010 and 50 interns from 2011 were included in the data analysis of patient volume, note characteristics, exposure to common presenting problems, and procedural experience.

Table 1.

Characteristics of Intern Cohorts From Before (2010) and After (2011) Restriction to a Maximum of 16 Consecutive Hours of Work, Vanderbilt University Medical Center

Characteristic	Class of 2010, control group No. (%)	Class of 2011, intervention group No. (%)
Categorical interns	35 (74)	38 (76)
Preliminary interns	12 (26)	12 (24)
Male	25 (53)	22 (44)
Education
Graduates of U.S. LCME-accredited medical school	45 (96)	47 (94)
International medical graduates	2 (4)	3 (6)
No. of preliminary interns matching to advanced position	12 (100)	12 (100)
Mean USMLE step 1 score	236	239
Mean USMLE step 2 CK score	244	254

^*A total of 47 interns were studied before the restriction; 50 interns were studied afterward

LCME indicates Liaison Committee on Medical Education; ACGME, Accreditation Council for Graduate Medical Education

USMLE indicates United States Medical Licensing Examination

Patient volume

When comparing the first six blocks (24 weeks) of 2010 and 2011, interns cared for more unique patients (a mean of 118 versus a mean of 140 patients per intern, P = .005) after implementation of the 16-hour maximum shift (Table 2). Interns also wrote significantly more history and physicals (a mean of 73 versus a mean of 88 per intern, P = .005) after 16-hour shift restrictions. There was no difference in the number of daily progress notes (a mean of 286 versus a mean of 298 per intern, P = .92) or overall notes (a mean of 360 versus a mean of 376 per intern, P = .51) written by interns during the study periods.

Table 2.

Number of Daily Notes Written By Interns and Number of Unique Patient Encounters Before (2010) and After (2011) Restriction of Interns to a Maximum of 16 Consecutive Hours of Work, Vanderbilt University Medical Center

Clinical volume measure	2010 No. (95% CI)	2011 No. (95% CI)	P value
Total notes written per intern	359.7 (321.9–397.4)	376.2 (342.1–410.3)	0.51
History and physical notes (95% CI)	73.2 (64.9–81.5)	87.5 (81.7–93.3)	0.005
Daily progress notes (95% CI)	286.4 (255.1–317.8)	288.7 (258.6–318.9)	0.92
Unique patients seen per intern (95% CI)	118.3 (106.9–129.8)	140.0 (130.2–149.8)	0.005

^*A total of 47 interns were studied before the restriction; 50 interns were studied afterward.

Notes were captured in a Web-based

Note characteristics

Table 3 shows the differences in clinical note content for history and physicals and progress notes between the two study periods. From the 2010 group, 3,434 history and physicals and 13,447 progress notes were analyzed. From the 2011 group, 4,443 history and physicals and 14,943 progress notes were analyzed. The word length of history and physicals increased (median of 618 words versus median of 711 words, P = 001) after implementation of the 16-hour maximum shift. Likewise, word length of progress notes also increased (median of 528 versus median of 582 words, P = .001). Interns also discussed more total concepts after the 16-hour maximum shift implementation in both history and physicals and progress notes; these increases were by 14% (338 versus 387, P = .001) and 10% (316 versus 349, P = .001), respectively. Interns discussed more unique diseases, signs/symptoms, and medications after the duty hour change; however, these differences may not be clinically meaningful (Table 3).

Table 3.

Differences in Clinical Note Content for History and Physicals, and for Progress Notes, Before (2010) and After (2011) Restriction of Interns to a Maximum of 16 Consecutive Hours of Work, Vanderbilt University Medical Center

	History and Physicals			Progress Notes
Characteristic of documentation	2010 Median (IQR)	2011 Median (IQR)	P value	2010 Median (IQR)	2011 Median (IQR)	P value
Total length in words	618 (503–760)	711 (582–860)	<0.001	528 (425–659)	582 (469–720)	<0.001
Total concepts discussed	338 (271–417)	387 (314–476)	<0.001	316 (253–397)	349 (279–432)	<0.001
Number of concepts (by semantic type)
Diseases	16 (12–21)	18 (13–23)	<0.001	13 (9–18)	14 (9–19)	<0.001
Signs/Symptoms	15 (11–19)	17 (12–21)	<0.001	8 (6–11)	10 (7–13)	<0.001
Medications	17 (11–23)	18 (12–24)	<0.001	26 (18–34)	26 (19–34)	<0.001
Number of concepts (by section of note)
History*	70 (53–94)	79 (60–102)	<0.001	16 (9–30)	19 (11–31)	<0.001
Physical Exam	42 (22–53)	43 (10–61)	0.034	25 (19–33)	32 (24–42)	<0.001
Laboratory	31 (20–44)	29 (20–39)	<0.001	20 (12–25)	20 (14–26)	<0.001
Assessment/Plan	82 (57–114)	101 (74–136)	<0.001	32 (23–45)	32 (23–42)	<0.001

A total of 47 interns were studied before the restriction; 50 interns were studied afterward. The documentation reported in this table was identified by an automated natural language processing tool. From the 2010 group, 3,434 history and physicals and 13,447 progress notes were analyzed. From the 2011 group, 4,443 history and physicals and 14,943 progress notes were analyzed.

To further investigate this effect, we evaluated the time (by hour of day) at which admission history and physical notes were finalized into the EMR (Figure 2). In the overnight call system (2010), the times at which notes were signed were relatively evenly spaced throughout the day. After implementation of the 16-hour maximum shift (2011), most notes were signed near the ends of shifts, 5–7 AM for night shift and 6–8 PM for day shift. The number of concepts per note was higher at almost every hour in 2011 than in 2010, demonstrating that interns on both day and night shifts wrote more in-depth notes and the effect was not limited to just day or night.

Figure 2.

Characteristics of interns’ admission history and physical, by hour of day at Vanderbilt University Medical Center before (2010, 47 interns) and after (2011, 50 interns) duty hours restrictions were implemented. The authors compared the two groups of interns’ work and educational exposures during the first six four-week rotations. Numbers of admission history and physical notes are shown by vertical bars, and the average numbers of biomedical concepts discussed in each note are shown by line graphs.

Exposure to common presenting problems

Interns wrote more notes matching each of the five selected topics after the duty hour change: fever (7% versus 10% of notes, P = .001); shortness of breath (30% versus 37% of notes, P = .001); dysuria (1.3% versus 2.6% of notes, P = .001); chest pain (2.0% versus 4.3% of notes, P = .001); and abdominal pain (1.8% versus 2.6% of notes, P = .024). An admission note could match more than one presenting problem.

Procedural experience

The median number of overall procedures performed per intern did not change significantly after implementation of the 16-hour maximum shift. In the first six blocks of 2010 interns performed a median of six procedures (IQR 3–10) compared to six procedures (IQR 4–8) in the first six blocks of 2011 (P = .94). (“IQR” stands for interquartile range.) The proportion of interns completing at least one central line placement increased significantly after implementation of the 16-hour maximum shift (62% in 2010 versus 82% in 2011, P = 0.03). The proportion of interns completing at least one of each of the other four procedures selected for study did not vary significantly between 2010 and 2011: arterial line placement (23% of interns in 2010 versus 30% in 2011, P = .46); lumbar puncture (62% versus 70%, P = .39); paracentesis (72% versus 78%, P = .52); and thoracentesis (53% versus 58%, P = .63).

Didactic opportunities

Attendance at a weekly noon conference led by the chief residents increased significantly after implementation of the 16-hour maximum shift (60% of expected attendees in 2010 versus 68% in 2011, P = .001). There was no difference in attendance at morning report conferences (79% versus 78%, P = .49).

Resident duty hours

Duty hours logs were available for 44/47 (94%) interns in 2010 and 49/50 (98%) interns in 2011. The average number of weekly duty hours did not change after implementation of the 16-hour maximum shift (mean 69.0 hours/week in 2010 versus 67.8 hours/week in 2011, P = .24).

Discussion

The resident educational environment has changed dramatically over the past two decades. The cornerstone of the July 2011 ACGME duty hour restrictions limited intern shift length to 16 hours or fewer, effectively eliminating overnight call. This reduction in shift length raised significant concerns regarding housestaff training in many specialties, including internal medicine. Our findings demonstrate that clinical exposures for the interns we studied did not decrease after implementing 16-hour shifts. In fact, to our surprise, interns cared for more unique patients, performed more initial workups (history and physicals), wrote longer and more detailed notes, discussed more diseases in their notes, and had higher attendance at a weekly educational conference. Meanwhile, duty hours and procedural volumes did not change. These data suggest that changes in duty hours to eliminate overnight call do not diminish intern patient care opportunities and may actually increase them.

These outcomes may be have been fostered to some degree by the anticipatory changes our institution made within the Department of Medicine to ensure a smooth transition to 16-hour shifts for interns. The conversion from an overnight call system to a night float system was implemented in April 2011 with experienced residents and interns on the inpatient wards. This allowed a three-month period for schedule refinement and troubleshooting before July 2011 when the new ACGME duty hour limit took effect. In addition, resident handover education programs were expanded in anticipation of an increased number of patient handovers (Figure 1). Finally, the housestaff actively led and participated in the redesign of inpatient services, and their feedback was solicited throughout both the design and pilot phases.

Evaluating clinical exposures

Evaluating concept exposure is a new paradigm in the assessment of clinical experiences. As time spent on patient care in residency training has decreased, ensuring that housestaff have adequate exposure to key concepts becomes increasingly important. The informatics-based approach in this study allowed us to assess the breadth of concepts that residents considered in their patient evaluations. We demonstrate that limiting interns to 16-hour shifts resulted in an increase in the number of concepts considered. This was consistent across both day and night shifts, which raises several possible explanations. Interns may be more rested with limited shift lengths and write more thoughtful, comprehensive notes. Alternatively, interns may be writing broader, less focused notes. A third consideration is that interns write more in-depth notes to ensure patient safety in the setting of increased handovers/transitions of care.

Using concept exposure as an evaluative tool in resident education has broad implications. Capturing and evaluating the intellectual development during internship is complex, and it has been suggested that certain aspects of professional development, such as decision-making and procedure skills, may be better evaluated using proficiency-based tools.^38,39 Concept exposure as tracked in trainee notes could be used in concert with standardized testing metrics like in-training and board examinations to more comprehensively evaluate critical thinking in medicine.

Possible limitations

This study should be interpreted in the context of several possible limitations. While there was no difference in average weekly duty hours per intern between the two study periods, we acknowledge the possibility of cohort bias as a limitation of this study. The individuals making up one intern class are different from those in the subsequent year. Correcting for this is challenging, however, as a randomized controlled study is likely not feasible, considering the complex logistics of residency scheduling, especially given current regulations forbidding extended shifts. Also, this is a single-institution study, similar to nearly all prior similar studies.⁴⁰ While this approach minimizes the wide variability of service structure at different institutions, as cross-institution electronic medical records become more commonplace, a multiprogram study may overcome this limitation. In our study, the counts of common presenting problems likely under-represents true exposure, as the algorithm was tuned for specificity instead of sensitivity. Similarly, the median number of procedures performed by each intern over a six month period was lower than expected in both groups. Our methods did not capture procedures performed during inpatient time spent at the VA. Additionally, while procedure notes should be written for any procedure completed, we do not have independent observation to evaluate the fidelity of note capture. For both these measures, the relative sensitivity of the algorithm to identify exposures between different years should be similar. Finally, the system used to track duty hours was a self-reporting program separate from the one that the EMR used to track notes and procedural exposure, and it is possible that clinical experiences analyzed in the study occurred outside of reported duty hours.

Other regulatory changes

While limiting intern shift length to 16 hours was the primary change that we evaluated in this study, several other changes were concomitantly instituted by the ACGME: residents at the postgraduate year two level and above were limited to 24 hours of continuous duty with an additional four hours for patient safety and resident education; residents must have 8 hours between shifts (and should have 10 hours between shifts); and residents must not be scheduled for more than six consecutive nights of night float.³ The structural changes our institution implemented to adhere to the 16-hour restriction also accounted for the other recommended regulatory changes described above.

Conclusions

We found that implementation of the ACGME 16-hour rule change did not negatively impact multiple objective measures of intern clinical exposure for the interns studied. This is one of the first studies to our knowledge to offer a comprehensive, objective assessment of an internal medicine training experience, evaluating multiple spheres of intern education: patient volume, common presenting problems, breadth of concepts considered in patient workups, procedure volume, and conference attendance. Using an informatics-based approach, we provide a novel template to assess concept exposure, a critical measure to track as housestaff training time decreases. Understanding the long-term impact on knowledge base and patient care outcomes will be important in ongoing evaluation of duty hour changes.