Effect on Patient Safety of a Resident Physician Schedule without 24-Hour Shifts

Abstract

BACKGROUND

The effects on patient safety of eliminating extended-duration work shifts for resident physicians remain controversial.

METHODS

We conducted a multicenter, cluster-randomized, crossover trial comparing two schedules for pediatric resident physicians during their intensive care unit (ICU) rotations: extended-duration work schedules that included shifts of 24 hours or more (control schedules) and schedules that eliminated extended shifts and cycled resident physicians through day and night shifts of 16 hours or less (intervention schedules). The primary outcome was serious medical errors made by resident physicians, assessed by intensive surveillance, including direct observation and chart review.

RESULTS

The characteristics of ICU patients during the two work schedules were similar, but resident physician workload, described as the mean (±SD) number of ICU patients per resident physician, was higher during the intervention schedules than during the control schedules (8.8±2.8 vs. 6.7±2.2). Resident physicians made more serious errors during the intervention schedules than during the control schedules (97.1 vs. 79.0 per 1000 patient-days; relative risk, 1.53; 95% confidence interval [CI], 1.37 to 1.72; P<0.001). The number of serious errors unitwide were likewise higher during the intervention schedules (181.3 vs. 131.5 per 1000 patient-days; relative risk, 1.56; 95% CI, 1.43 to 1.71). There was wide variability among sites, however; errors were lower during intervention schedules than during control schedules at one site, rates were similar during the two schedules at two sites, and rates were higher during intervention schedules than during control schedules at three sites. In a secondary analysis that was adjusted for the number of patients per resident physician as a potential confounder, intervention schedules were no longer associated with an increase in errors.

CONCLUSIONS

Contrary to our hypothesis, resident physicians who were randomly assigned to schedules that eliminated extended shifts made more serious errors than resident physicians assigned to schedules with extended shifts, although the effect varied by site. The number of ICU patients cared for by each resident physician was higher during schedules that eliminated extended shifts. (Funded by the National Heart, Lung, and Blood Institute; ROSTERS ClinicalTrials.gov number, NCT02134847.)

Since publication of a study in 1971 showing that sleep-deprived resident physicians made more errors in reading electrocardiograms,¹ a robust literature has accumulated indicating that sleep deprivation adversely affects the alertness and performance of resident physicians.^2–13 In a previous randomized, controlled trial, we found that resident physicians who worked according to a schedule that included frequent shifts of 24 or more consecutive hours (extended-duration work schedule) made 36% more serious medical errors than when they worked a schedule that cycled them through day and night shifts limited to no more than 16 consecutive hours.^11,13

In recent years, policy regarding resident physician work hours has shifted. In 2008, the National Academy of Medicine recommended that resident physicians work no more than 16 consecutive hours without sleep.¹⁴ In 2011, the Accreditation Council for Graduate Medical Education (ACGME) partially acted on this recommendation, prohibiting shifts exceeding 16 consecutive hours for first-year residents.¹⁵ In 2017, the ACGME reversed its policy¹⁶ and again began allowing shifts of 24 to 28 consecutive hours for all resident physicians after the FIRST (Flexibility in Duty Hour Requirements for Surgical Trainees) trial showed that no changes in the incidence of death or serious surgical complications were associated with shift limits among first-year surgical residents, although most of them spend a minority of their time in the operating room.¹⁷ More recently, the iCOMPARE (Individualized Comparative Effectiveness of Models Optimizing Patient Safety and Resident Education) trial also showed no change in mortality among medical patients when shift limits were implemented,¹⁸ although we believe that the power of the iCOMPARE trial was suboptimal.¹⁹

Questions remain as to why the duration of shifts for resident physicians appears to be a major driver of patient safety in some studies and inconsequential in others. Possibly, differing approaches to eliminating extended shifts (e.g., having resident physicians cycle through day and night shifts vs. having them work six consecutive night shifts) have differing effectiveness in promoting resident physician performance.²⁰ Alternatively, poorly managed transitions between shifts (known as handoffs) in some settings could undermine the potential benefits of reducing sleep deprivation in residents.^21–23 A third possibility is that reduced staffing levels²⁴ could counterbalance any benefit to patient safety of reduced work hours in some settings,²⁵ since, contrary to National Academy of Medicine recommendations,¹⁴ the ACGME 2011 work-hour limits were not accompanied by firm workload limits or funding to support increased staffing.

To address these knowledge gaps, we conducted a multicenter, cluster-randomized, crossover trial of the effects on patient safety of implementing a rapidly cycling work roster that eliminated extended shifts. We concurrently captured data on resident physician work schedules, sleep, workload, and other systemic factors.^26,27

METHODS

TRIAL DESIGN

The Randomized Order Safety Trial Evaluating Resident-Physician Schedules (ROSTERS) was a multicenter, cluster-randomized, crossover trial conducted from July 1, 2013, to March 5, 2017, in six pediatric intensive care units (ICUs) across the United States. Trial investigators obtained a certificate of confidentiality from the National Institutes of Health to protect the privacy of the participants, and institutional review board approval was granted. Detailed methods for the trial have been described previously.²⁸ We studied pediatric ICUs because medical errors occur at high rates in critical care settings, and the pediatric ICUs we included were staffed by resident physicians who were second-year and above and thus not subject to the ACGME’s changing policies for first-year residents.^29,30

To be considered for the trial, each participating pediatric ICU was required to have resident physicians who were following a schedule that included extended work shifts at baseline. The frequency of extended shifts varied across sites from every third shift (which required staying overnight in the hospital every fourth night) to every fourth shift (which required staying overnight in the hospital every fifth night); between extended shifts, resident physicians worked shorter day shifts and had occasional days off. This baseline schedule at each site served as the control schedule for our trial. Each ICU had an established handoff process in place at baseline (Table S1 in the Supplementary Appendix, available with the full text of this article at NEJM.org). All patients (except the subgroup of patients cared for during the day primarily by resident physicians working extended shifts) had their care handed off to resident physicians working extended shifts in the evening.

The trial was completed over several years, with each site beginning a 2-year trial at a different time. Sites were paired on the basis of the date they began the trial; one site from each pair was randomly assigned to start with the extended-shift schedule (control schedule), and the other site started with the schedule that eliminated extended shifts (intervention schedule). Each site had a 4-month wash-in interval before data collection began during which resident physicians followed the schedule that was about to be tested. Eight months of data were then collected on this schedule. This interval was followed by another 4-month wash-in interval during which sites crossed over to the other schedule. Then 8 months of data were collected on this second schedule. This design allowed each site to serve as its own control, matched by time of year.

INTERVENTION SCHEDULE DESIGN

During the intervention schedule, resident physicians typically worked a night shift followed by approximately 24 hours off duty, and then two or three consecutive day shifts (depending on the site); this pattern was repeated over the course of a month-long rotation, with occasional additional days off. Specific details about the schedule for each site have been reported previously (Table S2).²⁸ Our objective for the intervention schedule was to eliminate extended-duration (≥24 hours) work shifts and increase the amount of sleep for residents. Owing to substantial site-level differences in unit organization at baseline, sites made individual determinations about how best to organize staffing to accommodate the intervention.²⁸ All patients had their care handed off to night-shift resident physicians in the evening.

STUDY OVERSIGHT

Written informed consent was obtained from resident physicians for the collection of identifiable information (Fig. 1). Families of patients were informed that the trial was being conducted, and the institutional review boards waived informed consent for the collection of patient safety data. Data were reviewed on a regular basis by a data and safety monitoring board. A subgroup of resident physicians also gave written informed consent to provide data on their sleep, work hours, neurobehavioral performance (e.g., on the basis of psychomotor vigilance tasks), and subjective sleepiness, as reported previously.^26–28 The authors vouch for the accuracy and completeness of the data and for the fidelity of trial to the protocol, available at NEJM.org.

Figure 1. Participants and Rotations.

The control schedules included shifts of 24 hours or more; the intervention schedules eliminated extended shifts and cycled residents through day and night shifts of 16 hours or less. Of resident physicians who completed more than 1 rotation, 58 completed 2 rotations, 5 completed 3 rotations, and 3 completed 4 rotations. The 333 resident physicians who completed 410 rotations include 27 participants who rotated through at least one control and one intervention cycle.

COLLECTION OF DATA AND CATEGORIZATION OF SERIOUS MEDICAL ERRORS

We used an intensive data collection and adjudication method to capture and classify adverse events and medical errors.²⁸ This method was used in our earlier trial of resident physician work schedules and patient safety¹¹ and was adapted from a well-tested approach used in multiple studies.^31–34 Categories of errors and adverse events are described in Table 1.

Table 1.

Classification of Errors and Adverse Events.

Term	Definition
Medical error	Any error in the delivery of medical care, whether harmful or trivial
Serious medical error	A medical error that causes harm or has substantial potential to cause harm (i.e., the sum of preventable adverse events plus near misses). Errors with little or no potential for harm are not serious errors, nor are nonpreventable adverse events.
Adverse event	Any injury due to medical management
Nonpreventable adverse event	Injury caused by medical care, without any apparent error
Preventable adverse event	Injury caused by an error in medical management
Near miss	An error in care that has substantial potential to cause harm but does not, either because it is intercepted or because it unexpectedly causes no apparent harm despite reaching the patient
Error with little or no potential for harm	An error in care delivery that is unlikely to injure a patient
Exclusion	An incident detected on initial surveillance that is determined on review to be neither an error nor an adverse event

At each hospital, a team of chart reviewers (nurses) and observers (physicians) who were centrally trained through a series of webinars collected data, which were supplemented by voluntary reports from clinical staff. The team of physician observers followed participating resident physicians around the clock during each schedule, gathering information on any suspected serious errors. Concurrently, research nurses performed chart reviews (generally 5 days a week, with Monday reviews including charts from the weekend) and gathered reports of incidents of suspected serious errors from clinical staff. Incidents were classified as being attributable to resident physicians or to other staff.

Data were collected on electronic forms and securely transferred to the trial data coordinating center. Subsequently, data on all suspected incidents were sent to trained physician reviewers who were unaware of site and schedule and who independently classified each suspected incident (Table 1). Two physicians independently rated each suspected incident, classifying it as an adverse event, near miss, error with little or no potential for harm, or excluded event. Adverse events were further classified according to preventability with the use of a 4-point Likert scale; events were subsequently dichotomized to preventable or nonpreventable incidents. Disagreements were resolved by discussion; pre-discussion interrater reliability was good (weighted kappa score, 0.52 to 0.67).

PATIENTS PER RESIDENT PHYSICIAN

We obtained work rosters for each site. Average hourly resident physician staffing for a 24-hour interval was derived from these rosters, from which an average estimate of daily staffing by resident physicians at each site and for each schedule was determined. The number of ICU patients per resident physician for each site–schedule combination was calculated as the average of the estimates of daily patient census at each site per schedule divided by the average number of resident physicians present daily at each site per schedule.

STATISTICAL ANALYSIS

The unit of analysis for our primary analysis was the rate of serious medical errors (preventable adverse events and near misses) made by resident physicians per admission. In accordance with the prespecified statistical analysis plan, we compared the rates of serious medical errors during one schedule with those during the other schedule using log-link Poisson models, with patient admission to the pediatric ICU as the unit of analysis; with site, period of randomization, and schedule as fixed effects; with robust standard errors to account for potential overdispersion; and with the log of adjusted patient-days at risk as an offset. All patients in the participating units were included in the analysis; there were no dropouts. Adjusted patient-days at risk were estimated, with exclusion of shifts that were not observed, although estimates that did not exclude missed shifts (in sensitivity analyses) were essentially unchanged. Rates are presented as numbers of medical errors per 1000 adjusted patient-days at risk during the two schedules. Secondary outcomes included rates of unitwide serious medical errors. Overall rates as well as site-specific rates are reported. A two-tailed P value (with P<0.05 considered to indicate statistical significance) is reported for the primary outcome in the primary analyses. There was no prespecified plan to account for multiple comparisons; for all analyses other than the primary analyses, point estimates and 95% confidence intervals are reported without P values. Confidence intervals have not been adjusted for multiple comparisons, and inferences drawn from them may not be reproducible.

We conducted secondary analyses comparing the rates of medical errors during the two schedules in which we adjusted for the number of patients per resident physician as a potential confounder, because the number of patients per resident physician was unbalanced between the trial groups. In these analyses, resident physician rotation was used as the unit of analysis, with the log of the length of resident physician rotation as an offset, since this analysis accounted for varying lengths of individual residents’ rotations when we adjusted for workload as a potential confounder. To assess the effects of these potential confounders, we used log-link Poisson regression with robust standard errors. The model included linear and quadratic terms for number of patients per resident physician and for site, schedule, and period. We also assessed variation in the number of patients per resident physician by site and schedule and conducted post hoc analyses to further explore site-related and workload-related effects (Fig. S1).

RESULTS

CHARACTERISTICS OF SHIFTS

In total, 38,821 patient-days (18,749 in the control schedule with extended shifts and 20,072 in the intervention schedule with extended shifts eliminated) were studied, representing 7099 admissions (3508 and 3591, respectively). Resident physicians consented to be directly observed for patient safety data during 413 of 432 rotations (a total of 72,102 hours of observation).

Patient characteristics varied among hospitals but were generally similar between the two schedules (Table 2; site-specific data are shown in Table S3). Unit characteristics differed between schedules; specifically, the mean (±SD) number of patients per resident physician was higher during the intervention schedules than during the control schedules (8.8±2.8 vs. 6.7±2.2) (Table 2).

Table 2.

Patient and ICU Characteristics.^*

Characteristic	Control Schedule	Intervention Schedule
Patients — no.	3,267	3,310
ICU admissions — no.	3,508	3,591
Patient-days — no.	18,749	20,072
Age — yr	7.3±6.7	7.1±6.6
Male sex — no./total no. (%)	1853/3508 (52.8)	1943/3591 (54.1)
Median length of stay (IQR) — days	2 (2–5)	2 (2–5)
Median chronic condition indicator (IQR)†	2 (1–4)	2 (1–4)
ICU patients per resident physician — no.‡	6.7±2.2	8.8±2.8

^*Plus–minus values are means ±SD. The control schedule included shifts of 24 hours or more. The intervention schedule eliminated extended shifts and cycled resident physicians through day and night shifts of 16 hours or less. ICU denotes intensive care unit, and IQR interquartile range.

^†The chronic condition indicator is a marker of a patient’s coexisting conditions, derived from administrative billing codes. Higher numbers indicate the presence of more coded chronic conditions.³⁵

^‡The number of ICU patients per resident physician is calculated as the average census of patients at each site during each schedule divided by the average number of resident physicians present at each site during each schedule.

As reported previously,²⁷ residents’ mean weekly work hours were lower during the intervention schedule than during the control schedule (61.9±4.8 hours vs. 68.4±7.4 hours), and mean weekly sleep hours were greater (52.9±6.0 hours vs. 49.1±5.8 hours). The percentage of 24-hour intervals with fewer than 4 hours of sleep was 25% in the control group and 9% in the intervention group.

SERIOUS MEDICAL ERRORS

Resident physicians made significantly more serious medical errors during the intervention schedules (without extended shifts) than during the control schedules (with extended shifts) (1723 vs. 1268; unadjusted rates, 97.1 vs. 79.0 per 1000 patient-days at risk; adjusted relative risk, 1.53 [95% confidence interval {CI}, 1.37 to 1.72]; P<0.001) (Fig. 2). There were wide discrepancies in the effect of the intervention across sites (Fig. 2). At three sites, resident physicians made more serious errors during the intervention schedule than during the control schedule (adjusted relative risk, 1.51, 2.38, and 5.90); at two sites, there was no difference; and at one site, resident physicians made fewer serious errors during the intervention schedule (adjusted relative risk, 0.24).

Figure 2 (facing page). Serious Errors, Adverse Events, and Near Misses by Site and Schedule.

The control schedules included shifts of 24 hours or more; the intervention schedules eliminated extended shifts and cycled resident physicians through day and night shifts of 16 hours or less. The relative risk is for the intervention schedule as compared with the control schedule. Panel A shows the relative risk of serious medical errors, both resident-physician–related (primary outcome) and unitwide, Panel B the relative risk of preventable adverse events, and Panel C the relative risk of near misses.

INCIDENCE OF ERRORS UNITWIDE

The unitwide incidence of serious errors (including those that involved resident physicians and those that did not) was higher during the intervention schedule than during the control schedule (unadjusted rates, 181.3 vs. 131.5 per 1000 patient-days at risk; adjusted relative risk, 1.56 [95% CI, 1.43 to 1.71]) (Fig. 2). There was wide variability in the incidence of serious errors at the site level (Fig. 2).

RELATIONSHIP BETWEEN WORKLOAD AND PATIENT SAFETY

Wide site-level variability existed in the number of patients per resident physician at baseline, and the degree of change in the number of patients per resident physician with implementation of the intervention schedule also varied among sites. A secondary analysis with resident physician rotation as the unit of analysis and with adjustment for the number of patients per resident physician as a continuous variable showed that the relative risk of a serious error during the intervention schedule as compared with the control schedule was 0.54 (95% CI, 0.35 to 0.85). However, when the number of patients per resident physician was included as a categorical variable, in quartiles and thirds, the relative risk estimate was 0.74 and 1.32, respectively, in the statistical model, which suggests instability of the model. However, in these secondary analyses, there was a substantial interaction between schedule and workload variables, making interpretation of results difficult. In additional post hoc analyses, we observed that at the three sites with the highest number of patients per resident physician at baseline (i.e., with the control schedule), the incidence of medical errors worsened when intervention schedules were implemented; conversely, at the site with the lowestnumber of patients per resident physician at baseline, the incidence of medical errors declined when the intervention schedule was implemented (Fig. S1A). Rates of serious errors made by resident physicians increased with increasing numbers of patients per resident physician (Fig. S1B).

DISCUSSION

Contrary to our hypothesis, introduction of a schedule that eliminated extended shifts for resident physicians in six pediatric ICUs was associated with a significant increase in the rates of serious medical errors. There was substantial site-level variability in the effect of the intervention, however, with three sites having higher incidents of serious medical errors with the schedule that eliminated extended shifts (the intervention schedule) than with the extended-shift schedule (control schedule), one site having fewer serious medical errors with the intervention schedule, and two others having no significant difference in the incidence of serious medical errors between the two schedules. These data were not explained by differences in the demographics or complexity of illness of the patients. However, we noted that hospitals with the highest resident physician workloads had the most negative results with the intervention. Secondary analyses suggested that the results might have been confounded by concurrent increases in workload with the intervention, although this finding should be viewed as exploratory.

Our trial builds on a growing literature evaluating the effects of eliminating extended shifts. Our previous randomized trial^11–13 showed a benefit of eliminating extended shifts, as did a systematic review.³⁶ The more recent FIRST and iCOMPARE trials, by contrast, showed no benefit.^17,18 The FIRST trial, involving surgical programs, did not standardize the manner in which hospitals implemented schedule changes, which made the effects of any particular approach to scheduling unknown. In addition, programmatic data on resident physician workload, patient census, and other variables were not gathered. The iCOMPARE trial, in which internal medicine programs were randomly assigned to allow or prohibit extended shifts, likewise did not specify an approach to eliminating extended shifts.¹⁸

Our current trial adds to this literature in several respects. We focused on a particular approach to the intervention scheduling, which cycled resident physicians through day and night shifts. Patient safety worsened under this schedule. However, we concurrently collected detailed data that allowed us to explore possible reasons for this.

We found that our intervention led to a decrease in weekly work hours and an increase in residents’ hours of sleep.²⁷ In addition, as reported elsewhere, we observed an improvement in residents’ neurobehavioral performance,²⁶ and poorer neurobehavioral performance has been correlated with a higher risk of serious medical errors.²⁶ Since sleep and neurobehavioral performance improved on the intervention schedule as expected, it appears unlikely that the worsening in patient safety was due to worsening fatigue on this schedule.

A possible explanation for the deterioration in patient safety despite improvements in sleep and neurobehavioral performance is the increase in handoff frequency across sites. The number of patients whose care was handed off each evening increased at all six sites during the intervention schedule. However, only three sites had worse patient safety outcomes with the intervention schedule than with the extended-shift schedule, and one had substantially better safety outcomes with the intervention, which suggests that the increase in handoffs overall was unlikely to account for our results. Moreover, in our previous trial, safety improved after extended shifts were eliminated, despite increased handoffs.¹¹ It is possible that handoff processes at some sites might have protected against degradations in safety more effectively than the processes at other sites, but no obvious trends were apparent to support this possibility (Table S1).

Increases in resident physician workload that occurred as programs eliminated 24-hour shifts could account for our findings. There is evidence that when ICU physicians care for more than seven patients per day, patient safety may deteriorate.³⁷ In our previous trial, in which a schedule eliminating extended work shifts (intervention schedule) was shown to be beneficial, an additional resident physician was added to the roster in trial units during months with the intervention schedule (i.e., four resident physicians were in the units during the intervention schedule vs. three during the control schedule), in order to keep the daily workload for resident physicians constant as each resident’s average work hours decreased.^11–13 By contrast, in the current trial, resident physician workload increased overall when the intervention schedule was introduced. In secondary analyses that controlled for the increase in workload, we did not observe increases in errors during the intervention schedule. However, we did not set out to explicitly test the effects of workload on our intervention.

This trial has several limitations. First, although our methods for collecting data on medical errors are well established, measuring and classifying medical errors is an imperfect science. Our primary data collectors were aware of the residents’ schedules. We provided all primary data collectors standardized training to minimize bias and variability in data collection. In addition, all final incident classification was made at a second stage by two independent physicians who were unaware of site and schedule and who classified with good reliability. Despite these measures, some variability in data collection may have occurred across sites, but we believe that this is unlikely to account for our main findings.

Second, although our results suggest that variability in workload may have influenced the intervention, other site-level factors (e.g., unmeasured differences in handoff processes and attending physicians’ supervision or performance) may have influenced these findings. Our workload findings should be viewed as exploratory and tested further in future research, although they raise the possibility that the debate currently playing out in some states regarding health care provider–patient ratios may be germane to physicians as well as to nurses.³⁸

Finally, we studied the effects on patient safety of a specific work schedule in pediatric ICUs. Although our findings may be relevant to other settings, particularly other ICUs, generalizability is uncertain. We found that local systems of care and variation in implementation had a substantial effect on the effectiveness of the intervention schedule.

In this multicenter trial, incidents of harmful medical errors by resident physicians were higher during an intervention schedule that eliminated extended work shifts than during a schedule that included shifts of 24 hours or more. However, the intervention schedule also increased residents’ workload. Residents’ sleep and neurobehavioral performance improved with the intervention,^26,27 as we expected. A decade ago, the National Academy of Medicine¹⁴ recommended that resident physician work-hour reduction should not occur without an investment of resources to support adequate staffing and infrastructure. Excessive work hours degrade patient safety, but so too do excessive workloads and poor handoffs. The results of our trial suggest that future interventions to address the persistent patient safety problems in academic health centers must address and rigorously evaluate all these challenges concurrently.