Design and Recruitment of the Randomized Order Safety Trial Evaluating Resident-physician Schedules (ROSTERS) Study

Abstract

Introduction:

While the Accreditation Council for Graduate Medical Education limited first year resident-physicians to 16 consecutive work hours from 2011–2017, resident-physicians in their second year or higher were permitted to work up to 28 hours consecutively. This paper describes the Randomized Order Safety Trial Evaluating Resident-physician Schedules (ROSTERS) study, a clustered-randomized crossover clinical trial designed to evaluate the effectiveness of eliminating traditional shifts of 24 hours or longer for second year or higher resident-physicians in pediatric intensive care units (PICUs).

Methods:

ROSTERS was a multi-center non-blinded trial in 6 PICUs at US academic medical centers. The primary aim was to compare patient safety between the extended duration work roster (EDWR), which included shifts ≥24 hours, and a rapidly cycling work roster (RCWR), where shifts were limited to a maximum of 16 hours. Information on potential medical errors was gathered and used for classification by centrally trained physician reviewers who were blinded to the study arm. Secondary aims were to assess the relationship of the study arm to resident-physician sleep duration, work hours and neurobehavioral performance.

Results:

The study involved 6577 patients with a total of 38821 patient days (n=18749 EDWR, n=20072 RCWR). There were 413 resident-physician rotations included in the study (n=203 EDWR, n=210 RCWR). Resident-physician questionnaire data were over 95% complete.

Conclusions:

Results from data collected in the ROSTERS study will be evaluated for the impact of resident-physician schedule roster on patient safety outcomes in PICUs, and will allow for examination of a number of secondary outcome measures.

1. Introduction

Studies conducted in laboratory and occupational settings over the past several decades have established that sleep deficiency and circadian rhythm disruption degrade human alertness and performance [1]. These risk factors are particularly problematic for physicians-in-training, given their long work hours and irregular schedules, as well as the high consequences of medical errors [2]. Beginning in the early 2000s, a series of studies found that first-year resident-physicians [interns; Post Graduate Year 1 (PGY1)] working extended duration work rosters (EDWR) (≥24 hours) made more serious medical errors (SMEs) than those working shifts of ≤16 consecutive hours; moreover, PGY1s working EDWRs suffered more needle stick injuries, and had an increased risk of motor vehicle crashes (MVCs) on the drive home from work [2–8]. In 2009, after a year-long study, the Institute of Medicine concluded that while it remained unclear whether resident-physician sleep deficiency led to patient harm, “the scientific evidence base establishes that human performance begins to deteriorate after 16 hours of wakefulness”[10,11]. They consequently called for the elimination of resident-physician shifts without sleep over 16 consecutive hours.

In response, beginning in July 2011, the Accreditation Council for Graduate Medical Education (ACGME) limited interns to 16 consecutive hours of work; second year (PGY2) and higher resident-physicians were permitted to work up to 28 consecutive hours and 88 hours weekly, averaged over 4 weeks [12]. Despite extensive literature demonstrating the hazards of sleep deprivation, however, questions have persisted about whether the 2011 ACGME standards would be beneficial, as this unfunded mandate increased the number of handovers of care [13] (albeit without negative impact in earlier trials [2]), and may lead to decreases in staffing and physician-patient ratios. The ACGME limits were proposed without advice on how to operationalize such changes, and were generally left to hospital staff and administrators to apply, leading to mixed approaches of implementation. Studies evaluating the effects on the safety of the ACGME’s 2011 standards have shown mixed results [13–17]. Furthermore, the outcomes in these studies may not have been sufficiently sensitive to measure important adverse effects and did not rigorously capture the work hours, sleep, or neurobehavioral performance of the resident-physicians. In addition, most prior studies, have not assessed the effects of roster changes on PGY2 and higher resident-physicians.

The primary goal of the Randomized Order Safety Trial Evaluating Resident-physician Schedules (ROSTERS) study was to assess whether implementation of a schedule that eliminated shifts ≥24 hours for resident-physicians (PGY2+) would result in improved patient safety. The primary outcome was the rate of SMEs, defined as a preventable adverse event or near miss. Specifically, we sought to compare rates of SMEs when resident-physicians worked a traditional EDWR, compared with a rapidly cycling work roster (RCWR), during which shifts were limited to 16 consecutive hours. The novel study design included data from many levels (patient-level, resident-physician-level, hospital unit level). This allowed for assessment of additional secondary aims, including the relationship between schedules and resident-physician outcomes including sleep duration, MVC risk, work-related accidents, depression, quality of work experience, neurobehavioral performance, and sleepiness during tasks.

2. Methods

2.1. Study design

Medical error rates in pediatric intensive care units (PICUs) are among the highest in any hospital setting, making them a leading priority for implementation of safety efforts [18–20]. The ROSTERS study was a two arm multi-center cluster-randomized crossover clinical trial designed to evaluate the effectiveness of a RCWR on patient safety outcomes compared to an EDWR. The ROSTERS trial took place from July 2013 to March 2017 in six academic medical center sites in the United States. The data collected in the study are summarized in Table 1, and include resident-physician-related data [questionnaire data, sleep and work diaries, drive diaries, actigraphically measured objective sleep data, psychomotor vigilance tests (PVTs)], patient data (patient days, comorbid conditions index, gender, age), potential medical errors, physician observer shift data, and study tracking data.

Table 1.

Data collected throughout the study.

Level	Data	Use	Description
Resident-physicians	Baseline questionnaire	Potential covariates	Demographics, lifestyle, medical history, Berlin Sleep Questionnaire, Morningness-Eveningness Questionnaire, commute information.
	End of rotation survey	Secondary outcomes, potential covariates	Sleepiness during tasks, motor vehicle accidents or near misses, needle sticks and other bodily fluid exposures, lifestyle, height, weight, medication use, depression scale, quality of work experience.
	Drive diary	Secondary outcome	Information about drives to and from work.
	Sleep and work diary	Secondary outcome	Information about sleep, naps, and work schedules.
	Psychomotor vigilance task (PVT) testing	Secondary outcome	A 10 minute long test of sustained visual attention. Resident-physicians had tests during one shift per week, at the beginning and end of the shift and every 5 hours in between.
	Actigraphy	Secondary outcome	The actigraph is a small wrist-worn device worn for the duration of the rotation which measures activity and light exposure. It supplies information about sleep, light, activity.
Tracking	Enrollment, screening, withdrawa l forms	Study tracking	Information about rotation, consent type, reason for withdrawal.
	Serious adverse event	Resident-physician safety	Information about serious adverse events that occurred to a resident-physician
Physician observer staff	Observer shift summary	Study tracking	Information about the observer shifts, including date and time, resident- physicians observed, procedures performed by resident-physicians.
	Missed shift summary	Study tracking	Noted when there was no observer working, the patient census during the missed shift, reason why missed.
Event	Potential medical error form	Primary outcome	Date, time, patient ID, source of error, description of possible event, harm level, incident category.
Patient	Patient days log	Used in primary analysis	Admit and discharge dates, number of days spent in the PICU, age and gender of patient.
	ICD-9 or ICD-10 codes	Covariate	Used to create a comorbid conditions index for each patient stay.

Abbreviations: PICU, pediatric intensive care unit; ICD, International Classification of Diseases.

2.1.1. Site requirements

Each site under consideration for study participation was required to have an individual willing to serve as the site principal investigator (PI), a Clinical and Translational Science Award program, and a PICU. Each site under consideration also had to have a current resident-physician work schedule of ≥24 hour shifts for PGY2s and PGY3s. To be eligible for participation, each site was required to obtain letters of support from administrative officials with decision-making authority, such as the president of the hospital, the chairman of pediatrics, the director of the PICU, the residency program director and a relevant dean, as appropriate, stating their willingness to implement the RCWR schedule. Many academic medical centers nationwide were not eligible to participate, as they had previously eliminated shifts over 16 hours for resident-physicians working in their PICUs. A convenience sample of seven sites under consideration met all of the entry criteria.

2.1.2. Participating sites and institutions

Ultimately, six sites participated in the study: Boston Children’s Hospital; Children’s Hospital Colorado; University of Iowa Stead Family Children’s Hospital; Seattle Children’s Hospital; Cincinnati Children’s Hospital Medical Center; and University of Virginia Children’s Hospital. In presenting results, sites have been de-identified by assigning a reference letter (A through F) at random. In addition to the six sites, a Clinical Coordinating Center (CCC) was established at Brigham and Women’s Hospital (Boston, MA) and a Data Coordinating Center (DCC) was established at the San Francisco Coordinating Center at California Pacific Medical Center Research Institute (a Sutter Health affiliate, San Francisco, CA). The CCC was responsible for study design, site selection, protocol development and implementation, Institutional Review Board (IRB) management, site coordination, patient safety, and staff training. The DCC was responsible for developing standard operating procedures for data collection, processing and editing data, generating reports tracking data quality and completeness, statistical analysis and report generation for the Data Safety and Monitoring Board (DSMB) and statistical analysis of the primary aim of the study.

2.1.3. Site pairing and schedule design

The sites were assigned to pairs and the study was conducted in three waves to spread out the data collection efforts over the 5-year funding period. (Table 2) In each wave, PICUs were randomly assigned to start out with either the EDWR (i.e., overnight shifts of 24–28 hours scheduled every 4 or 5 nights, with shorter day shifts between) or to the RCWR (i.e., resident-physicians were limited to 16 consecutive work hours with night shifts scheduled every four to five days, sleep before night shifts was encouraged, time off after night shifts was arranged to allow for recovery sleep). Prior to data collection, there was a 4-month wash-in interval to allow the PICU to become accustomed to the schedule, after which 8 months of data collection took place. After these 8 months, the PICU crossed-over to the alternate schedule. Another 4-month wash-in interval followed, after which 8 months of data collection took place again (Table 2). The two sites within each wave gathered data during the same time of year to account for any seasonal differences in patient load. The nature of this study made it impossible to blind investigators, resident-physicians and study staff to schedule assignment, although all CCC staff and site PIs were blinded to interim analyses.

Table 2.

Study timeline by site and schedule type.

			Study Dates
Wave	Site	Schedule	Wash in	Data Collection	Days of data collection	Daysb without a Physician Observer
1	A	EDWR	July 2013–October 2013	November 2013–June 2014	234	59.3c
		RCWR	July 2014–October 2014	November 2014–June 2015	240	42.5
1	B	RCWR	July 2013–October 2013	November 2013–June 2014	221	14.1
		EDWR	July 2014–October 2014	November 2014–June 2015	224	19.2
2	C	EDWR	September 2014–December 2014	January 2015–September 2015	246	36.5
		RCWR	September 2015–December 2015	January 2016–September 2016	253	38.4
2	E	RCWR	March 2014–June 2014	July 2014–March 2015	244	42.2c
		EDWR	March 2015–June 2015	July 2015–March 2016	245	45.6
3	D	EDWR	February 2015–May 2015	June 2015–January 2016	251	23.1c
		RCWR	March 2016–June 2016	July 2016–March 2017a	226	0.5
3	F	RCWR	February 2015–May 2015	June 2015–January 2016	242	9.1c
		EDWR	February 2016–May 2016	June 2016–January 2017	244	14.5

^aBreak in data collection from 12/22/16 to 1/13/17.

^bTime missed was summed, then divided by 24.

^cp<0.05 comparing the schedule within site.

Abbreviations: EDWR, extended duration work roster; RCWR, rapidly cycling work roster.

2.1.4. Sample size planning

Based on the previously conducted Intern Sleep and Patient Safety Study [2], during which 2,203 patient-days were accrued under both schedules over a total of 8.5 months at a single center with 2 ICUs totaling 20 beds, we estimated that approximately 46,080 patient-days would be accrued across the six sites in this study. Assuming that the error rates per-patient day on the EDWR were the same as those observed in the Intern Sleep and Patient Safety Study (136.0 per 1000 patient days), this current study would provide 90% power to detect relative rate reductions on the RCWR of 19.8% (a priori assuming a Poisson distribution) for the primary outcome of resident-physician-related SMEs. The actual reductions seen in the preliminary study (22%) were larger than this threshold. Thus, we expected to be adequately powered to draw definitive conclusions regarding the effectiveness of the RCWR.

2.2. Enrollment and resident-physician participation

Site PIs at each academic medical center made presentations to the pediatric resident-physicians describing the study and requesting voluntary participation. Incoming second-year resident-physicians were contacted about participation during their first year, and incoming third-year resident-physicians during their second year. All second- and third-year resident-physicians were invited to enroll in the study. Resident-physicians were given information about the data which would be collected, potential risks and benefits of participation, the timeline for participation, and offered an incentive (e.g., iPad® or cash equivalent) for participation. The duration of each resident-physician’s rotation was approximately one month. Resident-physicians could complete multiple rotations in the PICU. Therefore, they were allowed to enroll in the study multiple times, which makes statistical analyses more complex but is representative of the true work environment of the PICU. Resident-physicians could agree to full participation in the study (i.e., monitoring of sleep, work hours, and performance data collected as well as direct observation) or observation only, which meant they agreed to be directly observed by a physician observer while working in the PICU but did not agree to collection of other personal data. Resident-physicians could further indicate consent for the collection of salivary samples for DNA extraction. All resident-physicians in the PICU were assigned the work schedule under evaluation, regardless of their participation in the study. Data were collected on all PICU patients who were cared for by resident-physicians, regardless of the resident-physician’s participation in the study.

The IRB at each academic medical center, as well as at Sutter Health (DCC) and Partners Human Research Committee (CCC), approved the study. Data use agreements were established between each site and the DCC, and the trial was registered on ClinicalTrials.gov (NCT02134847). A DSMB was established and approved the study protocol prior to data collection. Study investigators obtained a Certificate of Confidentiality from the National Institutes of Health to protect the privacy of research subjects. Each site obtained a waiver of informed consent for the collection of patient data, and written informed consent was obtained from all resident-physicians enrolled in the study. PICU patients and their families were provided an information sheet with details of the study and were not considered as study participants.

2.3. Schedule implementation

On the EDWR, five of the six sites worked a 4 day rotation schedule consisting of 2 day shifts of approximately 12 hours long, followed by one overnight shift that started in the morning one day, ending in the morning the next day (about 28 hours long, Table 3). The PICU at one site worked a 5 day rotation with 2 day shifts of about 12 hours long, a day off, then one shift starting at 11:00AM that was about 24 hours long.

Table 3.

Summary of study implementation by site and schedule.

		Example of Shift Types
		Day 1			Day 2			Extended (day 3/4)a
Site	Schedule	Start	End	Length (hrs)	Start	End	Length (hrs)	Start	End	Length (hrs)
Aa	EDWR	6:00	18:00	12	6:00	18:00	12	11:00	11:00	24
	RCWR	6:00	19:00	13	6:00	19:00	13	18:00	10:00	16
B	EDWR	6:00	18:00	12	6:00	18:00	12 (or 6)	6:00	10:00	28
	RCWR	6:00	20:00	14 (or 6)	6:00	17:00	11 (or 14)	18:00	10:00	16
C	EDWR	6:00	18:00	12	6:00	14:00	8	6:00	10:00	28
	RCWR	7:00	20:00	13	6:00	18:00	12	19:00	11:00	16
D	EDWR	6:00	18:00	12	6:00	18:00	12	6:00	10:00	28
	RCWR	6:00	19:00	13 (or 14)	6:00	17:00	11 (or 6)	18:00	10:00	16
E	EDWR	7:00	18:00	11	7:00	18:00	11	7:00	8:00	25
	RCWR	7:00	21:00	14	7:00	18:00	11	19:00	11:00	16
F	EDWR	6:00	15:00	9 (or 8)	6:00	15:00	9 (or 8)	6:00	10:00	28
	RCWR	6:00	21:00	15 (or 8)	6:00	21:00	15 (or 8)	20:00	12:00	16

^aHad a 5 day rotation, with day 3 scheduled as a day off, so the extended shift is day 4/5.

Abbreviations: EDWR, extended duration work roster; RCWR, rapidly cycling work roster.

During the RCWR, resident-physicians at five of the six sites were scheduled to work in a sequence of shifts in a repeating four day cycle. The approximate schedule was 2 days shifts (11–15 hours long) and one 16 hour long overnight shift that started in the evening and ended the next morning. The PICU at one site worked a 5 day rotation with 2 day shifts of about 13 hours long, a day off, then one shift starting at 6:00PM that was about 16 hours long.

When work shifts are shortened, by necessity patient care is transferred between physicians more frequently [2, 21]. PICUs were asked to use a structured handover of care during the evening shift change. This was in addition to the daily morning rounds, and the structure of the evening handover was not proscribed, varying from site to site. A fellow or faculty member typically oversaw the evening handover of care, which usually occurred between 8:00PM and 9:00PM.

The daily patient census was derived from the data collected on the patient days log form. The average daily resident-physicians present on unit was estimated using resident-physician schedules provided by each site, calculated during daytime hours between rounds and evening handover of care. ICU patients per resident-physician (IPRP) was calculated as the average daily patient census over the average resident-physicians present on the unit. These IPRPs varied considerably among PICUs, both on the EDWR and RCWR. On the EDWR, the mean daily resident-physicians on unit varied from 1.6 to 2.9, with IPRPs ranging from 4.1 ± 0.1 to 10.0 ± 0.2. In contrast, on the RCWR, the mean daily resident-physicians on unit varied from 1.3 to 3.0, with IPRPs ranging from 5.0 ± 0.1 to 13.0 ± 0.2.

With implementation of the RCWR, each PICU was asked to adjust its staffing as needed to preserve adequate staffing while implementing the reduced-hours schedule, but the manner in which each site adjusted staffing to accommodate the scheduling intervention varied. Daytime resident-physician staffing fell on many but not all PICUs, and workload as measured by IPRPs increased at all sites. To offset this, at least in part, some PICUs reallocated fellow-physician time to increase fellow supervision on the PICUs, or increased the use of nurse practitioners to cover some of the workload formerly handled by resident-physicians.

2.4. Site study staff

The research efforts at each participating academic medical center required, at a minimum, a site PI, a research nurse, five physician observers and a research assistant. The site PI and/or research assistant were responsible for obtaining IRB approval for the study and working out details of the RCWR implementation. They also were responsible for initial outreach to potential resident-physician enrollees.

Research assistants collected a saliva sample. Research assistants also provided instruction to the resident-physicians on the completion of daily sleep/work and driving diaries, examined diaries to ensure there were no inappropriate overlaps (e.g., a drive reported during a sleep episode) and that there was no missing information. The research assistants also provided demonstrations on the use of the actigraphs and PVT testing, and downloaded and transferred this data from the site to the DCC for processing.

Physician observers shadowed participating physician-residents and reported potential medical errors, and were responsible for collecting additional data on any other potential medical errors they observed while on the PICU. Physician observers were scheduled to ensure that at least one was on the PICU at all times. Each site developed an observation schedule to ensure that all resident-physicians were observed for approximately equal amounts of time during their rotations.

The research nurse was responsible for following up on and adding any additional detail to the potential medical errors identified by the physician observers, and completing forms for potential medical errors identified by other staff, found in the medical records or otherwise identified or recorded in the hospital database. The research nurse was also responsible for maintaining a patient log that tracked the number of patient days on the PICU during the trial.

2.5. Training

The CCC hosted webinars for the physician observers and research nurses during the wash-in intervals for each study wave. The webinars provided instruction on how to identify, record and classify potential medical errors. The training covered general information on patient safety, definitions of terms and reviewed a set of test cases. Links to the webinars were posted to the study website as a reference and a tool for new staff. The research assistant, research nurse and physician observers were trained by the DCC on the use of data collection forms and the study website, where they could address data edits, update data, and access study resources. The research assistant was also trained in the operation of the actigraph and PVT. Training was conducted by webinars or onsite as feasible. Site visits were performed by CCC and DCC staff on an as-needed basis to ensure correct implementation of the protocols.

2.6. Data collection

The ROSTERS study website and data system were designed and managed by the DCC. Electronic data entry and internet technology were utilized to provide real-time data access. Study data other than the diaries, actigraphy and PVT data were collected via REDCap™ and submitted electronically using online data collection instruments, accessed via a tablet, laptop/PC or smart phone [22]. The data collection forms were designed to prevent missing data, skip pattern errors, and inconsistent and out of range responses. Study data were subject to further daily error-checking programs following submission. Select research staff had permissions to resolve edits and update data as necessary via the website. These changes to the data were recorded in an audit trail. The study website was password protected and only accessible with a direct link. This website listed forms that were expected to have been submitted, which helped ensure data completeness. Diaries were collected electronically using a software program created by the CCC. Numerous reports were generated monthly to summarize data completeness and timeliness, track observer shift coverage, recruitment, event adjudication progress, and diary completeness. Monthly Steering Committee and Quality Assurance conference calls were held to review these reports with all study investigators and research assistants. This allowed for communication between all sites across the country, and helped to identify potential problems or site performance issues and determine solutions.

Resident-physicians were provided instructions on the collection of all resident-physician-level data entered. Resident-physicians entered responses to a baseline questionnaire within a week of the start of their rotation and an end-of-rotation survey within a week of the end of their rotation. This was done online on a secure resident-physician portal website. Resident-physicians also maintained an electronic sleep and work diary, completed daily. Drive diaries were completed online after each drive to or from work. Site staff provided notification via email or phone call as necessary to remind resident-physicians to complete these diaries.

Site staff entered data for resident-physician enrollment, actigraph tracking, PVT test information, and patient admission and discharge dates plus length of PICU stay. Site staff had two encrypted, password protected tablets for physician observers to collect data. These were linked to online data collection forms. Physician observers also recorded information about their shifts, and noted when a shift was missed.

2.7. Resident-physician safety

Weekly reports regarding resident-physician safety were generated, and included information on drowsy driving and accidental exposures (i.e. needle sticks, body fluid exposures). For each rotation, if a resident-physician had any MVCs, had two or more near misses while driving, fell asleep while driving, or had two or more episodes of drowsy driving the PI at the site discussed other transportation options or precautions to stay alert while driving. If a resident-physician had any accidental exposures, the PI insured that the resident-physician had followed hospital protocol for exposure, and discussed how to minimize subsequent exposures. If thoughts of suicide were reported, the PI discussed options for mental health counseling within 48 hours of the initial report. Sites reported serious adverse events (SAEs) that occurred to a resident-physician to the DCC within 24 hours and complied with their IRB reporting requirements.

2.8. Data safety and monitoring board (DSMB) and interim analyses

The DSMB was responsible for safeguarding the interests of study participants, assessing the safety and validity of study procedures, and for monitoring the overall conduct of the study and outcomes data. The DSMB members were an independent advisory group to the National Heart, Lung, and Blood Institute (NHLBI) Director and were required to provide recommendations about starting, continuing, and stopping the study. Strict procedures were in place to avoid any member conflict of interest, which was updated every year and at the start of each DSMB meeting. The DSMB was also asked to make recommendations, as appropriate, to the NHLBI about other aspects of the study. The study investigators did not communicate with DSMB members about the study directly, except when making presentations or responding to questions at DSMB meetings. The DSMB met approximately twice per year, and the DSMB recommended that the study continue at all interim meetings.

All interim and final analyses on the endpoints of resident-physician-related and unit-wide SMEs were performed by an unblinded statistician at the DCC. Interim analyses occurred after the completion of the first wave, and again after completion of the second and third wave. These results were reviewed by the DSMB for assessing benefit or harm. Lan-DeMets methods were used for defining symmetric stopping boundaries [23]. The DSMB was to consider stopping the study early for efficacy, safety or futility.

2.9. Classification of the primary outcome

All clinical types of potential medical errors were included in the study, including medication-related, procedure-related, diagnostic test-related, therapy-related, and nosocomial infections, among others.

An intensive four-pronged approach was used to gather data on potential medical errors in a prospective manner. A team of physician observers conducted direct observation of resident-physicians, documenting all potential medical errors. Forms were made available for voluntary reporting of possible errors by nurses, residents, and other clinic staff. Formal hospital incident reports were collected and charts were reviewed. Information collected for each potential medical error included National Coordinating Council for Medication Error Reporting (NCCMERP) Index harm level, whether it was preventable, clinical category of the incident, and whether the potential medical error was made by a resident-physician. A detailed narrative description of the potential medical error was also gathered.

After these prospective data on potential medical errors were collected, final classification of all incidents was subsequently carried out by physician reviewers, who underwent centralized training. All potential medical errors were assigned by the DCC for examination by two physician reviewers, with 4 different pairings of reviewers. Physician reviewers received all information about the incident, including the harm level and whether the event was preventable based on the initial report. Physician reviewers also received information about the patient (age, gender, length of stay), and were blinded as to whether the incident happened during the EDWR or RCWR arm of the study.

Physician reviewers classified each potential medical error as one of the following:

1. adverse event/harm, defined as any harm due to medical care (or the absence of medical care)

2. A “near miss” (aka a potential adverse event), an error with potential for harm (error defined as something that goes wrong in the care delivery process, whether it causes harm or not)

3. error with little or no potential for harm

4. exclusion, defined as any incident reported by site staff that does not meet one of the above definitions (e.g., neither an adverse event nor an error)

If classified as adverse event/harm, the reviewer further classified the incident for harm level and whether or not the incident was preventable, as:

Harm level: (according to the modified NCC-MERP scale):

• E. Temporary harm to the patient

• F. Temporary harm to the patient and required prolonged hospitalization

• G. Permanent patient harm

• H. Intervention required to sustain life

• I. Patient death

Was the incident preventable?

1. Definitely preventable

2. Probably preventable

3. Probably not preventable

4. Definitely not preventable

Discordant initial reviews were resolved via periodic teleconferences of paired reviewers. Agreement of the physician reviewers was examined on a monthly basis. Agreement of the initial reviews for event classification were moderate (>0.4 to 0.6) to substantial (>0.6 to 0.8). The weighted kappa (95% confidence interval) for the initial reviews of the 4 reviewer pairs were 0.59 (0.55, 0.63), 0.61 (0.58, 0.64), 0.67 (0.66, 0.69), and 0.52 (0.48, 0.56).

The primary outcome was the rate per 1000-patient days of resident-physician-related SMEs, with SME defined as a potential medical error classified as an adverse event determined to be definitely or probably preventable or a near miss. These SMEs were classified as resident-physician-related if the study staff member who initially reported the potential medical error (physician observer or research nurse) noted that the position of the provider who made the error was a resident-physician. A secondary outcome was the rate per 1000-patient days of all unit-wide SMEs, i.e., those that involved resident-physicians and those that did not.

2.10. Deviations from protocol or changes to methods after implementation

The resident-physician enrollment option of “observation only” was added during the first wave to allow resident-physicians to enroll in the study who were willing to be shadowed by physician observers but did not want to provide any other resident-physician-specific data, e.g., sleep/work diaries, actigraphy, or performance testing.

The tracking of missed physician observer shifts was added during wave 1, schedule 1 in July 2014. The missed shifts from wave 1, schedule 1 had to be entered retrospectively.

All data needed to compute IPRP was not collected in a database during the course of the study. The average daily resident-physicians on unit was estimated retrospectively, allowing only for a crude estimate per site and schedule rather than an estimate for each day of the schedule. This estimate did not include other ICU staff caregivers.

One of the original sites that had planned to participate in wave 2 of the study had a change of leadership before embarking on the study, and the new leadership did not support proceeding with the study. Another hospital was selected as the replacement site, but this change delayed the expected start time by 6 months (Table 2).

Although the sites were required to have a research nurse on staff, one site did not have this position filled for most of the first schedule (their EDWR arm). Other sites fell below the minimum of five physician observers at times. While each site was meant to have coverage from a physician observer at all times, there were some exceptions on a day to day basis and systematically some sites did not schedule coverage for some holidays. The lack of coverage of physician observers was tracked.

One PICU underwent a renovation during the first six months of their schedule 2 (their RCWR arm). During the renovation, the PICU beds were divided between two locations. Data collection was put on hold for approximately 3 weeks during this schedule over the winter holidays and while the PICU moved back into its renovated space. Schedule 2 was extended to account for the break in data collection. Prior to schedule 2, study investigators from the DCC and CCC visited the site to determine the impact of the PICU move, and felt satisfied that the study could proceed through the renovation period. The form capturing data regarding potential medical errors was modified to indicate the specific location of the event to allow investigators to later explore if the location of the temporary beds had an impact on potential medical error rates.

Drowsiness while driving to and from work was initially planned to be measured objectively using non-invasive, validated technology that uses infrared oculography to monitor alertness continuously. Resident-physicians who wore prescription eye glasses or had a commute shorter than 20 minutes were ineligible for this measurement. Too many resident-physicians met these exclusion criteria (75%), so use of this technology was discontinued during wave 2 and the drive diary was used to assess self-reported drowsiness while driving to and from work.

Initially, there were nine physician reviewer pairs who were to classify the potential medical errors. These physician reviewers were selected from each of the six sites. The larger number of reviewer pairs made it more difficult to manage data classification and to ensure timeliness and consistency. The levels of agreement of initial reviews within these nine reviewer pairs was low (weighted kappas in the 0.3 to 0.4 range). In December 2015, a new system for review was implemented. A set of four dedicated reviewers, all located in Boston, were trained. These reviewers re-adjudicated all prior potential medical error reviews as well as all new incoming potential medical errors. The agreement between reviewers improved (weighted kappas 0.52 to 0.67).

3. Results

There were 413 individual resident-physician rotations included in data collection during the study, exceeding the goal of 300 (Figure 1). The majority of resident-physicians agreed to full participation in the study (n=380 rotations covered), and 33 (8%) rotations covered were observation only. These 413 rotations had 336 unique resident-physicians, with 58 enrolled twice, five enrolled three times and three enrolled four times (Table 4). The average rotation length was 28.2 days. Within site, the average rotation length did differ by schedule for three sites (p<0.05) but did not differ when all sites were combined (Table 5). The resident-physicians who participated were on average 29.4 ± 2.3 years of age, and predominantly white (81.2%) and female (62.3%) (Table 6). The majority noted their specialty as pediatrics (86.4%), and most (64.7%) were in their second year of residency. The nineteen resident-physicians who did not agree to participate were similar to those in the study in age, gender, race, ethnicity and specialty. Those who refused participation were less likely to be in their third year of residency than those who participated (10.5% vs. 35.3%, respectively, p=0.03).

Figure 1.

Resident-physician Rotation Recruitment and Data Completeness

^a Only required of those who drove to or from work.

Table 4.

Description of resident-physicians enrolled in the study multiple times.

	Total	A	B	C	D	E	F
Total Rotations	413	74	57	71	76	68	67
Repeat enrollments, n (%)	77 (19)	16 (22)	3 (5)	8 (11)	13 (17)	30 (44)	7 (11)
Unique resident-physicians	336	58	54	63	63	38	60
Unique resident-physicians enrolled >1 time	66	16	3	8	12	20	7
Times resident-physicians enrolled
1	270	42	51	55	51	18	53
2	58	16	3	8	11	13	7
3	5	0	0	0	1	4	0
4	3	0	0	0	0	3	0
Post Graduate Year (if known), n(%)a
With the same PGY	26 (39)	15 (94)	2 (67)	0	2 (17)	7 (35)	0
With a different PGY	32 (49)	0	0	8 (100)	6 (50)	12 (60)	6 (86)
By Schedule, n (%)a
With different schedules	27 (41)	0	0	3 (38)	7 (58)	13 (65)	4 (57)
All EDWR schedule	23 (35)	7 (44)	3 (100)	4 (50)	4 (33)	3 (15)	2 (29)
All RCWR schedule	16 (24)	9 (56)	0	1 (12)	1 (8)	4 (20)	1 (14)
Participation type, n (%)a
All full participation	58 (88)	15 (94)	2 (67)	8 (100)	8 (67)	19 (95)	6 (86)
All observation only	2 (3)	0	1 (33)	0	0	1 (5)	0
Both types	6 (9)	1 (6)	0	0	4 (33)	0	1 (14)

Abbreviations: PGY, post graduate year; EDWR, extended duration work roster; RCWR, rapidly cycling work roster.

^aFor those n=66 resident-physicians with repeat enrollment.

Abbreviations: PGY, post graduate year; EDWR, extended duration work roster; RCWR, rapidly cycling work roster.

Table 5.

Recruitment by site and schedule.

		Rotations	Rotations	Rotation
		Screened	Enrolled	Length
Site	Schedule	N	N (%)	mean ± SD
A	EDWR	36	31 (86.1)a	29.9 ± 1.9b
	RCWR	43	43 (100)	28.0 ± 4.2
B	EDWR	32	32 (100)a	27.8 ± 0.4b
	RCWR	37	25 (67.6)	28.8 ± 1.9
C	EDWR	36	35 (97.2)	28.6 ± 2.7
	RCWR	36	36 (100)	28.3 ± 3.4
D	EDWR	37	37 (100)	26.5 ± 4.8
	RCWR	39	39 (100)	24.1 ± 7.2
E	EDWR	34	34 (100)	27.3 ± 1.8b
	RCWR	35	34 (97.1)	28.9 ± 2.2
F	EDWR	34	34 (100)	30.9 ± 2.9
	RCWR	33	33 (100)	30.7 ± 2.1
All	EDWR	209	203 (97.1)	28.5 ± 3.2
Sites	RCWR	223	210 (94.2)	28.0 ± 4.6
Total		432	413 (95.6)	28.2 ± 3.9

^ap<0.05 comparing refusal rate between schedules (chi-square test or a Fisher’s exact test).

^bp<0.05 comparing rotation length between schedules (t-test).

Abbreviations: EDWR, extended duration work roster; RCWR, rapidly cycling work roster; SD, standard deviation.

Table 6.

Resident-physician rotation characteristics by participation.

	Refused	Enrolled	P-value
Characteristic	(N=19)	(N=380)
Gender			0.59
Female	13 (68.4)	233 (62.3)
Male	6 (31.6)	141 (37.7)
Age, years	30.4 ± 4.4	29.4 ± 2.3	0.54
Race			0.10
White	7 (70.0)	303 (81.2)
Black	2 (20.0)	15 (4.0)
Asian	0	31 (8.3)
More than one race	1 (10.0)	15 (4.0)
Other	0	9 (2.4)
Ethnicity			0.46
Hispanic/Latino	1 (111)	23 (6.4)
Not Hispanic/Latino	8 (88.9)	338 (93.6)
Year of residency program			0.03
PGY2	17 (89.5)	242 (64.7)
PGY3	2 (10.53)	132 (35.3)
Specialty			0.11
Pediatrics	16 (84.2)	323 (86.4)
Internal medicine/pediatrics	2 (10.5)	8 (2.1)
Family practice	0	0
Anesthesiology	0	14 (3.7)
Emergency medicine	0	21 (5.6)
Other	1 (5.3)	8 (2.1)
Marital statusa
Married		207 (55.4)
Separated		0
Divorced		1 (0.3)
Widowed		0
Never married		166 (44.4)
Body mass index, kg/m2a		23.2 ± 3.4

Data shown as n(%) or mean ± SD

P-values from a chi-square test, Fisher’s exact test, or t-test.

^aNot gathered among those who refused.

Abbreviations: PGY, post graduate year; SD, standard deviation.

Of the 380 rotations with full participation, seven withdrew from full participation but agreed to remain in the study as observation only, and three withdrew completely from the study (Figure 1). The most common reasons for withdrawal were unwillingness to complete the diaries (n=2) and unwillingness to complete the PVT testing (n=3). There were two SAEs that occurred to resident-physicians, both of which were determined to be unrelated to the study (one on EDWR, one on RCWR). Three resident-physicians reported thoughts of suicide during the RCWR, none during the EDWR. Almost all questionnaire data was completed (99% baseline questionnaire, 96% end-of-rotation survey). Almost all had some actigraphy (94%) and PVT (98%) data collected. Fewer agreed to saliva collection (64%). The completion of the drive diary differed by schedule type (97% RCWR, 92% EDWR, p=0.046). Rates of refusals, full participation, withdrawals, and all other types of data completeness did not differ by schedule type (p>0.05).

The study involved 6577 patients (3267 during the EDWR, 3310 during the RCWR), representing 7099 admissions (3508 during the EDWR, 3591 during the RCWR). (Table 7). Data were collected over a total of 2870 days among six PICUs, for a total of 38821 patient-days (18749 during the EDWR, 20072 during the RCWR). Patients were on average 7.2 ± 6.7 years of age, with slightly more male than female patients (53.5% vs. 46.5%, respectively) (Table 7). The median length of unit stay was 2 days (range 1 to 244 days). There were no statistically significant differences of patient characteristics across schedule type.

Table 7.

Patient characteristics by schedule.

Characteristic	Total	EDWR	RCWR	P-value
Number of patients	6577	3267	3310	0.60
Number of unit admissions	7099	3508	3591	0.32
Number of patient-days	38821	18749	20072	0.20
age, year, mean ± SD	7.2 ± 6.7	7.3 ± 6.7	7.1 ± 6.6	0.14
Gender, n (%)
Female	3303 (46.5)	1655 (47.2)	1648 (45.9)	0.27
Male	3796 (53.5)	1853 (52.8)	1943 (54.1)
Length of unit stay, days
mean ± SD	5.5 ± 11.9	5.3 ± 11.4	5.6 ± 12.3	0.20
median (inter quartile range)	2 (2, 5)	2 (2, 5)	2 (2, 5)
Patient Chronic Condition Indexa, range 0–18
mean ± SD	2.6 ± 2.0	2.6 ± 1.9	2.6 ± 2.0
median (inter quartile range)	2 (1, 4)	2 (1, 4)	2 (1, 4)	0.53

p-values from a chi-square test or Wilcoxon rank-sum test.

^aDeveloped by Agency for Healthcare Research and Quality. Higher scores denote more chronic conditions.

Abbreviations: EDWR, extended duration work roster; RCWR, rapidly cycling work roster; SD, standard deviation.

Of the 2870 days covered by the study at all sites, a total of 345 days (12%) were not covered by a physician observer (summing hours with lack of coverage/24). (Table 2) Of these, 198.2 were in the EDWR, 146.8 in the RCWR (p<0.001).

4. Discussion

Results from data collected in the ROSTERS study will be evaluated for the impact of changes in pediatric resident-physician work schedules on patient outcomes in PICUs, together with intermediary variables such as sleep, performance, health and safety of individual resident-physicians. The study will also allow for a number of secondary analyses, such as further examination of SME rates across the variety of schedule types that were implemented, examination of specific types of harm level and category of events (i.e. medication-related, procedure-related). The data collected allow for adjustment of analyses for potential confounding factors, including resident-physician characteristics, such as experience level, and volume characteristics, such as the number of patients per resident-physician. Analyses can also be adjusted for patient-level characteristics, such as comorbidity index.

The secondary outcomes collected allow for examination of the impact of schedule on additional resident-physician-level outcomes, such as depression, MVCs, vigilance, sleepiness, and sleep duration and work quality.

Including six sites of varying size from around the country will improve the generalizability of findings to other PICUs, particularly in the academic setting.

Strengths of the study include the large sample size, the cross over design to minimize local effects, the geographic variation of the sites, and the collection of data on a number of levels (patient, resident-physician, medical errors). Limitations of the study include the lack of data collection on handovers of care, retrospective measurement of patient load (IPRP), and limited data gathered on patient complexity (ICD-9 and ICD-10 codes). The lack of these data limits the measurement of the impact of resident-physician workload and stress level on patient outcomes. Physician reviewers were given the harm level and whether or not the potential event was considered preventable by the initial reporter, which may have biased their decisions for classification of the primary outcome.

There were challenges to conducting this study. As discussed above, the difference in the EDWR scheduling and staffing of the study sites led to variations in the manner in which the RCWR schedule was implemented across sites. The frequency of resident-physician extended shifts, resident-physician staffing, IPRP, and the manner in which handovers of care were conducted varied. Any of these factors could potentially have modified the effects of the RCWR schedule. While efforts were made to standardize certain features of the implementation schedule (e.g., avoiding recurrent night shifts, ensuring sufficient days off each month), the differences in site characteristics, as well as the manner in which site PIs and program directors chose to implement the schedule within CCC and DCC guidelines could have had important effects. Analyzing site-specific differences in the performance of the RCWR schedule, and exploring possible contributors to any differences, will thus be of importance in interpreting the results of the trial.

The intensive data collection requirements of the study also posed challenges. These requirements made it difficult to find sites that were willing and able to participate in the study. The data collection was done in three waves due to annual budget restrictions, which required additional staff time from the DCC and CCC. Given that potential medical errors were collected by a combination of methods (direct observation, chart review, etc.) there was a possibility for duplicate reporting requiring additional diligence and training of physician observers and DCC staff to detect and remove duplicates from the database. There were some failures with the actigraph equipment leading to loss of data, although in most cases back-up devices were available to minimize data loss. While almost all resident-physicians have some PVT recording data, difficulty scheduling resident-physician test times led to some incomplete data collection. The design of the electronic diary data collection website allowed for duplicate data entry, so additional time was spent processing these data to remove overlapping or duplicate information. About 12% of study time was not covered by a physician observer.

The data collection was conducted over four years. In that time a few studies have examined 2011 work hour reforms for interns (PGY1). An examination of the medical records of Medicare beneficiaries the year before the 2011 ACGME work hour reforms and a year after revealed no significant difference in 30-day mortality or readmission rates by level of intensity of teaching [15]. Another study enrolled 2323 interns at 14 teaching hospitals before and after the 2011 reforms. Interns after the reforms did report working fewer hours but did not report any increases in sleep or decreases in rates of depressive symptoms. The rate of self-reported concern about making medical errors increased after the reform [14]. A study comparing a 2003-compliant model to the 2011 model among 43 interns at one teaching hospital did show increased sleep duration along with an increase in handovers of care and less educational opportunities [13]. These studies did not include any PGY2 or PGY3 resident-physicians, and did not have adjudicated medical errors or objectively measured sleep. One large trial found no difference in operative mortality and major complication rates when surgical resident-physicians abided by the 2011 ACGME rules, as compared with when they did not. This study did include PGY2+ resident-physicians (73%), but results for patient-related outcomes were not presented stratified by PGY [16]. The iCOMPARE trial (Individualized Comparative Effectiveness of Models Optimizing Patient Safety and Resident Education trial) was conducted in 63 internal medicine residency programs in the United States during 2015–2016. Residency programs underwent cluster randomization to a schedule following the 2011 ACGME standards, or a schedule that permitted more flexible duty hours (removing the 16-hour restriction on shift length). Comparing outcomes from the prior year to the trial year, allowing flexible schedules did not adversely affect 30-day mortality or several other measured outcomes of patient safety [24].

5. Conclusions

Data collection for the ROSTERS trial was successfully completed in March 2017, including resident-physician level data and patient level data. Classification of potential medical errors was completed in October 2017. Resident-physician enrollment exceeded the goal of 300. Number of patient-days observed were less than expected (84%) based on projections. The study design allows for analyses on a number of aims, which are ongoing.

List of abbreviations:

ACGME

Accreditation Council for Graduate Medical Education

CCC

Clinical Coordinating Center

DCC

Data Coordinating Center

DSMB

Data Safety and Monitoring Board

EDWR

Extended duration work roster

ICD

International Classification of Diseases

IRB

Institutional Review Board

MVC

Motor vehicle crash

NCC-MERP

National Coordinating Council for Medication Error Reporting

NHLBI

National Heart, Lung, and Blood Institute

PGY

Post graduate year

PI

Principal Investigator

PICU

Pediatric intensive care unit

PVT

Psychomotor vigilance test

RCWR

Rapidly cycling work roster

ROSTERS

Randomized Order Safety Trial Evaluating Resident-physician Schedules

SAE

Serious adverse event

SME

Serious medical errors