Estimation of Surgical Resident Duty Hours and Workload in Real Time Using Electronic Health Record Data

Abstract

OBJECTIVE:

To explore the use of electronic health record (EHR) data to estimate surgery resident duty hours and monitor real time workload.

DESIGN:

Retrospective analysis of resident duty hours logged using a voluntary global positioning system (GPS)-based smartphone application compared to duty hour estimates by an EHR-based algorithm. The algorithm estimated duty hours using EHR activity data and operating room logs. A dashboard was developed through Plan-Do-Study-Act cycles for real-time monitoring of workload.

SETTING:

Single tertiary/quaternary medical center general surgery residency program with approximately 90 categorical, preliminary, and integrated residents at eight clinical sites.

PARTICIPANTS:

Categorical, preliminary, and integrated surgery residents of all clinical years who volunteered to pilot a GPS application to track duty hours.

RESULTS:

Of 2,623 work periods by 59 residents were logged with both methods. EHR-estimated work periods started later than GPS logs (median 0.3 hours, interquartile range [IQR] −0.1 – 0.3); EHR-estimated work periods ended earlier than GPS logs (median 0.1 hours, IQR −0.7 –0.3); and EHR-estimated duty hour totals were less than totals logged by GPS (median −0.3 hours, IQR −0.8 – +0.1). Overall correlation between weekly duty hours logged by EHR and GPS was 0.79. Correlations between the 2 systems stratified from PGY-1 through PGY-5 were 0.76, 0.64, 0.82, 0.87, and 0.83, respectively. The algorithm identified six 80-hour workweek violations (averaged over 4 weeks), while GPS logs identified 8. EHR-based duty hours and operational data were integrated into a dashboard to enable real time monitoring of resident workloads.

CONCLUSIONS:

EHR-based estimation of surgical resident duty hours has good correlation with GPS-based assessment of duty hours and identifies most workweek duty hour violations. This approach allows for dynamic workload monitoring and may be combined with operational data to anticipate and prevent duty hour violations, thereby optimizing learning.

INTRODUCTION

Resident duty hour management has been a longstanding strategy employed by the Accreditation Council for Graduate Medical Education (ACGME) to optimize the clinical work environment for resident education and wellbeing.¹ The limitation of residents’ clinical and educational work hours to 80 hours per week, averaged over a 4-week period, is a key requirement.² Although these regulations have existed for several years, compliance with this rule has proven difficult for surgical residencies. The Flexibility in Duty Hour Requirements for Surgical Trainees (FIRST) Trial provided evidence supporting the relaxation of some duty hour regulations, including confirmation that patient safety was maintained with flexible policies, but the 80-hour work week limit has remained constant.³ With increasing recognition of the toll of burnout on physicians and the importance of resident wellbeing, the ACGME has increased scrutiny of residency program work hours. During the ACGME review period ending in 2019, more than 160 of 320 accredited surgical programs were cited for duty hour violations.⁴

Consistent, timely, and reliable resident duty hours are challenging to record, complicating surgical programs’ efforts to manage resident workload.5 Resident duty hours are often self-reported, but self-reported duty hours have delayed availability and have been shown to be inaccurate.⁶ Residents may inaccurately report weekly duty hours because of concerns about consequences for noncompliance, inability to recall, or simply to save time by reporting uniform hours.^7,8 In contrast, the ACGME evaluates program duty hours via an anonymous annual survey of residents.⁹ As a result, surgical programs can be confronted with the dilemma of receiving ACGME citations for duty hour violations that were not revealed by resident-reported duty hours. These conflicting data make it challenging to address the underlying issue of resident workload.

In an attempt to address this gap, this surgical residency carried out a voluntary pilot program of a global positioning system (GPS)-based smartphone application to log duty hours automatically based on a resident’s physical location in relation to clinical sites. Residents who opted to use the application could edit the GPS logs as necessary before reporting the duty hours to the program to correct any inaccuracies. Unfortunately, residents’ privacy concerns and technical issues with the application limited universal adoption.

Given that the EHR is integral to the resident workday, we hypothesized that EHR interaction data could be leveraged to estimate duty hours as an alternative to GPS-based tracking. We also hypothesized that EHR-based duty hours could be assessed on a near real-time basis in combination with operational data to monitor resident workload, potentially informing interventions to optimize workforce deployment and educational opportunities.

MATERIALS AND METHODS

Study Design

The work was conducted at the University of California, San Francisco (UCSF) Health System and School of Medicine. A retrospective analysis was performed comparing resident duty hours logged using ResQ Medical™, a GPS-based smartphone application (ResQ Medical, Inc, Sacramento, California) with an EHR-based algorithm to estimate duty hours that has been reported by this group and is described below.⁷ General (categorical, preliminary, and integrated) and plastic surgery residents who volunteered to use the GPS application from January to September 2020 were included. Integrated surgical residents were primarily at the PGY-1 level and included all surgical subspecialties; duty hours were studied on general surgery services only.

This study was approved by the University of California San Francisco (UCSF) institutional review board (Protocol #20-32336).

Development of Algorithm for EHR-Based Duty Hour Logging

The UCSF Health Informatics team and the UCSF Department of Medicine developed an algorithm to estimate work hours leveraging EHR data, which was previously validated using internal medicine resident self-reported duty hours.⁷ This algorithm was adapted for surgical residents in this study. Residents’ logins and logoffs with their associated times and devices were updated daily using data from the EHR, Epic™ (Epic Systems Corporation, Verona, Wisconsin). Continuous duty hours were estimated by linking together on-campus EHR logins, with continuous work inferred between logins less than 5 hours apart and while trainees were in the operating room as measured by the surgical case logs. OR staff regularly log all operating surgeons. If a resident was logged as operating, the surgery start and end times were included by the algorithm. All on-site EHR usage was logged by the algorithm. Off-site EHR usage associated with active clinical care, such as writing notes, entering orders, or modifying problem lists and other editable portions of the EHR, also counted toward weekly duty hours as per ACGME guidelines. Off-site EHR usage consisting of “read-only” chart review was attributed to study or personal follow-up, and therefore was not counted toward duty hours.

Exploratory data of EHR-based work periods based on the internal medicine-derived algorithm were obtained. A convenience sample of general surgery residents was queried to assess the accuracy of the exploratory duty hours compared to the residents’ recall of work periods (not their self-reported duty hours to the program). This revealed undercounting of duty hours due to the algorithm estimating later start times for senior residents, who did not often interact with the EHR until after morning rounds. To account for this, a work period start time of 6 am was automatically assigned if the first EHR interaction or OR log of the day began between 6 am and noon. Examples of work periods calculated by the final algorithm for surgery residents are shown in Figure 1.

FIGURE 1. Diagram of EHR-based estimation of resident work periods.

The EHR algorithm estimated surgical resident work periods using a combination of logins and surgical case logs from EHR data, with daily updates. Work periods were delineated by gaps of at least 5 hours between EHR activity or case logs. Work periods were assumed to start at 6 am if later morning EHR activity or surgery was detected. In this example, on Days 1 and 3, the estimated work periods started at 6 am and ended after the last EHR activity. On Day 2, the estimated work period started with the first EHR activity before 6 am. An overnight emergency surgery was also logged as work hours.

GPS-Based Duty Hour Logging

ResQ Medical, a GPS location-tracking smartphone application, was piloted in an effort to augment self-reported resident duty hours. Participants downloaded a GPS-based location tracking application onto their smart phone. The GPS coordinates of clinical work site building boundaries, with adequate location accuracy to exclude non-clinical buildings, sidewalks, surface streets, and parking areas, were designated by the application’s developer with guidance from residency program leadership. Work periods began when residents crossed into the designated boundaries and ended when they crossed out. Residents were able to edit GPS-based duty hour logs for technical issues or non-work-related presence on clinical sites prior to submitting logs to the residency program. With time, usage of the GPS-based application was greatest among residents who experienced the fewest technical problems with the application, leading to consistent use of the application among residents who rarely edited the GPS-based data. Unedited GPS-based duty hours were used for this study as these were felt to be more sensitive to duty hour violations and excessive work.

Outcomes and Analysis

GPS-based duty hours were used to validate EHR-based duty hours, including work period start and stop times and total durations. Weekly duty hour totals were compared using the Pearson correlation coefficient (r) for the overall cohort and by PGY level. Correlation was not calculated between self-reported duty hours and EHR-based duty hours given that the GPS-based data were considered the gold standard. Participating residents’ self-reported duty hours during the study period were not independent of their GPS data given the pilot program reporting design, making this comparison less informative.

Weekly duty hour violations based on ACGME definitions (over 80 hours per week averaged over 4 weeks) were compared between GPS app-based duty hours and EHR-based duty hours. The 80-hour work week limit was selected as a focus because it is potentially preventable. Impending 80-hour work week violations can be recognized prior to the end of a rotation because they are calculated as 4-week moving averages, allowing a timely intervention to prevent the violation.

Maximum work period lengths (24 hours plus 4 hours for continuity or education), rest period limits (at least 8 hours off or 14 hours after a 24-hour work period), and monthly rest day limits (at least 1 day per week averaged over 4 weeks) were not assessed for several reasons. Maximum work period lengths and rest period violations can only be identified after the fact, violations are rare, and these violations require systematic steps for prevention. Existing scheduling practices prevent residents from being scheduled for more than 24 days out of 28, so rest day violations are exceedingly rare.

Statistical analysis was performed with R version 4.0.2 (The R Foundation for Statistical Computing).

Data Visualization

In order to operationalize the real-time monitoring of resident workload, Tableau™ (Tableau Software, Seattle, Washington) was used for data visualization. A dashboard was developed through a Plan-Do-Study-Act (PDSA) approach, with collaboration between program leadership, health informaticists, and surgery residents.¹⁰

RESULTS

GPS-based and EHR-based duty hour logs were compared from January 1, 2020, to September 24, 2020. Of 146 eligible residents (67 female, 46%) during the study period, 59 residents (36 female, 61%) participated in GPS-based duty hour logging (participation rate 40%). A total of 2,571 work periods were recorded, with 849 by PGY-1s, 341 by PGY-2s, 772 by PGY-3s, 349 by PGY-4s, and 260 by PGY-5s.

The median length of EHR-based duty hour periods was 12.4 hours (interquartile range [IQR] 10.3–13.4 hours). The median length of GPS-based duty hour periods was 12.6 hours (IQR 9.8–13.6 hours). EHR-based work period start times tended to be later than GPS-based start times, with a median difference of 0.3 hours later (Table 1). EHR-based work period end times tended to be earlier, with a median difference of 0.1 hours earlier. The overall median difference between EHR-based duty hour totals compared to GPS-based duty hour totals was −0.3 hours (IQR −0.8 – +0.1). Time discrepancies at the beginning and end of work periods, as well as total duty hours, likely reflect the added time required for residents to reach clinical work areas after arriving on site (Fig. 2). While differences in work period start times between the two methods were fairly consistent with narrow IQRs, work period end times were more variable, particularly for residents in the PGY-2 to PGY-5 years (Table 1). These discrepancies likely reflect end-of-day clinical work not captured by the EHR activity of these more senior residents, such as afternoon rounds.

TABLE 1.

Differences Between EHR-Based and GPS-Based Work Periods

	Work Period Start Time		Work Period End Time		Total Hours
	Median	IQR	Median	IQR	Median	IQR
Overall	0.3	−0.1 to 0.3	−0.1	−0.7 – 0.3	−0.3	−0.8 – 0.1
PGY-1	0.3	−0.1 to 0.3	−0.1	−0.5 – 0.2	−0.3	−0.6 – 0.0
PGY-2	0.1	−0.4 to 0.3	−0.7	−1.7 – 0.0	−0.8	−2.0 – −0.3
PGY-3	0.3	−0.1 to 0.3	−0.1	−0.6 – 0.4	−0.3	−0.7 – 0.3
PGY-4	0.3	−0.1 to 0.3	−0.1	−0.9 – 0.5	−0.3	−1.1 – 0.3
PGY-5	0.3	−0.1 to 0.3	−0.2	−1.0 – 0.3	−0.3	−1.1 – 0.1

All values in hours. Values represent GPS-based time subtracted from EHR-based time. Positive values indicate EHR-based time was later than GPS-based time. Negative values indicate EHR-based time was earlier (or shorter) than GPS-based time. HER, electronic health record; GPS, global positioning system.; IQR, interquartile range.

FIGURE 2. Diagram of discrepancies between EHR-based and GPS-based work periods.

EHR-based work periods were found to start later and end earlier than GPS-based work periods, resulting in shorter estimated duty hours.

Overall correlation between total weekly EHR-based and GPS-based duty hours was r = 0.79 (Fig. 3). Correlations between the 2 systems stratified from PGY-1 through PGY-5 were 0.76, 0.64, 0.82, 0.87, and 0.83, respectively. EHR-based duty hours revealed 6 violations of the weekly 80-hour limit, averaged over 4 weeks. GPS-based duty hours revealed 8 violations, including the 6 revealed by EHR data.

FIGURE 3. Correlation of weekly duty hours determined by GPS and EHR.

Line represents correlation of 1.0. Overall correlation is 0.79.

A dashboard was developed through PDSA cycles to operationalize the EHR-based resident duty hour estimates (Fig. 4). One associate program director, one vice chair for quality and safety, one health informaticist, and one PGY-4 general surgery resident participated. The objective was to create a dashboard that would reflect the overall resident workload and anticipate 80-hour work week limit violations. The first PDSA cycle was centered on algorithm refinement and resulted in displaying resident duty hour totals and running averages (Fig. 4A). This was felt to lack data on current workload, so after a second PDSA cycle, surgical service census trends were added (Fig. 4B). A third PDSA cycle added a breakdown of ICU vs. non-ICU and primary vs. consult censuses (Fig. 4C). Finally, these data were felt to lack representation of upcoming workload, so a fourth PDSA cycle added surgical case schedules with daily updates to allow forecasting of workload based on upcoming patients and surgical complexity (Fig. 4D). The PDSA participants felt the complete dashboard reflected current resident workload and allowed anticipation of 80-hour work week limit violations. The dashboard with daily updates was provided to program leadership and was accessed 312 times over 9 weeks, or approximately 7 times per weekday.

FIGURE 4. Components of dashboard for daily monitoring of resident workload.

Duty hours and operational data combined to allow monitoring of resident workload and anticipation of impending 80-hour work week limit violations. A, resident duty hour totals and running averages based on electronic health record algorithm. B, trends of individual surgical service censuses. C, breakdown of censuses by ICU vs. non-ICU and primary vs. consult patients. D, scheduled surgeries for anticipation of upcoming workload.

DISCUSSION

An algorithm for estimating surgical resident duty hours using Epic™ EHR data achieved close approximation of GPS-based duty hour logging, with satisfactorily high correlation with weekly duty hour averages across all PGY levels, validating its use for surgical resident duty hour tracking. EHR-based duty hours revealed the majority of duty hour violations noted using GPS data.

EHR-based duty hours avoided potentially invasive GPS tracking of residents. GPS applications and other location-based logging raise concerns over resident privacy.¹¹ Uptake of the GPS application for duty hours was limited to a minority of residents in our program, partly due to concerns over privacy, as well as technical issues for others. During the study period, the ResQ Medical™ application was available only for smartphones operating on iOS™ (Apple Inc, Cupertino, California). This motivated the use of a universal, less invasive, and more reliable means of tracking duty hours.

This study builds on previously described tools from non-surgical specialties, validating that EHR data can be used to approximate the resident work day once time in the OR as adequately accounted for.⁷ Dziorny et al. described a method of tracking pediatric resident duty hours using EHR interactions and validated it using self-reported data, revealing duty hour violations with EHR that were not self-reported.¹² Our study validated EHR-based duty hour estimates using GPS-derived duty hours. Shine et al reported a method of using EHR interactions to track medical resident duty hours and validated it with self-reported logs.¹³ However, their method was not automated and required manual tabulation of data.

Rigid adherence to duty hour restrictions may draw disproportionate attention from the underlying issues of resident workload management. Szymczak et al. qualitatively studied residents’ views of duty hour restrictions, noting that residents perceive the complex impact of duty hour restrictions on patient safety, workload management, continuity of care, and personal responsibility.⁸ Residents also may fear repercussions for reporting long hours, as Zamani et al. found that anonymized self-reported hours revealed more violations than identified self-reported hours.¹⁴ These complexities contribute to the unreliability of self-reported duty hours.⁶ Despite the large body of literature dedicated to its study, the relationship between duty hour restrictions and resident workload and well-being remains unsettled.¹⁵ Passive monitoring of duty hours using EHR data, rather than requiring active resident self-reporting, may more meaningfully capture resident workloads. To the extent that adherence to duty hour limits is advantageous, this development of an EHR-based algorithm for estimating resident duty hours may be a useful tool for program leaders, since it provides nearly real-time data without the need for self-reporting.

Fewer than 2% of ACGME residencies used a real time electronic duty hour logging system in 2016.⁵ Building on the algorithm, we developed and deployed an EHR-based dashboard that tracks resident duty hours in an up-to-date fashion, along with inpatient service censuses and upcoming case bookings. This combination of data may allow forewarning of impending duty hour violations by allowing program leadership to identify residents at risk of working over hours on services with high patient volumes. If additional resources can be applied, such as additional staffing with residents or advanced practice providers, or strategic days off can be given, weekly duty hour violations may be avoidable. This use of data may not be responsive enough to mitigate rest hour violations.

A strict focus on avoiding duty hour rule violations would oversimplify the interplay between resident workload and education. A 2014 systematic review found that objective educational measures including board certification performance did not improve after the implementation of ACGME duty hour restrictions, while subjective perceptions of educational quality worsened and perceptions of wellness improved with the 80-hour workweek limit.¹⁶ The FIRST trial and a subsequent 4-year follow-up study revealed that duty hour policies have complex effects, with residents in the flexible policy arm reporting better perceptions of clinical continuity, equivalent perceptions of overall education, and worse perceptions of personal time compared to residents in the standard policy arm.^3,17 The mixed impacts suggest that duty hour policy and monitoring are necessary but not sufficient for resident workload management.

There are several limitations to this study. A primary limitation is that even GPS-based duty hour logging may not accurately capture all resident duty hours. The ACGME defines resident duty hours as including clinical, administrative, and academic activities, excluding off-site reading and preparation time.¹⁸ However, work-related administrative and academic activities could and likely did take place off-site and would not have been captured by either method. The EHR algorithm excluded read-only off-site EHR usage since it could not be distinguished from reading or preparation that would not count as work by ACGME definition. The algorithm’s accounting for resident presence in the OR (and away from EHR interactions) was reliant on OR staff logging operating surgeons in the EHR. The sensitivity and specificity of OR staff inputs could not be determined, but was felt to be reliable, and the resulting algorithm estimated duty hours with reasonable accuracy using these data. Finally, this study was limited to data from a single center. Although the algorithm relied on an EHR with real-time logging of user interaction and OR participants, these data points are likely available to many programs and would not limit the algorithm’s generalizability. Off-site EHR activity was accounted for, but this may not be possible with all EHRs. While the COVID-19 pandemic overlapped with the majority of the study period, on-site resident work and duty hour reporting was not thought to be significantly altered in ways that would bias this study’s results.

CONCLUSION

EHR-based estimation of surgical resident duty hours was validated using GPS-based duty hours, with high accuracy and correlation with weekly duty hour totals. EHR-based estimates identified the majority of 80-hour weekly limit violations noted by GPS. Duty hours can be reliably derived from EHR data without the privacy issues of GPS tracking or the accuracy issues of self-reporting. Combined with operational data, EHR-based duty hours can provide program leaders near real-time insight into resident workload to optimize the clinical learning environment.