This cohort study of surgical residency and fellowship physicians at 2 academic institutions uses technical skill scores rated by attending physicians to assess the association of trainees’ self-reported daytime sleepiness with their intraoperative skill while performing septoplasty.
Key Points
Question
What is the association between daytime sleepiness and intraoperative technical skill among surgical trainees performing nasal septoplasty?
Findings
In this cohort study of 19 surgical trainees participating in septoplasty procedures, increased daytime sleepiness was associated with statistically significant lower technical skill scores as rated by supervising attending physicians.
Meaning
Daytime sleepiness of surgical trainees may be inversely associated with intraoperative technical skill when performing septoplasty.
Abstract
Importance
Daytime sleepiness in surgical trainees can impair intraoperative technical skill and thus affect their learning and pose a risk to patient safety.
Objective
To determine the association between daytime sleepiness of surgeons in residency and fellowship training and their intraoperative technical skill during septoplasty.
Design, Setting, and Participants
This prospective cohort study included 19 surgical trainees in otolaryngology–head and neck surgery programs at 2 academic institutions (Johns Hopkins University School of Medicine and MedStar Georgetown University Hospital). The physicians were recruited from June 13, 2016, to April 20, 2018. The analysis includes data that were captured between June 27, 2016, and April 20, 2018.
Main Outcomes and Measures
Attending physician and surgical trainee self-rated intraoperative technical skill using the Septoplasty Global Assessment Tool (SGAT) and visual analog scales. Daytime sleepiness reported by surgical trainees was measured using the Epworth Sleepiness Scale (ESS).
Results
Of 19 surgical trainees, 17 resident physicians (9 female [53%]) and 2 facial plastic surgery fellowship physicians (1 female and 1 male) performed a median of 3.00 septoplasty procedures (range, 1-9 procedures) under supervision by an attending physician. Of the 19 surgical trainees, 10 (53%) were aged 25 to 30 years and 9 (47%) were 31 years or older. The mean ESS score overall was 6.74 (95% CI, 5.96-7.52), and this score did not differ between female and male trainees. The mean ESS score was 7.57 (95% CI, 6.58-8.56) in trainees aged 25 to 30 years and 5.44 (95% CI, 4.32-6.57) in trainees aged 31 years or older. In regression models adjusted for sex, age, postgraduate year, and technical complexity of the procedure, there was a statistically significant inverse association between ESS scores and attending physician–rated technical skill for both SGAT (−0.41; 95% CI, −0.55 to −0.27; P < .001) and the visual analog scale (−0.75; 95% CI, −1.40 to −0.07; P = .03). The association between ESS scores and technical skill was not statistically significant for trainee self-rated SGAT (0.04; 95% CI, −0.17 to 0.24; P = .73) and the self-rated visual analog scale (0.19; 95% CI, −0.79 to 1.2; P = .70).
Conclusions and Relevance
The findings suggest that daytime sleepiness of surgical trainees is inversely associated with attending physician–rated intraoperative technical skill when performing septoplasty. Thus, surgical trainees’ ability to learn technical skill in the operating room may be influenced by their daytime sleepiness.
Level of Evidence
NA.
Introduction
Nearly half (44.4%) of residents in otolaryngology–head and neck surgery reported excessive or severe daytime sleepiness in a nationwide survey.1 Prolonged work hours and sleep interruption during nighttime shiftwork while in surgical training that leads to sleep deprivation can pose a safety risk to patient care. Despite resident duty-hour restrictions implemented by the Accreditation Council for Graduate Medical Education, no improvement in resident well-being or patient care has been found.2,3Sleep deprivation in surgical trainees is associated with cognitive dysfunction, including attention failures and impaired working memory and decision making,4,5 all of which can adversely affect trainees’ performance and learning in the operating room. Resident participation increases the duration of outpatient otolaryngology–head and neck surgical procedures, but does not seem to negatively affect patient outcomes,6,7,8,9 perhaps owing to adequate supervision.
To date, the association between sleepiness in surgical trainees and their intraoperative technical skill remains unknown. Previous research has only focused on resident fatigue and performance during physical or virtual-reality simulation in minimally invasive surgical procedures. In simulation, residents who reported sleepiness or fatigue performed simple training tasks with less efficiency and more errors,10,11,12,13 and they scored lower on an expert-rated global rating scale than did residents who reported otherwise.10 In addition, fatigue in residents nullified their previous learning and experience using the simulator.14 Conversely, findings in a few studies suggest that sleep-deprived surgical trainees adapt to the extent that their performance improves compared with their performance in a rested state.15,16 Our objective in this study was to determine the association between self-reported daytime sleepiness in surgical resident and fellowship physicians and their intraoperative technical skill while performing a septoplasty procedure within the Accreditation Council for Graduate Medical Education’s current 80-hour work schedule regulation.
Methods
Participants
We recruited residents in otolaryngology–head and neck surgery who were affiliated with 2 academic centers (Johns Hopkins University School of Medicine, Baltimore, Maryland, and MedStar Georgetown University Hospital, Washington, DC) during their rotation in facial plastic surgery, as well as fellowship-trained physicians in facial plastic surgery at Johns Hopkins University School of Medicine. The physicians were recruited from June 13, 2016, to April 20, 2016. The Johns Hopkins Medicine institutional review board, and the institutional review boards at MedStar Georgetown University Hospital and at the Washington DC Veterans Affairs Medical Center approved this study. Skill assessment is normal educational practice in surgical training and thus informed consent was not required. Data were deidentified for analysis. This analysis includes data that were captured between June 27, 2016, and April 20, 2018.
Survey Instruments
All surgical residents and fellows completed a demographics survey at enrollment in the study. We measured self-reported daytime sleepiness of surgical trainees using the Epworth Sleepiness Scale (ESS).17,18 Questions in the ESS inquire about the respondent’s propensity to sleep in 8 situations. Responses to each question are scored from 0 to 3 for a maximum of 24 on the complete scale. Higher ESS scores indicate more sleepiness. Surgical trainees completed the ESS before the first septoplasty procedure in which they participated on a given day. For this study, we deconstructed the septoplasty procedure into 5 essential phases: (1) opening the septum and making an incision; (2) elevating the septal flaps; (3) removing the deviated septum; (4) reconstructing the septum, including replacing straightened or scored cartilage; and (5) closing the incision. The supervising attending physician reported which of the 5 phases of the septoplasty the surgical trainee performed.
We measured trainees’ technical skills using 2 different scales. The attending physician rated the trainee’s skill using the Septoplasty Global Assessment Tool (SGAT) for the overall procedure19 and a skill-based visual analog scale (VAS) with a 1-point increment (range, 0-100; where 0 is novice and 100 is master) for each phase performed by the trainee. Attending physician ratings were based on direct observation while in the operating room, with endoscopic visualization available when it was used for clinical indication (only a small subset of procedures). The attending physicians were unaware of the trainee’s report on the ESS. Surgical trainees rated their own skill using the same SGAT (procedure-level) and VAS (phase-level). For each procedure, we computed the mean (SD) of VAS scores for all phases in which a trainee participated. Adopted from the global rating scale, a component of the Objective Structured Assessment of Technical Skills approach,20 the SGAT is a 7-item skill assessment tool with a 5-point Likert scale linked to clear descriptors. We analyzed the sum of the 7-item scores. The SGAT was previously validated against a checklist-based expert assessment for each task in the septoplasty.19 Finally, we recorded the attending physician’s perception of the technical complexity of each procedure using a technical complexity–based VAS (range of 0-10, in which 0 indicated below average complexity/difficulty; 5, average complexity/difficulty; and 10, most or very high complexity/difficulty).
Statistical Analysis
We analyzed the total score with the SGAT and VAS for skill as continuous variables. We counted each procedure only once for the descriptive analyses. To account for multiple procedures by the same trainee, we used generalized estimating equations to regress total ESS scores on measures of technical skill. For regression purposes, when 2 trainees participated in the same procedure (5 instances), we disregarded clustering by procedure and considered data from each trainee to be independent. We separately modeled the attending physician assessment of the trainee’s skill as well as the trainee’s self-assessment of skill using the SGAT and VAS. We adjusted for attending physician reported technical complexity of the procedure, and the sex, age (categorized as ages 25-30 years and ≥31 years), and training level or postgraduate year (junior resident [postgraduate years 1-3], senior resident [postgraduate years 4-7], and fellow) of the trainees.6,7,21 Based on hypotheses suggested in prior research,6,7,21 we also modeled modification of the association between ESS score and technical skill by trainee sex, age, and training level. We used the quasi-likelihood under the independence model criterion for model selection.22 We used R, version 3.4.2 (R Core Team) for all analyses.23 All statistical tests were 2-sided; we considered a P value of .05 or less to be significant.
Results
We analyzed data from 64 septoplasty procedures. Of 19 surgical trainees, 17 residents (9 female [53%] and 8 male [47%]) and 2 fellows (1 female and 1 male) performed a median of 3.00 septoplasty procedures (range, 1-9 procedures) supervised by 1 of 6 attending physicians (3 female and 3 male). Of the 19 trainees, 10 (53%) were aged 25 to 30 years and 9 (47%) were 31 years or older. Of the 17 surgical residents, 7 (41%) were junior residents and 10 (59%) were senior residents (Figure). Female trainees participated in 27 of 64 surgical procedures (42%), and male trainees participated in 37 of 64 procedures (58%) (Figure). Of 27 surgical procedures in which a female trainee participated, 16 (59%) were performed by a junior resident. Of 37 surgical procedures in which a male trainee participated, only 8 (22%) were by a junior resident, and 21 (57%) were by a senior resident.
Figure. Number of Surgical Trainees and Surgical Procedures by Sex, Age, and Postgraduate Year (PGY).
Skill assessments and ESS scores by trainee characteristics are summarized in the Table. The mean ESS score overall was 6.74 (95% CI, 5.96-7.52), with the median of 7.00 (range, 0-17). Trainees reported a normal level of daytime sleepiness (ESS ≤10) in 58 of 64 (90.6%) surgical procedures, mild to moderate excessive sleepiness (ESS of 11-15) in 5 of 64 (7.81%), and severe excessive sleepiness (ESS >15) in 1 of 64 (1.56%).
Table. Epworth Sleepiness Scale, Attending Physician–Rated Skill Scores, and Trainee Self-rated Skill Scores by Surgical Trainee Characteristicsa.
| Characteristic | ESS, Mean (95%CI) | Attending Physician–Rated SGAT, Mean (95% CI) | Attending Physician–Rated VAS, Mean (95% CI) | Surgical Trainee Self-rated SGAT, Mean (95% CI) | Surgical Trainee Self-rated VAS, Mean (95% CI) |
|---|---|---|---|---|---|
| Sex | |||||
| Female | 6.67 (5.57-7.77) | 25.52 (24.02-27.01) | 66.62 (62.19-71.05) | 25.43 (24.14-26.72) | 60.70 (54.50-66.90) |
| Male | 6.79 (5.69-7.90) | 29.76 (28.29-31.22) | 74.51 (71.12-77.90) | 28.47 (26.87-30.07) | 73.65 (70.02-77.29) |
| Age, y | |||||
| 25-30 | 7.57 (6.58-8.56) | 27.08 (25.50-28.65) | 71.91 (68.05-75.78) | 25.49 (24.16-26.81) | 66.47 (62.01-70.93) |
| ≥31 | 5.44 (4.32-6.57) | 29.27 (27.64-30.90) | 70.01 (65.74-74.28) | 29.41 (27.81-31.00) | 70.07 (63.69-76.44) |
| Postgraduate year | |||||
| 1-3 | 7.29 (5.98-8.60) | 24.42 (22.93-25.90) | 67.44 (62.75-72.13) | 25.00 (23.53-26.47) | 66.67 (61.54-71.80) |
| 4-7 | 6.79 (5.73-7.85) | 31.04 (29.68-32.39) | 75.34 (71.16-79.52) | 27.04 (25.43-28.64) | 65.18 (58.60-71.75) |
| Fellow | 5.33 (3.30-7.37) | 27.92 (25.58-30.25) | 69.02 (63.35-74.68) | 31.67 (29.67-33.66) | 77.18 (71.88-82.48) |
Abbreviations: ESS, Epworth Sleepiness Scale; SGAT, Septoplasty Global Assessment Tool; VAS, visual analog scale.
Details of the scores are given in the Survey Instruments subsection of the Methods section.
In a univariate analysis, there was a statistically significant association between ESS scores and attending physician–rated SGAT (–0.34; 95% CI, −0.62 to 0.06; P = .02) and a qualitatively different, statistically significant association with trainee self-rated SGAT (0.16; 95% CI, 0.01-0.31; P = .03). There was no statistically significant association between ESS scores and the attending physician–rated VAS (−0.64; 95% CI, −1.50 to 0.19; P = .13), or the trainee self-rated VAS (0.09; 95% CI, −1.00 to 1.20; P = .88).
In analyses adjusted for sex, age, postgraduate year, and technical complexity of the procedure, there was a statistically significant inverse association between ESS scores and attending physician–rated technical skill for both SGAT (−0.41; 95% CI, −0.55 to −0.27; P < .001) and VAS (−0.75; 95% CI, −1.40 to −0.07; P = .03). The association between ESS scores and technical skill was not statistically significant for trainee self-rated SGAT (0.04; 95% CI, −0.17 to 0.24; P = .73) and self-rated VAS (0.19; 95% CI, −0.79 to 1.2; P = .70). We did not observe a statistically significant interaction between sex, age, or postgraduate year with the association between their ESS scores and attending physician–rated technical skill using both the SGAT and VAS.
Discussion
There was an inverse association between daytime sleepiness of surgical trainees and attending physician–rated technical skill in septoplasty procedures. This association was not statistically significant for the trainee self-rated skill. These results indicate that ratings provided by attending physicians may have been sensitive to the association between daytime sleepiness of the trainees and their technical skill, even though the raters were unaware of the actual ESS scores reported by the trainees. In contrast, self-assessment of technical skill by the trainees did not seem to reflect their perception of their own daytime sleepiness.
The findings in this study may inform the ongoing debate regarding work hours in postgraduate surgical training. Sleep-deprived surgeons pose a risk for patient safety, and thus, resident concerns regarding fatigue and well-being have led to several changes in work hour restrictions. Several studies have shown associations between resident fatigue and errors in clinical care, such as prescription error and conflict with colleagues.4,5 Some studies have expressed the concerns that limiting resident duty hours would result in insufficient training.3 However, the relationship between trainees’ short-term fatigue or sleepiness and their technical skill has, to our knowledge, only been studied in simulation.10,11,12,13,14,15,16 The current study showed that sleepiness in trainees also adversely affected their technical performance in the operating room.
We may have underestimated the association between daytime sleepiness of surgical trainees and their intraoperative technical skill. In contrast to the mean ESS score of 6.74 in our study, a nationwide survey of otolaryngology residents reported a higher mean ESS score of 9.98.1 A larger proportion of residents in our study reported normal ESS scores (90.6%) compared with the national survey (55%) reported in Nida et al.1 One possible explanation for the low ESS scores that we observed is that trainees in our study adhered to Accreditation Council for Graduate Medical Education work hour regulations. A more likely explanation is that most trainees participated in our study during their facial plastic surgery rotation. As the nationwide survey showed, trainees reported higher ESS scores during their head and neck surgery rotation compared with other rotations. Thus, the small magnitude of association that we observed is likely an underestimate, and the association between sleepiness and technical skill among trainees should be studied within other rotations in otolaryngology.
There are other factors that are potentially relevant to understand the small magnitude of association between sleepiness and intraoperative technical skill. Previous research showed that surgeons are resilient to the influence of sleep when they have experience with the task.2 In addition, senior residents perceived themselves to be more resilient to the influence of sleep than junior residents.24 A senior resident or fellow participated in 40 of 46 procedures (87%) in our study. It is possible that the reported ESS scores may reflect a tendency among senior trainees to underreport their perceived sleepiness. Furthermore, our analyses did not account for the possibility that trainees may differ in their innate motor ability and its sensitivity to sleep deprivation. Although we estimated a population average effect in our analyses, measuring innate motor ability and its sensitivity to sleep deprivation can provide more granular insight into the association between sleepiness and the intraoperative technical skill of the trainees.
Our findings highlight a few hypotheses for further study. First, our analyses showed no statistically significant evidence that daytime sleepiness of trainees was differentially associated with their technical skill by sex, age, or postgraduate year. However, the marginal estimates for skill in our adjusted models suggest that female trainees were assigned lower scores by attending physician and trainee raters using SGAT and VAS. These findings suggest the possibility of a systematic difference in performance or skill assessment by trainees’ sex or a chance finding in our cohort. However, our findings do not currently support or refute any hypotheses on the source of any existent bias. Previous research in gastroenterology suggests a hypothesis that one source of systematic bias in technical skill is the sex of the evaluator and the trainee being evaluated.21 Therefore, our observation of differences in scores between male and female surgical trainees needs to be further studied. Second, the association between trainees’ daytime sleepiness and technical skill in rotations with different workloads should be assessed. Our findings and previous research suggest that the association between sleepiness and technical skill may be of larger magnitude in rotations with higher workloads.1 Third, the association between trainee sleepiness and technical skill acquisition in the operating room must be adequately characterized. Access to high-fidelity simulation technologies and an intensive focus on patient safety may make it unethical for trainees who have not progressed to a certain point on their learning curve to operate on patients.
Limitations
We studied a global level of sleepiness, but alternative measures of sleepiness or fatigue may be used in future research. Other scales such as the Stanford Sleepiness Scale or the Karolinska Sleepiness Scale, which measure short-term changes, may capture the effects of acute sleep deprivation.25 Although we studied sleepiness, other aspects of resident exhaustion may be better captured by measuring their fatigue using other scales, such as the Toronto Sleepiness and Fatigue Scale.25 We did not study the association of daytime sleepiness with learning curves of trainees in the operating room. In our study, trainees completed the ESS early in the morning before the first septoplasty procedure that day, which was nearly always the first procedure on the operative schedule. We did not study the association between daytime sleepiness of trainees and their technical skill for surgical procedures scheduled at different times during the day. We did not have a sufficient sample size to study additional interactions, such as trainees’ sex and age or sex and training level. For example, we did not see a significant association between ESS scores and trainee self-rated skill, which may have been either because of inaccurate self-assessments and/or interactions between trainee sex and training level, which we were unable to determine in this study. We also did not investigate whether trainees’ daytime sleepiness was associated with patient outcomes.
Conclusions
Daytime sleepiness in surgical trainees may be inversely associated with their intraoperative technical skill as assessed by a supervising attending physician. Our findings suggest that the ability of surgical trainees to acquire technical skill in the operating room may be influenced by their daytime sleepiness. Elucidating this association can inform attending physicians’ decisions regarding appropriate delegation of responsibility to trainees in the operating room.
References
- 1.Nida AM, Googe BJ, Lewis AF, May WL. Resident fatigue in otolaryngology residents: a web based survey. Am J Otolaryngol. 2016;37(3):210-216. doi: 10.1016/j.amjoto.2016.01.017 [DOI] [PubMed] [Google Scholar]
- 2.Institute of Medicine (US) Committee on Optimizing Graduate Medical Trainee (Resident) Hours and Work Schedule to Improve Patient Safety ; Ulmer C, Miller Wolman D, Johns MM, eds. Resident Duty Hours: Enhancing Sleep, Supervision, and Safety. Washington, DC: National Academies Press; 2009. [PubMed] [Google Scholar]
- 3.Bolster L, Rourke L. The effect of restricting residents’ duty hours on patient safety, resident well-being, and resident education: an updated systematic review. J Grad Med Educ. 2015;7(3):349-363. doi: 10.4300/JGME-D-14-00612.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Sturm L, Dawson D, Vaughan R, et al. Effects of fatigue on surgeon performance and surgical outcomes: a systematic review. ANZ J Surg. 2011;81(7-8):502-509. doi: 10.1111/j.1445-2197.2010.05642.x [DOI] [PubMed] [Google Scholar]
- 5.Sugden C, Athanasiou T, Darzi A. What are the effects of sleep deprivation and fatigue in surgical practice? Semin Thorac Cardiovasc Surg. 2012;24(3):166-175. doi: 10.1053/j.semtcvs.2012.06.005 [DOI] [PubMed] [Google Scholar]
- 6.Abt NB, Reh DD, Eisele DW, Francis HW, Gourin CG. Does resident participation influence otolaryngology-head and neck surgery morbidity and mortality? Laryngoscope. 2016;126(10):2263-2269. doi: 10.1002/lary.25973 [DOI] [PubMed] [Google Scholar]
- 7.Baker AB, Ong AA, O’Connell BP, Sokohl AD, Clinkscales WB, Meyer TA. Impact of resident involvement in outpatient otolaryngology procedures: an analysis of 17,647 cases. Laryngoscope. 2017;127(9):2026-2032. doi: 10.1002/lary.26645 [DOI] [PubMed] [Google Scholar]
- 8.Meier JC, Remenschneider AK, Gray ST, Holbrook EH, Gliklich RE, Metson R. The impact of surgical trainee participation on sinus surgery outcomes. Laryngoscope. 2016;126(2):316-321. doi: 10.1002/lary.25504 [DOI] [PubMed] [Google Scholar]
- 9.Vieira BL, Hernandez DJ, Qin C, Smith SS, Kim JY, Dutra JC. The impact of resident involvement on otolaryngology surgical outcomes. Laryngoscope. 2016;126(3):602-607. doi: 10.1002/lary.25046 [DOI] [PubMed] [Google Scholar]
- 10.Basaran K, Mercan ES, Aygit AC. Effects of fatigue and sleep deprivation on microvascular anastomoses. J Craniofac Surg. 2015;26(4):1342-1347. doi: 10.1097/SCS.0000000000001719 [DOI] [PubMed] [Google Scholar]
- 11.Taffinder NJ, McManus IC, Gul Y, Russell RC, Darzi A. Effect of sleep deprivation on surgeons’ dexterity on laparoscopy simulator. Lancet. 1998;352(9135):1191. doi: 10.1016/S0140-6736(98)00034-8 [DOI] [PubMed] [Google Scholar]
- 12.Tsafrir Z, Korianski J, Almog B, Many A, Wiesel O, Levin I. Effects of fatigue on residents’ performance in laparoscopy. J Am Coll Surg. 2015;221(2):564-70.e3. doi: 10.1016/j.jamcollsurg.2015.02.024 [DOI] [PubMed] [Google Scholar]
- 13.Uchal M, Tjugum J, Martinsen E, Qiu X, Bergamaschi R. The impact of sleep deprivation on product quality and procedure effectiveness in a laparoscopic physical simulator: a randomized controlled trial. Am J Surg. 2005;189(6):753-757. doi: 10.1016/j.amjsurg.2005.03.021 [DOI] [PubMed] [Google Scholar]
- 14.Yamany T, Woldu SL, Korets R, Badani KK. Effect of postcall fatigue on surgical skills measured by a robotic simulator. J Endourol. 2015;29(4):479-484. doi: 10.1089/end.2014.0349 [DOI] [PubMed] [Google Scholar]
- 15.Jakubowicz DM, Price EM, Glassman HJ, et al. Effects of a twenty-four hour call period on resident performance during simulated endoscopic sinus surgery in an accreditation council for graduate medical education-compliant training program. Laryngoscope. 2005;115(1):143-146. doi: 10.1097/01.mlg.0000150689.77764.ad [DOI] [PubMed] [Google Scholar]
- 16.Reznick RK, Folse JR. Effect of sleep deprivation on the performance of surgical residents. Am J Surg. 1987;154(5):520-525. doi: 10.1016/0002-9610(87)90269-8 [DOI] [PubMed] [Google Scholar]
- 17.Johns MW. A new method for measuring daytime sleepiness: the Epworth sleepiness scale. Sleep. 1991;14(6):540-545. doi: 10.1093/sleep/14.6.540 [DOI] [PubMed] [Google Scholar]
- 18.Johns MW. Reliability and factor analysis of the Epworth Sleepiness Scale. Sleep. 1992;15(4):376-381. doi: 10.1093/sleep/15.4.376 [DOI] [PubMed] [Google Scholar]
- 19.Obeid AA, Al-Qahtani KH, Ashraf M, Alghamdi FR, Marglani O, Alherabi A. Development and testing for an operative competency assessment tool for nasal septoplasty surgery. Am J Rhinol Allergy. 2014;28(4):e163-e167. doi: 10.2500/ajra.2014.28.4051 [DOI] [PubMed] [Google Scholar]
- 20.Reznick R, Regehr G, MacRae H, Martin J, McCulloch W. Testing technical skill via an innovative “bench station” examination. Am J Surg. 1997;173(3):226-230. doi: 10.1016/S0002-9610(97)89597-9 [DOI] [PubMed] [Google Scholar]
- 21.Thackeray EW, Halvorsen AJ, Ficalora RD, Engstler GJ, McDonald FS, Oxentenko AS. The effects of gender and age on evaluation of trainees and faculty in gastroenterology. Am J Gastroenterol. 2012;107(11):1610-1614. doi: 10.1038/ajg.2012.139 [DOI] [PubMed] [Google Scholar]
- 22.Pan W. Akaike’s information criterion in generalized estimating equations. Biometrics. 2001;57(1):120-125. doi: 10.1111/j.0006-341X.2001.00120.x [DOI] [PubMed] [Google Scholar]
- 23.R Core Team R: A Language and Environment for Statistical Computing [computer program]. Vienna, Austria: R Foundation for Statistical Computing; 2017. https://www.R-project.org/. Accessed September 7, 2018.
- 24.Woodrow SI, Park J, Murray BJ, et al. Differences in the perceived impact of sleep deprivation among surgical and non-surgical residents. Med Educ. 2008;42(5):459-467. doi: 10.1111/j.1365-2923.2007.02963.x [DOI] [PubMed] [Google Scholar]
- 25.Shahid A, Shen J, Shapiro CM. Measurements of sleepiness and fatigue. J Psychosom Res. 2010;69(1):81-89. doi: 10.1016/j.jpsychores.2010.04.001 [DOI] [PubMed] [Google Scholar]
