Abstract
Background: The American College of Physicians’ (ACP) Internal Medicine In-Training Examination (IM-ITE) is designed to evaluate the cognitive knowledge of residents to aid them and program directors in evaluating the training experience. Objective: To determine the impact of the curriculum reform accompanied by the Accreditation Council for Graduate Medical Education (ACGME)-I alignment and accreditation on the internal medicine residency program (IMRP) using residents’ performance in the ACP's ITE from 2008 to 2016, and where the IMRP stands in comparison to all ACGME and ACGME-I accredited programs. Methods: This is a descriptive study conducted at a hospital-based IMRP in Doha, Qatar from 2008 to 2016. The study population is 1052 residents at all levels of training in IMRP. The ACP-generated ITE results of all the United States and ACGME-I accredited programs were compared with IM-ITE results in Qatar. These results were expressed in the total program average and the ranking percentile. Results: There is a progressive improvement in resident performance in Qatar as shown by the rise in total average program score from 52% in 2008 to 72% in 2016 and the sharp rise in percentile rank from 3rd percentile in 2008 to 93rd percentile in 2016 with a dramatic increase during the period 2013 to 2014 (from 32nd percentile to 73rd percentile), which represents the period of ACGME-I accreditation. None of the factors (ethnicity, USMLE or year of residency) were statistically significant with a p value >0.05 and standard coefficient ( − 0.017–0.495). There was negligible correlation between the USMLE test scores with the residents’ ITE scores with a p value = 0.023 and a Pearson correlation r = 0.097. Conclusion: The initial ACGME-I alignment followed by the accreditation, together with whole curriculum redesign to a structured, competency-based program starting from 2008, has led to an improvement in the ITE scores in the IMRP. This was further evidenced by the lack of change in the residency entry selection criteria.
Keywords: internal medicine residency, in-training examination, ACGME-I, American College of Physicians
Introduction
The Internal Medicine In-Training Examination (ITE) was first created in 1988 as an instrument to aid residents’ self-assessment and to detect residents at risk of not passing their certifying examination.1 It is a standardized test organized by the American College of Physicians (ACP) and administered annually to residents at all training levels. The results are reported as a percentage of correct answers by individual residents and percentile ranking of the national and international programs participating. The ITE is useful and frequently initiates educational interventions based on results.2,3 It is an effective tool for evaluating the improvement in resident's base of knowledge over time4 and a good means of predicting residents’ future performance.5 Although several publications have reported interventions to improve ITE scores among residents such as educational conferences,6 online cases,7 practice questions,8 and group learning,9 we were unable to identify publications that specifically relate the Accreditation Council for Graduate Medical Education (ACGME) accreditation process to enhanced residents’ ITE scores.
The ACGME is the main accreditor of Graduate Medical Education programs in the United States. The ACGME-International (ACGME-I) was initiated consequent to petitioning by governments and institutions that the ACGME extend its accreditation model internationally. The mission of ACGME-I is to improve health care by assessing and advancing the quality of resident physicians’ education through accreditation.10 In July 2013, the internal medicine residency program (IMRP) at Hamad Medical Corporation, the primary and largest health care institution in the state of Qatar, was accredited by the ACGME-I. This period was preceded by an educational redesign phase, which was evident in the preparation for, and continued after, the accreditation process. The IMRP leaders invested substantial amount of time in adopting a holistic approach and integration of different measures in the educational curriculum. The purpose of this study was to determine the impact of the ACGME-I alignment and accreditation on the IMRP using residents’ performance in the ACP's ITE from 2008 to 2016 and where the IMRP stands in comparison to all ACGME and ACGME-I accredited programs.
Methods
Design
This is a retrospective cohort study that was conducted at the Internal Medicine Department at Hamad Medical Corporation (HMC) in Qatar. Data was extracted from the ACP-generated ITE results of all the United States and ACGME-International accredited programs. Then they were compared with the IMRP residents’ IM-ITE results, also ACP-generated. These results were expressed in two formats; the total program average and the ranking percentile, both reported by the ACP. The average program scores at the ITE examination was compared as well to the residents’ entry selection criteria.
Study participants
During the study period (2008–2016), a total of 366 new residents joined the IMRP. 74% were male and 26% were females. Among the 366 residents, the average duration from graduation year to joining the residency program was found to be 3 years. All residents sat for either the USMLE step 2 or the IFOM exams as a prerequisite for acceptance to the residency program. The critera for residency is set by the medical education department at HMC and include the following: all residents should be recent university graduates (i.e. less than 3 years) and demonstrate proficiency in English (TOFEL/ILETS). In addition, scores at the USMLE step 2 or International Foundation of Medicine Clinical Science Exam (IFOM-CSE) are considered. The USMLE step 2 scores are reported in 3 digits and fall between 1 and 300, with the mean score of 240 while the IFOM. The average score of the USMLE over the study period was found to be 248, and for the IFOM 660 which is approximately equivalent to 231 USMLE score.
During the four-year residency training, all residents in IMRP participate yearly in the ACP's IM-ITE. The examination was made mandatory for all residents to evaluate their progress and decide on measures for improvement. As all residents at all training levels sit for the ITE examination every year, a total of 1052 candidates sat for the IM-ITE during the study period. Their demographic characteristics are shown in Table 1. The study was approved by the medical research committee.
Table 1.
Participants’ (residents who sat for the ITE during the study period) demographic data.
| Total Number of residents in each year who sat for ITE | |||||||||
|
| |||||||||
| Training year | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2106 |
|
| |||||||||
| PGY 1 | 26 | 39 | 38 | 36 | 13 | 37 | 28 | 43 | 0 |
|
| |||||||||
| PGY 2 | 13 | 26 | 37 | 37 | 34 | 45 | 42 | 35 | 43 |
|
| |||||||||
| PGY 3 | 18 | 11 | 23 | 34 | 34 | 31 | 41 | 36 | 26 |
|
| |||||||||
| PGY 4 | 13 | 17 | 10 | 22 | 32 | 34 | 30 | 35 | 33 |
|
| |||||||||
| Total no. of residents/year | 70 | 93 | 108 | 129 | 113 | 147 | 141 | 149 | 102 |
|
| |||||||||
| Total no. of residents | 1052 | ||||||||
|
| |||||||||
| Gender | Male (74%) | ||||||||
|
| |||||||||
| Female (26%) | |||||||||
|
| |||||||||
| Nationalities | Pakistani 20% | ||||||||
|
| |||||||||
| Jordanian 18% | |||||||||
|
| |||||||||
| Sudanese 15% | |||||||||
|
| |||||||||
| Egyptian 13% | |||||||||
|
| |||||||||
| 5% Palestinian, Indian and Syrian | |||||||||
|
| |||||||||
| 3% Qatari | |||||||||
|
| |||||||||
PGY: Postgraduate year, ITE: In-Training Exam.
IMRP Reform Elements:
Implementing the ACGME-I program requirements
The initial ACGME-I alignment from 2008 to 2012, followed by the full adoption of the ACGME-I's IM program specific requirements from 2013 onwards that resulted in the accreditation, has culminated in a competency-based and structured educational curriculum with clear goals and objectives for various rotations. Also, the adoption of strict resident supervision, the formative and summative assessments utilizing the ACGME-I standardized assessment tools such as direct observation, Mini-CEX, chart stimulated recall, and OSCE all contributed to improving the IMRP residents’ performance.
Improving the morning Report
The morning report is the most valuable educational activity in the IMRP. In 2012, key changes to the format of the morning report were implemented, including improving organization, adding variety to the contents, enhancing case selection and the quality of presentations, and introducing patient safety and quality improvement topics into discussions. These changes were directed by residents’ suggestions.
Introduction of the annual board review and clinical exam preparation courses
As an initiative to improve the residents’ medical knowledge and board scores, IMRP introduced a comprehensive board review course, an evidence-based review that is delivered by the faculty of general IM and specialties in an interactive MCQs format. Delivered annually, this intensive IM board review course is a live, face-to-face, 4-day course. Materials are made available to all the residents through the electronic residency management program (Medhub) to allow them to study on their own time. Similarly, the hands-on clinical exam preparation course was introduced annually to improve residents’ performance in this domain.
Adapting an ambulatory curriculum
IMRP provided high-quality ambulatory care where IM residents are exposed to a variety of medical patients whose medical issues range from simple to more complex. Under faculty supervision, residents provide longitudinal care to this population and at the same time discuss different cases and challenges that they may encounter in ambulatory care. Additionally, the daily ambulatory educational activity, in which residents present various ambulatory related topics under faculty supervision and guidance, is conducted for one hour in the morning before the residents start their clinics.
The Mentoring and Remediation Program
IMRP has adopted and implemented trainee-centered educational approaches by developing the mentoring remediation program where the residency program focuses on filling the training practice gap by providing tailored educational experiences and remedial plans to residents who were not able to achieve proficiency in core competencies as demonstrated by their ITE scores as well as all other formative and summative evaluations.
Duty Hours restrictions
The ACGME implemented work hours restrictions for physicians in training in 2003 that were revised July 1, 2011. The IMRP in Qatar implemented duty hour restrictions in 2011, which allowed more protected time for residents’ education.
Data analysis
Descriptive statistics were used to summarize and determine the sample characteristics and distribution of responses. The normally distributed data and results were reported with mean and standard deviation; the remaining results were reported with median and interquartile range (IQR). Categorical data were summarized using frequencies and percentages. A Pearson correlation was used to examine and assess the correlation coefficient between USMLE step 2/IFOM scores and the residents’ ITE scores and test its statistical significance using corresponding p values. The Linear regression method was applied to assess the effect of different factors such as year, year of residency, and nationality on ITE scores. All p values presented were two-tailed, and p values < 0.05 was be considered as statistically significant. All Statistical analyses were done using statistical package SPSS version 23.0 (SPSS Inc. Chicago, IL).
Results
Qatar ITE scores as compared with all programs
There is a progressive improvement in performance of the residents in Qatar as shown by the total program average scores from 52% in 2008 to 72% in 2016. This is demonstrated in the histogram charts (Figure 1) and further summarized in the graph shown in Figure 2, where the mean ITE score was 61.48+/ − 12.4 (range 18–95; IOR 54–70). Additionally, the IMRP percentile rank also showed a steady and progressive improvement from 3rd percentile in 2008 to 93rd percentile in 2016, with a dramatic increment during the period from 2013 to 2014 (from 32nd percentile to 73rd percentile), which corresponds with the time of ACGME-I accreditation (Figure 3).
Figure 1.
Qatar total program average scores during the study period.
Figure 2.
Qatar ITE scores during the study period.
Figure 3.
Qatar IMRP ITE percentile scores during the study period.
Moreover, this study has shown that internal medicine residents at HMC have progressively outperformed many other ACGME and ACGME-I accredited programs starting from 2013 through 2016 as demonstrated in Figures 1 and 4.
Figure 4.
Comparison of the Qatar ITE scores and percentile rank with all other ACGME and ACGME-I accredited programs during the study period.
In order to determine the other factors, we compared the ITE scores to the residents’ entry selection criteria. As all residents were recent graduates (less than 3 years), and the ITE scores were compared to the residents’ ethnicity, year of residency, and USMLE score. We did regression analysis for the factors of ethnicity, USMLE and year of residency, and none of them were statistically significant, with a p value >0.05 and standard coefficient ( − 0.017–0.495). There was negligible correlation between the USMLE test scores with the residents’ ITE scores with a p value = 0.023 and a Pearson correlation r = 0.097 as shown in 4 and 5.
Figure 5.
Comparison of USMLE and ITE scores during the study period.
Discussion
The ACGME-I curriculum reform has resulted in a dramatic increase at the residents’ ITE scores. While resident selection criteria based on USMLE scores determine the quality of residents accepted into the residency program, this study did not find a correlation with their performance during the program as reflected by their ITE scores. This is the first study that relates the ACGME-I accreditation to improvement in residents’ ITE scores.
The core competencies of internal medicine define the minimum level of knowledge, skills, and attitudes that residents must attain for advancement to independent practice, fellowship, or other internal medicine career pathways. Residents should attain the cognitive skills of diagnostic reasoning, clinical examination, and the strategic use of testing and consultation to make diagnoses and implement treatment. Unlike the ACGME competencies in interpersonal and communication skills, professionalism, system-based practice, and practice-based learning and improvement, which apply to all specialties of medicine, medical knowledge competency and, to some extent, patient care competency, are more discipline-specific.11
The IM-ITE has been shown to be a validated measure that assesses the residents’ knowledge base during the training period.12 Additionally, IM-ITE is a valid measure of knowledge acquired during internal medicine training. Further, the ITE scores have been shown to correlate closely with the pass–fail results on the certifying examination of the American Board of Internal Medicine (CABIM).10,11,13,14 Numerous criteria have been utilized in an attempt to identify predictors of resident performance in the ITE as defined primarily by clinical work, professionalism, and performance on standardized examinations. For instance, performance on the ITE can accurately predict, and is highly correlated with, performance on the CABIM qualifying exam.13 The ITE can serve residents as an important measure of their preparedness for certification and can be useful in identifying the need for more intensive self-study strategies during the subsequent years14 as there is a high correlation between scores on the ITE and the American Board of Surgery Qualifying Examinations.15
Many factors have been linked to enhanced resident performance in the IM-ITE. Weekly reading assignments and weekly preparatory examinations have shown significant improvement in the American Board of Surgery ITE (ABSITE) scores.16 Conference attendance, previous performance, probationary status, amount of sleep, and amount of study contribute to residents’ successful ABSITE performance.17 Residents’ self-study and presentation program have been found to improve performance on Thoracic Surgery ITE.18
The morning report is one of the most ubiquitous and educationally useful conferences employed by internal medicine training programs. Its main goals are reviewing management decisions, examining therapeutic problems, and most importantly, acting as a case-oriented teaching session. During the curricular reform, there was a transformation of the morning report format that was resident-driven, resident-led, and that produced resident-focused learning that met residents’ needs.19 Moreover there was an adaptation of the ambulatory morning report curriculum. As the ACGME Program Requirements in Internal Medicine that became effective in July, 2001, states that clinical experiences must include a minimum of one-third of the time in ambulatory sites,20,21 residents complete a substantial amount of their training in the outpatient setting. Studies showed that use of dedicated ambulatory teaching residents is an appropriate mechanism to expose internal medicine residents to the breadth of topics covered by the American Board of Internal Medicine (ABIM) exam.22
Some studies relayed the improvement in ITE scores to the residency selection criteria including scores in USMLE step 1, where residents who had previously scored above 220 on the USMLE step I, had higher average Orthopedics ITE scores than those scoring below 220.23 In this study, the striking improvement in ITE scores among residents at IMRP in Qatar was not found to be statistically correlated to the residents USMLE score explained by reforming the educational curriculum during the ACGME-I accreditation period rather than a greater proficiency among the IMRP residents, as there were no changes in the admission criteria or the demographic characteristics of the residents.
Multimodal approaches for remediation of surgery,24 emergency medicine,25 and radiology26 residents have been shown to improve board scores. While remediation requires substantial resources, it was found to be successful for 90% of learners27. As part of the curriculum reform, the IMRP have adopted the Hauer et al.28 remediation trainee-centered model, which included assessment of the learner's competence, diagnosis of the learner's deficiency, and development of an individualized learning plan incorporating deliberate practice, feedback, reflection, and, finally, a focused reassessment. Through the biannually formative evaluation forms and the annual ITE scores, the program evaluation committee (PEC) identified the underperforming trainees and the lacking competencies. Each resident is assigned a mentor who provides feedback and encourage trainee's self-reflection.
The ACGME implemented work hours restrictions for physicians in training in 2003 that were revised July 1, 2011. The IMRP in Qatar has implemented the duty hour restrictions in 2011 which allowed more protected time for the residents’ education. While there was no statistically significant change in ITE performance after the implementation of the new duty hour regulations as shown by Vucicevic et al,29 or a change in general surgery patient outcomes or differences in resident examination performance,30 it resulted in lower levels of stress and improvement in the educational environment, as well as in providing adequate time for sleep, preventing fatigue, and improving the quality of life and overall well-being of the residents.31–33 Restricting physician work hours was thus part of the measures implemented to improve patient safety (National Research Council 2009).34
The main limitation of this study is the generalizability as it is based on a single ACGME-I accredited institution. It is suggested that future studies look for other ACGME-I accredited centers and relate accreditation to residents’ ITE scores.
Conclusion
While improvement in residents’ ITE scores is multi-factorial, the ACGME-I accreditation and restructuring of the program to competency-based curriculum have enriched the residents’ experience and resulted in improvement in the residents’ performance, as reflected by their ITE scores. This is the first study that have linked the ACGME-I accreditation to improvement in residents’ ITE scores. This reform included measures such as adopting continuous and formative residents assessment utilizing approved standardized assessment tools, implementation of the duty hour standards, introducing the mentoring remediation program, developing and following a strict adherence to masterplan of residents rotations including ambulatory care and consult rotation as well as mandating all medical specialties rotations.
References
- 1.Rollins LK, Martindale JR, Edmond M, Manser T, Scheld WM. Predicting pass rates on the American Board of Internal Medicine certifying examination. J Gen Intern Med. 1998;13(6):414–416. doi: 10.1046/j.1525-1497.1998.00122.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Garibaldi RA, Trontell MC, Waxman H, Holbrook JH, Kanya DT, Khoshbin S et al. The In-Training Examination in Internal Medicine. Ann Intern Med. 1994;121(2):117–123. doi: 10.7326/0003-4819-121-2-199407150-00008. [DOI] [PubMed] [Google Scholar]
- 3.Hawkins RE, Sumption KF, Gaglione MM, Holmboe ES. The in-training examination in internal medicine: resident perceptions and lack of correlation between resident scores and faculty predictions of resident performance. Am J Med. 1999;106(2):206–210. doi: 10.1016/s0002-9343(98)00392-1. [DOI] [PubMed] [Google Scholar]
- 4.Holmboe ES, Hawkins RE. Methods for evaluating the clinical competence of residents in internal medicine: a review. Ann Intern Med. 1998;129(1):42–48. doi: 10.7326/0003-4819-129-1-199807010-00011. [DOI] [PubMed] [Google Scholar]
- 5.Kay C, Jackson JL, Frank M. The relationship between internal medicine residency graduate performance on the ABIM certifying examination, yearly in-service training examinations, and the USMLE Step 1 examination. Acad Med. 2015;90(1):100–104. doi: 10.1097/ACM.0000000000000500. [DOI] [PubMed] [Google Scholar]
- 6.McDonald FS, Zeger SL, Kolars JC. Associations of conference attendance with internal medicine in-training examination scores. Mayo Clin Proceed. 2008;83(4):449–453. doi: 10.4065/83.4.449. [DOI] [PubMed] [Google Scholar]
- 7.Bode RS, Caputo GL. Reading program success on in-training and certifying exam scores. Acad Pediatrics. 2010; 10:7. Conference: 63rd Annual Session of the American Academy of Pediatric Dentistry, AAPD, Chicago, IL. Conference Start: 20100527 Conference End: 20100530. Conference Publication: (var.pagings)
- 8.Harthun NL, Schirmer BD, Sanfey H. Remediation of low ABSITE scores. Curr Surg. 2005;62(5):539–542. doi: 10.1016/j.cursur.2005.04.020. [DOI] [PubMed] [Google Scholar]
- 9.Shokar GS. The effects of an educational intervention for ‘at-risk’ residents to improve their scores on the Intraining Exam. Fam Med. 2003;35(6):414–417. [PubMed] [Google Scholar]
- 10.Accreditation Council for Graduate Medical Education. Manual of policies and procedures. Revised September 23, 2017. https://www.acgme.org/Portals/0/PDFs/ab_ACGMEPoliciesProedures.pdf. Accessed November 20, 2017.
- 11.Meyers FJ, Weinberger SE, Fitzgibbons JP, Glassroth J, Duffy FD, Clayton CP. Alliance for Academic Internal Medicine Education Redesign Task Force. Redesigning residency training in internal medicine: the consensus report of the Alliance for Academic Internal Medicine Education Redesign Task Force. Acad Med. 2007;82:1211–1219. doi: 10.1097/ACM.0b013e318159d010. [DOI] [PubMed] [Google Scholar]
- 12.Garibaldi RA, Subhiyah R, Moore ME, Waxman H. The in-training examination in internal medicine: an analysis of resident performance over time. Ann Intern Med. 2002;137(6):505–510. doi: 10.7326/0003-4819-137-6-200209170-00011. [DOI] [PubMed] [Google Scholar]
- 13.Grossman RS, Fincher RM, Layne RD, Seelig CB, Berkowitz LR, Levine MA. Validity of the in-training examination for predicting American Board of Internal Medicine certifying examination scores. J Gen Intern Med. 1992;7(1):63–67. doi: 10.1007/BF02599105. [DOI] [PubMed] [Google Scholar]
- 14.Waxman H, Braunstein G, Dantzker D, Goldberg S, Lefrak S, Lichstein E et al. Performance on the internal medicine second-year residency in-training examination predicts the outcome of the ABIM certifying examination. J Gen Intern Med. 1994;9(12):692–694. doi: 10.1007/BF02599012. [DOI] [PubMed] [Google Scholar]
- 15.Shetler PL. Observations on the American Board of Surgery in-training examination, board results, and conference attendance. Am J Surg. 1982;144(3):292–294. doi: 10.1016/0002-9610(82)90002-2. [DOI] [PubMed] [Google Scholar]
- 16.de Virgilio C, Chan T, Kaji A, Miller K. Weekly assigned reading and examinations during residency, ABSITE performance, and improved pass rates on the American Board of Surgery Examinations. J Surg Educ. 2008;65(6):499–503. doi: 10.1016/j.jsurg.2008.05.007. [DOI] [PubMed] [Google Scholar]
- 17.Godellas CV, Huang R. Factors affecting performance on the American Board of Surgery in-training examination. Am J Surg. 2001;181:294–296. doi: 10.1016/s0002-9610(01)00576-1. [DOI] [PubMed] [Google Scholar]
- 18.Bull DA, Stringham JC, Karwande SV, Neumayer LA. Effect of a resident self-study and presentation program on performance on the thoracic surgery in-training examination. Am J Surg. 2001;181(2):142–144. doi: 10.1016/s0002-9610(00)00567-5. [DOI] [PubMed] [Google Scholar]
- 19.Dousa KM, Muneer M, Rahil A, Al-Mohammed A, AlMohanadi D, Elhiday A et al. Tailoring morning reports to an internal medicine residency in Qatar. J Grad Med Educ. 2014;6(4):801–804. doi: 10.4300/JGME-D-14-00489.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Larson DB. Accreditation Council for Graduate Medical Education (ACGME) AJR Am J Roentgenol. 2007;188(1):3–4. doi: 10.2214/AJR.07.6100. [DOI] [PubMed] [Google Scholar]
- 21.Dugdale DC. Ambulatory internal medicine training: challenges and opportunities. J Gen Intern Med. 2003;18(4):313–314. doi: 10.1046/j.1525-1497.2003.30249.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Wenderoth S, Pelzman F, Demopoulos B. Ambulatory morning report: can it prepare residents for the American Board of Internal Medicine examination? J Gen Intern Med. 2002;17(3):207–209. doi: 10.1046/j.1525-1497.2002.10202.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Carmichael KD, Westmoreland JB, Thomas JA, Patterson RM. Relation of residency selection factors to subsequent orthopaedic in-training examination performance. South Med J. 2005;98(5):528–532. doi: 10.1097/01.SMJ.0000157560.75496.CB. [DOI] [PubMed] [Google Scholar]
- 24.Borman KR. Does academic intervention impact ABS qualifying examination results? Curr Surg. 2006;63(6):367–372. doi: 10.1016/j.cursur.2006.06.007. [DOI] [PubMed] [Google Scholar]
- 25.Sharma R, Sperling JD, Greenwald PW, Carter WA. A novel comprehensive in-training examination course can improve residency-wide scores. J Grad Med Educ. 2012;4(3):378–380. doi: 10.4300/JGME-D-11-00154.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Edeiken BS. Remedial program for diagnostic radiology residents. Invest Radiol. 1993;28(3):269–274. doi: 10.1097/00004424-199303000-00025. [DOI] [PubMed] [Google Scholar]
- 27.Guerrasio J, Garrity MJ, Aagaard EM. Learner deficits and academic outcomes of medical students, residents, fellows and physicians referred to a remediation Program, 2006-2012. Acad Med. 2014;89(2):352–358. doi: 10.1097/ACM.0000000000000122. [DOI] [PubMed] [Google Scholar]
- 28.Hauer KE, Teherani A, Kerr KM, O'Sullivan PS, Irby DM. Impact of the United States Medical Licensing Examination Step 2 on medical school clinical skills assessment. Acad Med. 2006;81(10):S13–S16. doi: 10.1097/01.ACM.0000236531.32318.02. [DOI] [PubMed] [Google Scholar]
- 29.Vucicevic D, Mookadam F, Webb BJ et al. The impact of 2011 ACGME duty hour restrictions on internal medicine resident workload and education. Adv Health Sci Educ Theory Pract. 2015;20(1):193–203. doi: 10.1007/s10459-014-9525-5. [DOI] [PubMed] [Google Scholar]
- 30.Rajaram R, Chung JW, Jones AT, Cohen ME, Dahlke AR, Ko CY et al. Association of the 2011 ACGME resident duty hour reform with general surgery patient outcomes and with resident examination performance. JAMA. 2014 Dec;312(22):2374–2384. doi: 10.1001/jama.2014.15277. [DOI] [PubMed] [Google Scholar]
- 31.Fletcher KE, Davis SQ, Underwood W, Mangrulkar RS, McMahon LF, Jr, Saint S. Systematic review: effects of resident work hours on patient safety. Ann Intern Med. 2004;141(11):851–857. doi: 10.7326/0003-4819-141-11-200412070-00009. [DOI] [PubMed] [Google Scholar]
- 32.Landrigan CP, Fahrenkopf AM, Lewin D, Sharek PJ, Barger LK, Eisner M et al. Effects of the Accreditation Council for Graduate Medical Education duty hour limits on sleep, work hours, and safety. Pediatrics. 2008;122(2):250–258. doi: 10.1542/peds.2007-2306. [DOI] [PubMed] [Google Scholar]
- 33.Hutter MM, Kellogg KC, Ferguson CM, Abbott WM, Warshaw AL. The impact of the 80-hour resident workweek on surgical residents and attending surgeons. Ann Surg. 2006;243(6)(864): discussion871–875. doi: 10.1097/01.sla.0000220042.48310.66. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Ulmer C, Wolman DM, Johns MME. Washington, DC: The National Academy Press; 2008. Committee on optimizing graduate medical trainee (resident) hours and work schedule to improve patient safety. (Resident duty hours: Enhancing sleep, supervision and safety, Washington, DC: The National Academy Press). [Google Scholar]