Abstract
Background
The in-training examination (ITE) offers formative assessments of residents' developing medical knowledge. Identification of an ITE performance level associated with success on the specialty board examination allows identification of “at risk” residents.
Objective
This study sought to identify a threshold score for obstetrics and gynecology residents' performance on the Council on Resident Education in Obstetrics and Gynecology (CREOG) ITE that predicts successful performance on the American Board of Obstetrics and Gynecology (ABOG) written examination.
Methods
We analyzed ITE and ABOG results of 80 residents who completed 4 years of CREOG ITEs at 2 institutions between 2002 and 2012. We assessed the level of performance associated with successful performance on the ABOG written examination.
Results
Data analyzed included scores for 71 of 80 residents (89%), with an overall pass rate of 82%. A postgraduate year (PGY) 4 score of 200 on the CREOG ITE or twice in any of the PGY training years was associated with a 100% ABOG pass rate. Scoring ≥ 205 in any PGY also was associated with a 100% pass rate. Residents who did not attain a score of 200 had a 35% to 45% chance of failing the ABOG written examination, depending on the PGY of the ITE performance.
Conclusions
Our findings suggest that a CREOG ITE score of at least 200 twice, or as a PGY-4, offers assurance of successful performance on the ABOG examination. Scores lower than this threshold may be used to identify “at risk” residents for added learning and provide program elements in need of improvement.
What was known and gap
Being able to use the in-training examination (ITE) to predict success on the board examination would facilitate identification of residents in need of additional learning.
What is new
A Council on Resident Education in Obstetrics and Gynecology ITE score of at least 200 offers assurance of success on the American Board of Obstetrics and Gynecology examination.
Limitations
The study was conducted in 2 community-based programs, which may limit generalizability.
Bottom line
An ITE score of lower than 200 is useful for identifying “at risk” residents, as well as program areas in need of improvement.
Introduction
The Council on Resident Education in Obstetrics and Gynecology (CREOG) in-training examination (ITE) serves as a tool to provide a formative assessment of resident competence in medical knowledge, as well as evaluate a program's strengths and areas for improvement, as required by the Accreditation Council for Graduate Medical Education (ACGME). To provide an outcome-based measure in congruence with the ACGME Outcomes Project and Core Measures of a competent physician, an earlier study examined 69 postgraduate year (PGY) 4 residents.1 This study found that the ITE was a weak predictor of failure on the examination for a score less than 187.5, but a strong predictor of success if a resident scored greater than 187.5.1 However, the correlation was weak (r = 0.38), leaving further questions regarding use of the ITE for the purpose of predicting examination performance.
The objective of this study was to identify a threshold score on the CREOG ITE for success on the American Board of Obstetrics and Gynecology (ABOG) written examination. Our hypothesis was that the CREOG ITE can be used as a predictor of ABOG examination outcomes.
Methods
Data Collection
Data were collected at 2 urban independent academic institutions located in Pennsylvania and South Carolina (Reading Hospital and Greenville Health System). The sample for the study encompassed 80 residents who took the CREOG ITE examination over a 4-year period, from 2002–2009 (Greenville) and 2002–2012 (Reading), respectively. During the study years, neither program provided organized group level nor 1-on-1 ABOG reviews. We examined whether a given level of performance on the CREOG ITE scores would predict success on the ABOG written examination.
Institutional Review Board exemption was obtained from Reading Hospital and Greenville Health System Institutional Review Boards.
Data Analysis
Data were analyzed using logistic regression to identify key factors in predicting ABOG written examination outcomes. Categorical data were analyzed using χ2 tests. Sensitivity and specificity, and positive and negative predictive values, were calculated and a receiver operating characteristic (ROC) curve was generated from which an area under the curve (AUC) was calculated. SAS version 9.3 (SAS Institute Inc, Cary, NC) was used for data analyses, and P < .05 was deemed statistically significant.
Results
Seventy-one of 80 residents (89%) were included in the analysis (figure 1). Of these, 58 residents (82%) passed the ABOG written examination on the first attempt. Female residents made up a larger proportion of the sample than male residents (70% versus 30%). Average CREOG ITE scores differed between residents who passed versus failed in each PGY year (figure 2).
Figure 1.
Flow Diagram of Residents Analyzed by Institution
Figure 2.
Average CREOG In-Training Examination Scores by Year
Abbreviation: CREOG, Council on Resident Education in Obstetrics and Gynecology.
Of the 58 residents who passed the ABOG written examination, 7 scored less than 200 in all 4 years, and 51 scored 200 at least once on the CREOG ITE. Of the 13 residents who failed, 2 scored 200 once. There were 4 residents who passed the ABOG without scoring above 200 in any of their 4 attempts. A score of 200 on the CREOG ITE as a PGY-4, or 200 twice during PGY-1 to PGY-4, was associated with a 100% ABOG written examination pass rate (P < .0001). A score of ≥ 205 in any training year also was associated with a 100% pass rate. If the resident was unable to obtain 200, as defined above, the resident had a 35% to 45% chance of failing the ABOG written examination depending on the PGY that the ITE was taken (table). There was also a difference in score trends in those who failed versus those who passed. Those who failed the ABOG written examination scored progressively lower by PGY, while those who passed scored progressively higher in each per PGY (P = .001).
Table.
Area Under the Curve of CREOG ITE Score in Predicting ABOG Written Examination Outcome by Year Associated With 100% ABOG Written Success and Chance of Failing the ABOG Written Examination
In an effort to identify a predictor for success on the ABOG written examination, sensitivity and the AUC were calculated. The positive predictive value (PPV) of passing was 98% in those who obtained a score of ≥ 196 in PGY-1 or PGY-2 (P < .05), whereas scoring ≥ 198 in PGY-1 had a 100% PPV (P < .0001). Scoring ≥ 202 in PGY-2 had a 100% PPV (P < .0001). The PPV of passing was 100% in those who obtained a score of ≥ 204 in PGY-3 and ≥198 in PGY-4 (P < .0001). Negative predictive values were less than 45% for the aforementioned thresholds in each PGY. Calculations using all 4 years resulted in an AUC of 0.920, significantly better (P < .001) than the null hypothesis of 50%.
Trends in CREOG ITE scores over the 4 years appear to be important as well. Of those residents who failed the ABOG examination, none reached the PPV thresholds described above. In addition, the CREOG ITE scores trend downward (ie, have a negative slope) when compared with those who passed the ABOG examination (figure 3). The 4 residents who passed the ABOG examination without scoring above 200 at any time during training had scores that trended positively.
Figure 3.
Trends of CREOG Score by Year of Residents Who Passed and Failed the ABOG Written Examination
Abbreviations: CREOG, Council on Resident Education in Obstetrics and Gynecology; ABOG, American Board of Obstetrics and Gynecology; PGY, postgraduate year.
Discussion
Prediction of ABOG written examination success is correlated with residents scoring successively better each year the ITE was taken with thresholds for passing identified. The PPV is ≥ 98% for passing based on attaining scores of at least 196, depending on PGY.
One study examined CREOG ITE scores of 26 PGY-3 residents in 2 programs, and then compared them to ABOG pass rates. The findings showed that a score of less than 190 was associated with a risk of failing the ABOG written examination.2 This study is similar in design and produces similar findings. One key difference is the calculation of the PPV, which could be used by residents and programs to assess the adequacy of preparation for the ABOG written examination. The other key area of difference is that our study focused on data for all PGYs, which provides an earlier identification and intervention for residents and programs.
Another study of 194 internal medicine residents to develop a predictive tool found that ITE scores correlated in all PGYs with the best predictive value in PGY-3,3 and a study of 97 anesthesiology residents found that the ITE was a strong predictor of passing the written board examination.4 Both studies, and their findings, were similar to ours.
Spellacy et al2 noted that scores less than 190 were associated with an increased risk of failing the board examination. Our study focused on the performance associated with success on the ABOG examination, not performance associated with failing the ABOG written examination. However, not meeting the criterion of greater than 200 in PGY-4, or greater than or equal to 200 in any 2 PGYs, was associated with a 35% to 45% chance of failing the ABOG written examination.
Limitations of this study include generalizability, given that data were collected at only 2 institutions, and both were university-affiliated, community programs of similar size. Future studies should include detailed analysis of the percentage of correct scores and possible development of a nomogram for predicting success.
Conclusion
Based on this study, residents who attained a CREOG ITE score of at least 200 predicts assurance of success on the ABOG examination. Scores lower than this value may be used by the resident and program to identify areas in need of improvement as well as provide appropriate educational interventions.
References
- 1. Withiam-Leitch M, Olawaiye A. Resident performance on the in-training and board examinations for the ACGME outcome project. Teach Learn Med. 2008; 20 2: 136– 142. [DOI] [PubMed] [Google Scholar]
- 2. Spellacy WN, Carlan SJ, McCarthy JM, Tsibris JC. Prediction of ABOG written examination performance from the third-year CREOG in-training examination results. J Reprod Med. 2006; 51 8: 621– 622. [PubMed] [Google Scholar]
- 3. Brateanu A, Yu C, Kattan MW, Olender J, Nielsen C. A nomogram to predict the probability of passing the American Board of Internal Medicine Examination. Med Educ Online. 2012; 17: 18810. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Kim PY, Wallace DA, Allbritton DW, Altose MD. Predictors of success on the written Anesthesiology Board certification examination. Int J Med Educ. 2012; 3: 225– 235. [Google Scholar]