Abstract
Background:
In-training examinations (ITEs) are commonly used by residency programs to measure competency in their respective fields. It has been demonstrated that success on the ITE is correlated to First Time Pass Rate (FTPR) on the boards. Therefore, it is important to motivate residents to perform well on these exams. Previous studies indicate positive incentivization may contribute to improvement on examinations. The objective of our study was to determine whether introduction of a positive incentive could improve resident performance on the ITE and/or FTPR on the advanced certifying exam.
Methods:
A positive incentive was introduced in 2017 (certificate of commendation, curriculum vitae honor, public recognition, and $500 in their books/travel allowance) to residents who achieved the target score on the ITE (80th percentile). A survey was then provided to these residents to determine which incentives contributed most to their motivation.
Results:
Before the incentivization, 21 (15.1%) of the previous 149 senior residents reached the target score on the annual ITE. After incentivization, this improved to 28 (30.9%) of 81 (P = .0056). The FTPR on the advanced certifying exam was 90% before incentivization and 97.6% after (P = .14). The survey found that the primary motivators were extra funding, honor on their curriculum vitae, and public recognition.
Conclusions:
We found that our residents had significant improvements on the annual ITE after the introduction of positive incentives. This incentivization may be easily implemented by program directors in their respective medical residencies to increase examination performance.
Keywords: Incentivization, residency education, in-training examination, certifying exam, motivation in residency, anesthesiology residency
Introduction
Examination performance remains fundamental to the certification process of every field in medicine. Many institutions have adopted strategies to optimize performance on evaluations.1 It has been demonstrated that extrinsic motivation improves quantitative performance on examinations.2 Therefore, incentivization as extrinsic motivation may be a promising educational strategy to impact exam scores. Preliminary studies indicate incentive-based approaches may significantly improve certifying exam scores and clinical outcomes in both ophthalmology and internal medicine residencies.3,4
Our anesthesiology residency program chose to pursue a deliberate incentive program in 2017 through rewarding performance on the annual in-training examination (ITE). In prior years, residents were required to achieve a threshold minimum score of the 30th percentile on the ITE to qualify for participation in moonlighting activities, similar to the study performed by Joseph et al.5 However, we speculate that this approach only had a modest impact on the First Time Pass Rate (FTPR) because the qualifying scores to moonlight were too low to correlate with passing the advanced certifying exam, and many of the residents did not wish to moonlight. Thus, we chose to pursue the development of a positive incentive that might help motivate all of our residents.
We announced the program with considerable fanfare: those who scored at the 80th percentile or greater on the ITE were to be publicly recognized and receive a certificate of commendation. The commendation was named after an alumnus who had achieved a perfect score on the ITE many years before. Furthermore, if residents met the target score, we committed to increase their books and travel allowance. The benchmark target score chosen to be the 80th percentile correlates with an FTPR of 98%.6
The goal of our study was to determine whether this positive incentive improved resident performance on the ITE.
Materials and Methods
Given our institution policy, an institutional review board exemption was granted for this study. Between 2012 and 2019, our categorical residency program consisted of on average 54 residents (~14 in each class). Anesthesiology is a 4-year residency with 3 clinical training years following a medical, surgical, or transitional internship. We defined a senior resident as being a PGY-3 or -4, a junior resident as PGY-2, and an intern as PGY-1. There were 220 senior residents and 111 PGY-4s included in the study.
All residents take the ITE in February. The ITE consists of a standardized examination composed of 200 multiple-choice questions in 4 hours and is conducted by the American Board of Anesthesiologists.7 We set the target score for the ITE near the 80th percentile because our residents demonstrated the ability to achieve this score, and it correlated with 98% passage on the advanced certifying exam.6 The advanced certifying exam is taken on completion of the residency program. We defined the FTPR on the advanced certifying exam as the percentage of PGY-4 residents who passed the advanced certifying exam on their first attempt after graduating.
We introduced the positive incentives in 2017. These consisted of a certificate of commendation, an honor to be added to the resident’s curriculum vitae (CV), public recognition, and $500 added to their books and travel allowance. These were awarded to residents who reached the target score on the ITE. These could be awarded year over year if the resident is able to hit the target score more than once.
We compared the percentage of senior residents who reached the target score on the ITE in the years of 2012 to 2016 (before incentivization) to that of residents in 2017 to 2019 (after incentivization) using a χ2 test for proportions. Similarly, we compared the PGY-4 FTPR on the advanced certifying exam for the years before incentivization to that of PGY-4s in the years after incentivization using a χ2 test for proportions. We compared the median Step 1 and 2 US Medical Licensing Examination (USMLE) scores of the residents before incentivization with those after the incentivization was introduced using a Mood median test.
In addition, a survey of the current senior residents who received the commendation in 2019 was performed to determine which of the incentives contributed most to driving their motivation (Appendix A). For each incentive, we asked the resident to rate its contribution on a scale of none at all, a little, a moderate amount, and a great deal.
Results
USMLE Step Scores
From 2012 to 2016, the median step 1 score was 225, whereas that from 2017 to 2019 was 233.5 (P = .842). From 2012 to 2016, the median step 2 score was 242.5, whereas that from 2017 to 2019 was 246 (P = .889).
Target Score on ITE
From 2012 to 2016, 21 of 149 (15.1%, 95% confidence interval [CI] 8.9%–20.7%) senior residents reached the target score on the annual ITE. After incentivization, from 2017 to 2019, 28 of 81 (30.9%, 95% CI 25.3%–46.0%) senior residents reached the target score (P = .0056; Figure 1). See Table 1 for a breakdown in scores for the 8-year period.
Figure 1.
Percentage of all senior residents in the years before incentivization (2012–2016) and those in the years after (2017–2019) who reached the target score on the in-training examination (ITE).
Table 1.
Number and Percentage of 220 Senior Residents Hitting the Target Score on In-training Exam from 2012–2019
| Year | No. of Senior Residents Reaching Target Score | Total Taking Examination | Percentage |
|---|---|---|---|
| 2012 | 7 | 28 | 25.0 |
| 2013 | 3 | 29 | 10.3 |
| 2014 | 4 | 27 | 14.8 |
| 2015 | 4 | 28 | 14.3 |
| 2016 | 3 | 27 | 11.1 |
| 2017a | 6 | 25 | 24.0 |
| 2018 | 9 | 28 | 32.1 |
| 2019 | 13 | 28 | 46.4 |
a Incentives introduced in 2017.
FTPR on Advanced Certifying Exam
From 2012 to 2016, the FTPR on the advanced certifying exam was 90% (95% CI 80.5%–95.9%). After introduction of incentivization, the FTPR on the advanced certifying exam was 97.6% (95% CI 87.1%–99.9%; P = .14).
Survey of Motivators
All 9 of the current senior residents who reached the target score on the ITE in 2019 responded to our survey (Figure 2). The certificate of commendation was shown to contribute none at all by 44.4% of residents, a little by 22.2%, a moderate amount by 32.3%, and a great deal by 0.0%. The honor added to the CV was shown to contribute none at all by 0.0% of residents, a little by 33.3%, a moderate amount by 55.6%, and a great deal by 11.1%. Public recognition was shown to contribute none at all by 11.1% of residents, a little by 11.1%, a moderate amount by 66.7%, and a great deal by 11.1%. The extra funding was shown to contribute none at all by 0.0% of residents, a little by 22.2%, a moderate amount by 33.3%, and a great deal by 44.4%. Moonlighting was shown to contribute none at all by 55.6% of residents, a little by 11.1%, a moderate amount by 22.2%, and a great deal by 11.1%.
Figure 2.
Survey results from current senior residents who reached the target score on the in-training examination in 2019 indicating the contribution that each incentive made in increasing their motivation. Abbreviation: CV, curriculum vitae.
Discussion
We found that after implementation of positive incentives, our residents significantly increased their performance on the annual ITE. The number of senior residents who reached the target score was increased by twofold after this initiative. Thirteen senior residents won the commendation this year, along with the first junior resident to ever hit this mark. This further demonstrates the motivation the incentive has instilled in our residents early on in their training. Furthermore, there was a resident who earned the first perfect score in our department since the commendation’s namesake 19 years prior. The FTPR had dropped below the national average for our specialty (90.9%) in 3 of the 5 years before the incentive. Since implementation of this incentive, however, only 1 graduating resident has failed to pass the advanced certifying exam on their first attempt (97.6%). We did not see a significant improvement in the FTPR on the written advanced certifying exam; however, many studies have shown a correlation between ITE performance and board passage.8–14
According to our survey, the increase in motivation to perform well on the ITE was attributed primarily to the monetary benefit. In addition, the honor on their CV and public recognition were the next highest motivators, and several residents specifically commented on how the honor might enhance their fellowship applications.
Other studies have examined incentives as motivators. Jenkins et al15 found a positive correlation between the introduction of financial incentives and performance. Garbers and Konradt16 also found a positive correlation between incentives and performance, but cited the importance of tailoring the reward to its benefactor. We similarly found that motivators should be targeted based on resident desires, because our residents specifically valued extrinsic incentives (monetary reward, CV honor, and public recognition). A survey soliciting the driving factors of the specific residents in a program might help tailor an incentive program and lead to more success.16 Alfandi and Alkahsawneh17 also demonstrated that a mixture of concrete and moral incentives optimized satisfaction and motivation.
Some studies have shown detrimental effects on intrinsic motivation when an extrinsic incentive was introduced.15 It was believed that these incentives brought about too narrow of a focus when performing the task and too little time spent on the task.15 Jenkins et al,15 however, demonstrated that on the addition of a financial incentive there was no difference in its effect on performance when the task was performed intrinsically versus extrinsically. In addition, Xiong et al18 determined that extrinsic motivators led to the most student engagement compared with intrinsic and social motivators, which correlated to higher overall knowledge retention.
In a meta-analysis performed by Balliet et al,19 the difference between rewards and punishments shaping performance was found to be nonexistent. However, Wächter et al20 found that incentivization may have a greater impact on learning and retention than punishments. They showed that punishment leads to greater short-term behavior changes, whereas rewards bring about better long-term learning improvements.20 Although both are effective in increasing performance, we thought a greater percentage of our residents would respond to positive incentivization due to the motivational and supportive aspect of learning in our educational setting.
Some limitations of our study include the lack of generalizability because it was performed at only one institution, and our survey was only to the current residents who hit the target score last year. There were also improvements made in our marketing during this time period, including improvements in the residency interview experience, which may have led to enhanced residency recruitment. Although these changes were made, the median step scores were not significantly different between residents recruited before and after the incentivization program.
In conclusion, this study demonstrated the value of incentives in resident education for improving examination performance. We believe incentivization may be generalized to all levels of education in a variety of ways: recognition, monetary compensation, or other appropriate benefits. Program directors could easily apply some of these incentivization strategies to improve performance on their exams.
Appendix
Appendix A. Survey
In-training Examination (ITE) Success Drivers
Thank you for taking this survey. The purpose of this survey is to investigate what impact established rewards may have had (positive or indifferent) on helping to promote your successful performance on the in-training exam.
For each of the questions below, please rank the impact that each of these “rewards” had on influencing you to put forth the effort that resulted in your high performance on the ITE.
-
Departmental and/or public recognition of high performance on ITE:
A great deal
A moderate amount
A little
None at all
-
Ability to claim the honor on your CV:
A great deal
A moderate amount
A little
None at all
-
Receiving a frameable certificate acknowledging your achievement and honor:
A great deal
A moderate amount
A little
None at all
-
Extra funding deposited in books and travel spending account:
A great deal
A moderate amount
A little
None at all
-
Achieving a score high enough to be able to participate in moonlighting activities:
A great deal
A moderate amount
A little
None at all
-
Any other comments on what drove you to put forth the effort that resulted in your high performance?
__________________________________________________________
__________________________________________________________
__________________________________________________________
References
- 1.Bowen JL. Educational strategies to promote clinical diagnostic reasoning. N Engl J Med. 2006;355(21):2217–25. doi: 10.1056/NEJMra054782. [DOI] [PubMed] [Google Scholar]
- 2.Cerasoli CP, Nicklin JM, Ford MT. Intrinsic motivation and extrinsic incentives jointly predict performance: a 40-year meta-analysis. Psychol Bull. 2014;140(4):980–1008. doi: 10.1037/a0035661. [DOI] [PubMed] [Google Scholar]
- 3.Zafar S, Wang X, Srikumaran D, et al. Resident and program characteristics that impact performance on the ophthalmic knowledge assessment program (OKAP) BMC Med Educ. 2019;19(1):190. doi: 10.1186/s12909-019-1637-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Bolyard J, Viswanathan V, Fribourg D, Narayanan R. MIPS in residency? A look at merit-based incentives in an internal medicine residency outpatient practice. J Grad Med Educ. 2019;11(1):79–84. doi: 10.4300/JGME-D-18-00239.2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Joseph JA, Terry CM, Waller EJ, et al. Enhancement of anesthesiology in-training exam performance with institution of an academic improvement policy. J Educ Perioper Med. 2014;16(6):E072. [PMC free article] [PubMed] [Google Scholar]
- 6.Warner DO. 2017 American Board of Anesthesiologists exams report. http://www.theaba.org/PDFs/Presentations/SAAA-2017-Exams-Report Accessed March 30, 2020.
- 7.Philip J, Whitten CW, Johnston WE. Independent study and performance on the anesthesiology in-training examination. J Clin Anesth. 2006;18(6):471–3. doi: 10.1016/j.jclinane.2006.01.003. [DOI] [PubMed] [Google Scholar]
- 8.McClintock JC, Gravlee GP. Predicting success on the certification examinations of the American Board of Anesthesiology. Anesthesiology. 2010;112(1):212–9. doi: 10.1097/ALN.0b013e3181c62e2f. [DOI] [PubMed] [Google Scholar]
- 9.Kim PY, Wallace DA, Allbritton DW, Altose MD. Predictors of success on the written anesthesiology board certification examination. Int J Med Educ. 2012;3:225–35. [Google Scholar]
- 10.Kearney RA, Sullivan P, Skakun E. Performance on ABA-ASA in-training examination predicts success for RCPSC certification. American Board of Anesthesiology-American Society of Anesthesiologists. Royal College of Physicians and Surgeons of Canada. Can J Anaesth. 2000;47(9):914–8. doi: 10.1007/BF03019676. [DOI] [PubMed] [Google Scholar]
- 11.Waxman H, Braunstein G, Dantzker D, et al. Performance on the internal medicine second-year residency in-training examination predicts the outcome of the ABIM certifying examination. J Gen Intern Med. 1994;9(12):692–4. doi: 10.1007/BF02599012. [DOI] [PubMed] [Google Scholar]
- 12.Klein GR, Austin MS, Randolph S, Sharkey PF, Hilibrand AS. Passing the Boards: can USMLE and orthopaedic in-training examination scores predict passage of the ABOS Part-I examination? J Bone Joint Surg Am. 2004;86(5):1092–5. [PubMed] [Google Scholar]
- 13.Grossman RS, Fincher RM, Layne RD, Seelig CB, Berkowitz LR, Levine MA. Validity of the in-training examination for predicting American Board of Internal Medicine certifying examination scores. J Gen Intern Med. 1992;7(1):63–7. doi: 10.1007/BF02599105. [DOI] [PubMed] [Google Scholar]
- 14.Ellis E III, Haug RH. A comparison of performance on the OMSITE and ABOMS written qualifying examination. Oral and Maxillofacial Surgery In-Training Examination. American Board of Oral and Maxillofacial Surgery. J Oral Maxillofac Surg. 2000;58(12):1401–6. doi: 10.1053/joms.2000.18275. [DOI] [PubMed] [Google Scholar]
- 15.Jenkins GD, Jr, Mitra A, Gupta N, Shaw JD. Are financial incentives related to performance? A meta-analytic review of empirical research. J Appl Psychol. 1998;83(5):777–87. [Google Scholar]
- 16.Garbers Y, Konradt U. The effect of financial incentives on performance: a quantitative review of individual and team-based financial incentives. J Occup Organ Psychol. 2014;87(1):102–37. [Google Scholar]
- 17.Alfandi AM, Alkahsawneh MS. The role of the incentives and reward system in enhancing employee’s performance. “A case of Jordanian travel and tourism institutions.”. IJ-ARBSS. 2014;4(4):326–41. [Google Scholar]
- 18.Xiong Y, Li H, Kornhaber ML, Suen HK, Pursel B, Goins DD. Examining the relations among student motivation, engagement, and retention in a MOOC: a structural equation modeling approach. Global Educ Rev. 2015;2(3):23–33. [Google Scholar]
- 19.Balliet D, Mulder LB, Van Lange PA. Reward, punishment, and cooperation: a meta-analysis. Psychol Bull. 2011;137(4):594–615. doi: 10.1037/a0023489. [DOI] [PubMed] [Google Scholar]
- 20.Wächter T, Lungu OV, Liu T, Willingham DT, Ashe J. Differential effect of reward and punishment on procedural learning. J Neurosci. 2009;29(2):436–43. doi: 10.1523/JNEUROSCI.4132-08.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]