Millennial Resident Study Habits and Factors that Influence American Board of Anesthesiology In-Training Examination Performance: A Multi-Institutional Study

Abstract

Background:

In this study, we described Millennial anesthesiology residents' learning preferences and study habits and how they correlate with performance on the In-Training Exam (ITE).

Methods:

A confidential questionnaire including personal characteristics, previous examination performance, study habits, study material preferences, and perceived residency program support was emailed to 1047 anesthesiology residents from 30 ACGME-accredited residency programs across the United States.

Results:

Four hundred and twelve residents (39.4%) responded to the survey, and 240 of those respondents (58.3%) self-reported their 2017 ITE scores. The majority (95.9%) were Millennials. Respondents preferentially used online multiple-choice questions (92.3%) to prepare for the ITE, but many also used traditional anesthesiology textbooks (35.5%) and review books (46.7%). Respondents preferred independent study (94.6%) to group study (5.4%), and handwritten notes (69.4%) to taking notes on a laptop (26.8%) or tablet (3.8%). Less than half (47.5%) of respondents felt supported by their residency program in exam preparation, and 30.7% felt lack of support. Factors correlated with ITE scores on univariate analysis included prior USMLE 1 scores (p < .0000) and USMLE 2 scores (p < .0000), clinical anesthesia year (p < .0000), test anxiety score (p = .0004), prior failure of the basic exam (p = .0026), and prior failure of any board exam (p = .0124).

Conclusions:

Millennial learners have consistent performance on ITE exams regardless of personal characteristics, preferred study methods, or materials used. Prior exam performance is the most important predictor of future performance. Currently available study materials are meeting residents' needs and preferences, and while residency program offerings do not affect ITE performance, residents would like to feel more supported.

Introduction

The Millennial generation now predominates the current resident physician work-force.¹ Social science researchers have described a set of common life experiences with resulting values, attitudes, behaviors, and preferences that set the Millennials (born approximately 1980–2000) apart from prior generations.^2,3 Millennials were raised in an era of instant access, increasing global connectivity, and explosive growth in digital information technology. Many have never known a world without email, the Internet, and personal computing devices. They are often profiled as technologically savvy, group learners, goal and achievement oriented, with aptitude for self-directed learning. These differences are important: Studies across fields demonstrate that this generation absorbs information and learns differently than learners even 10 years ago.^4,5

At the same time, resident physicians need to master an ever-expanding body of knowledge to be successful in anesthesiology. They must do this while balancing the competing demands of providing full-time quality patient care and acquiring essential technical skills. To ensure mastery of knowledge, the American Board of Anesthesiology (ABA) uses a staged examination process consisting of 2 written multiple-choice exams (the BASIC and ADVANCED examinations) and a structured oral exam (the APPLIED examination). A candidate must pass all components to achieve ABA board certification.⁶ Board certification, in turn, is instrumental to future success. Patients and the general public view board certification as a marker for quality and excellence, and job prospects are better for board-certified physicians.⁷

During each year of residency training, the ABA also administers an In-Training Exam (ITE) to all physicians enrolled in anesthesiology residency training programs. This exam is designed to identify gaps in knowledge and evaluate residents' progress toward meeting educational objectives necessary to achieve board certification.⁸ Performance on the ITE has been established as an independent predictor of performance on the ABA and Royal College of Physicians and Surgeons of Canada (RCPSC) board certification examinations.^9,10

Despite the established importance of the ABA ITE and board examinations, there is little consensus or existing literature on optimal preparation strategies or whether unique characteristics of the Millennial generation influence exam performance. Based on research in other fields³ and anecdotal conversations with current residents, we hypothesized that traditional study materials and residency program curriculums may not be meeting the preferences and needs of Millennials. We therefore conducted a large-scale survey study to describe Millennial residents' learning preferences and study habits to determine how they correlate with performance on examinations.

Methods

This study was approved with exempt status by the UCSF (Protocol #17-21808) and UCLA (Protocol #17000431) Institutional Review Boards, and the need for written informed consent was waived. Investigators emailed all Accreditation Council for Graduate Medical Education (ACGME)–approved anesthesiology residency programs across the United States using the email addresses listed on the ACGME website. Permission to solicit residents was received from program directors of 30 programs. Residents of these programs were emailed invitations to complete a survey that consisted of 36 questions regarding personal characteristics, previous exam performance, study habits, study material preferences, and perceived residency program support. The Test Anxiety Inventory short form (TAI-5)—a validated self-report inventory designed to measure test-taking anxiety and differences in anxiety-proneness—was also included in the questionairre.¹¹

Eligible participants were current anesthesiology residents—clinical anesthesia (CA)-1, CA-2 and CA-3 years—at the time of the 2017 ABA ITE exam. Data were collected April 24, 2017 to June 25, 2017. The survey window was open for approximately 1 month for each participating resident, with weekly reminder emails sent to those who had not yet opened or completed the survey. No compensation was provided to respondents for survey completion.

Statistical Analysis

Data were collected and managed using Qualtrics survey software (Qualtrics, Provo, Utah). Statistical analysis of survey responses was performed using STATA 13.1 (StataCorp LLC, College Station, Texas). Descriptive statistics were used to analyze the responses to the survey items. The data were evidenced to be normally distributed, thus Pearson's chi-squared tests, independent samples t-tests, one-way ANOVA, and simple linear regression were used as appropriate to assess associations with ITE scores and subgroup comparisons. P < 0.05 was considered statistically significant.

Results

Email invitations with links to participate in the survey study were sent to 1047 residents at 30 institutions. Participating residency programs were diverse in terms of geography, program size, and clinical settings (Table 1). Overall response rate was 39.4% with 412 responses received. Of these resident responses, 240 (58.3%) also self-reported 2017 ABA ITE scaled scores.

Table 1.

Participating Residency Programs

Resident Characteristics

The mean (SD) age of the respondents was 31.7 (3.3) years, and 63.1% were male (Table 2). The majority of residents (95.9%) were Millennials, born between 1980 and 2000. Respondents were CA-1 (37.1%), CA-2 (35.0%), or CA-3 (27.9%) residents. Interns were excluded from our study. There was an even distribution of postgraduate years 2 through 4 (34.7%, 34.2%, and 27.9%, respectively), with 3.1% of respondents reporting postgraduate year 5 or above.

Table 2.

Respondent Characteristics

English was the first language of 83.0% of respondents, and 94.2% graduated from US medical schools. Several respondents reported having children at home (21.9%) while few had been pregnant in the past year (1.7%).

Past Exam Performance

Residents in our study self-reported a mean (SD) USMLE 1 score of 232.4 (21.2), and a mean (SD) USMLE 2 score of 241.6 (19.3). The mean (SD) self-reported 2017 ABA ITE scaled score was 36.6 (6.6). A small percentage of respondents reported failing the basic exam on their first attempt (5.6%), and 6.1% of respondents reported having failed any board exam in the past. Test-taking accommodations had been requested by 1.5% of respondents in the past.

Study Material Preferences

Residents strongly felt that practice questions best prepared them for the ABA ITE and board exams (Table 3). Electronically delivered practice multiple choice questions were used by 92.3% of residents, and paper forms of practice multiple choice questions were used by 34.8%. Other popular resources for studying were anesthesiology review books (46.7%), smart phone applications (38.3%), and major anesthesiology textbooks (35.5%).

Table 3.

Study Materials Used to Study for ABA ITE and Board Exams

Study Strategies

A large majority of respondents prefer independent study (94.6%) to group study (5.4%). Most residents (69.4%) prefer handwritten notes, while 26.8% prefer to take notes on their laptop and 3.8% on a tablet. When asked about when the majority of ITE preparation occurred, 58.5% of residents answered “in the 1 to 2 months before the exam,” 28.5% “spread evenly over the 6 months before the exam,” 7.3% “in the last 1 to 2 weeks,” and 5.7% “spread evenly over the 12 months before the exam.” Residents reported working mean (SD) 51.6 (22.2) hours per week leading up to the ITE, and 20.6% reported having vacation time or other time off during the month before the 2017 ITE.

Residency Program Support

According to 39.1% of residents, their residency programs offer a structured test preparation curriculum. The most common type of curriculum was independent study (32.5%), followed by attending led (26.2%), peer-led (17.3%), and mixed attending/peer led (24.1%). Residents reported that their residency program offered a paid subscription to exam preparation materials 56.8% of the time. Interestingly, not all residents from the same programs reported the same offerings.

Sixty-six percent of residents reported their residency curriculum included other standardized tests (ie, Anesthesia Knowledge Tests) separate from the ABA ITE or board exams.

Residency programs hosted a post-ITE exam debriefing session, according to 22.4% of residents, while 65.3% were aware of some kind of remediation program offered by their residency program for poor performance on exams.

When asked if they felt their residency program was adequately supporting their board exam preparation, 47.5% felt definitely or probably supported, and 30.7% felt definitely or probably not supported.

Test-Taking Anxiety

The TAI-5 has a possible score range of 0 to 5 with higher scores indicating higher levels of test-taking anxiety and anxiety-proneness.¹¹ Our study sample revealed overall low levels of test-taking anxiety with a mean score (SD) 0.95 (1.23).

Factors Associated with ABA-ITE Performance

Factors positively associated with self-reported 2017 ABA ITE scores on univariate analysis included prior USMLE 1 scores, prior USMLE 2 scores, and higher CA year (Table 4). Factors negatively correlated with ITE scores included higher test anxiety score, prior failure of the basic exam, and prior failure of any board exam. Multiple variables without a significant association with ITE scores are detailed in Table 4.

Table 4.

Factors Associated with ABA-ITE Performance

Subgroup Comparisons

There were few significant differences between respondents who reported their 2017 ABA ITE scores versus those who did not. Male resident respondents were more likely to report their scores than females (p = 0.017) and graduates from foreign medical schools were less likely to report scores (p = 0.034). There were no significant differences in age, CA year, having children at home or speaking English as a first language. Those who did not report ABA ITE scores did report longer study hours: mean (SD) 12.4 (8.6) hours per week versus 10.3 (7.8) hours per week, p = 0.0417. There was no significant difference in failure of prior board examinations, test-taking accommodations, independent versus group study preference, or preferred study resources.

Analysis of respondents who failed the anesthesia basic exam or another prior board examination (7.3%) versus those who did not report a prior examination failure revealed no significant differences by age, gender, CA year, graduation from a foreign versus United States medical school, having children at home, or speaking English as a first language. There were also no significant differences in independent versus group study preference, note-taking style, or weeks of preparation for the ABA ITE examination. There was a difference in self-reported study hours with respondents reporting a prior examination failure studying more: mean (SD) 14.7 (10.0) hours per week versus 10.6 (7.8) hours per week, p = 0.0200.

Finally, residents reporting lower USMLE board examination scores (below more than 1 standard deviation from the mean on step 1 or step 2) versus higher USMLE examination scores (above more than 1 standard deviation from the mean on step 1 or step 2) have no significant difference by gender, CA year, graduation from a foreign versus United States medical school, having children at home, or speaking English as a first language. The lower scoring group was slightly older: mean (SD) 31.7 (3.8) years old versus 30.5 (1.8) years old, p = 0.0130. The lower scoring group also reported spending more hours per week studying for the ABA ITE: mean (SD) 12.4 (7.6) hours per week versus 9.0 (7.4) hours per week, p = 0.0095. There were no significant differences in independent versus group study preference, note-taking style, or weeks of preparation for the ABA ITE examination.

Discussion

This study represents, to our knowledge, the first nationally representative cohort of predominantly Millennial generation anesthesiology residents reporting on current study habits, study preferences, and perceptions of residency program examination preparation curriculums. This is also, to our knowledge, the first report on which of these factors correlate with improved ABA ITE performance. There are many important takeaways from our results.

Current anesthesiology residents appreciate a mix of newer digital and traditional hardcopy study material. In our sample, electronically delivered practice multiple-choice questions were the most popular form of studying with over 92% utilization. Other popular study materials included smart phone applications but also major anesthesiology textbooks and review books. The increased use of electronic and mobile devices to retrieve and study information reflects well-documented trends and characteristics of Millennial learners.^3–5 There is also a trend for current residents to focus more on test preparation resources over traditional textbooks and primary literature. This is not completely surprising: Millennials have been described as more accustomed to frequent testing and focused on scoring high on examinations.¹³

Importantly, there was no correlation between residents' preferred study materials and ITE scores. This suggests that residents can devote study time to the material that best suits their own determined study needs and preference. Whether the study material was used by the majority of residents (such as online questions banks) versus only a few residents (such as in-person board review courses), the results of dedicating time to studying were equal. In addition, although technology is important and provides rapid and easily accessible information, faculty need not focus on providing new digital information alone. We should be flexible and support residents with a wide variety of material they can use to support their education.

Although many of the group stereotypes for Millennials apply to our current resident classes, they also defy the stereotypes in many instances. Though often characterized as team-oriented and socially networked group learners, Millennial residents in our study strongly prefer independent study over group study. Handwritten notes are also still preferred to electronic note taking.¹

Residents in our study reported good time management. Only 7.3% waited until the last 2 weeks to study for the 2017 ITE. This is despite the majority of residents reporting long work hours and lack of time off to prepare in the month before the exam. This behavior is consistent with descriptions from previous higher education scholars describing Millennial students as goal-directed, achievement-oriented, proficient in structuring their time, working from schedules, and following rules.^12,13 Undoubtedly, the day-to-day learning that occurs while residents are on clinical rotations is also an important source of knowledge, however this specific factor is difficult to assess as a defined variable.

Personal characteristics of residents do not predict success on the ABA ITE. Resident performance is not affected by gender, age, or stressors such as pregnancy, minors at home, or English as a second language. This is contrary to a prior study at a single anesthesiology residency program that found female residents in their program performed worse than males on the ITE and written board examinations.⁹ Prior military experience or graduation from a non-US medical school also did not correlate with ITE performance.

Resident performance on ITE exams is not affected by residency program offerings. Mandatory Anesthesia Knowledge Tests, exam debriefing sessions, and known availability of remediation do not correlate with improved ITE scores. Regardless, the majority of residents reported not feeling supported in exam preparation by their program. Further, even though test-taking anxiety was an issue for a small minority of learners in this study, it is correlated with worse exam performance and it would be worthwhile to identify interventions for these residents.

Not surprisingly, if residents have done well on standardized exams in the past, they have a higher rate of future success. Of all the factors we examined, only prior standardized exam performance, level of training in Anesthesiology, and test-taking anxiety scores correlate with ITE performance.

Limitations

There are limitations to our study. The potential for recall bias is present with any study involving self-reported information, and self-reported scores may not be completely reliable. However, by administering the survey immediately after residents received their 2017 ABA ITE scores, we hoped to minimize the effects of recall bias and reporting errors. Subgroup comparisons also reveal few significant differences between respondents who reported their 2017 ABA ITE scores versus those who did not. Additionally, survey studies are susceptible to participation bias, and we do not know if data from nonresponders would change our findings. However, we are somewhat reassured that mean the USMLE 1 score (232.4) and USMLE 2 score (241.6) reported in our study line up with National Resident Matching Program data of mean scores of candidates matching into Anesthesiology residencies nationwide (mean USMLE1 score 232, USMLE 2 score 242).¹⁴ Respondents with poor exam scores may also—either consciously or unconsciously—report study habits, level of support, and anxiety levels differently. Finally, while the online survey had the potential to increase ease of response within our target population, results may be skewed toward a more digitally savvy sample if some members of the target population did not participate because of their discomfort with online survey tools.

Strengths

Strengths of our study include the large sample size and diversity of participating residency programs, increasing generalizability of our results.

Conclusion

The importance of performing well on ITE and board exams has been well documented in anesthesiology and a variety of other medical specialties. Our research reveals that Millennial learners have consistent performance on ITE exams regardless of personal characteristics, preferred study methods, or materials used. The unique characteristics and preferences of Millennial generation residents should not be seen as challenges or weaknesses. We conclude that the variety of current study materials available are meeting current resident needs and preferences, and while residency program offerings do not affect ITE performance, many residents would like to feel more supported.