Emergency medicine residents’ attitudes and opinions of in-training exam preparation

Abstract

Purpose

Emergency Medicine (EM) residents take the American Board of Emergency Medicine (ABEM) in-training exam, and performance on this exam has been shown to correlate to performance on the ABEM qualifying exam. Though many residencies have in-training exam preparation activities, there is little data on the effectiveness of these efforts. This study aimed to elicit resident perspectives about the exam and exam preparation in order to generate hypotheses and better inform future preparation efforts.

Methods

Second- and third-year EM residents at a single institution were interviewed using a semi-structured format. Qualitative methodology was used to analyze the data.

Results

Thirteen EM residents participated in the study. Eight major themes and 18 sub-themes were identified. These were further characterized as relating to the exam itself or to exam preparation. Residents generally value the in-training exam. Sixty-nine percent noted that it provided an assessment of their current knowledge and deficiencies. Thirty-eight percent noted that it improved familiarity with the qualifying exam. Regarding exam preparation, residents stated that a question format was preferred, especially when accompanying explanations were of high quality. Additionally, practical considerations, such as portability, impacted resident selection of study tools.

Conclusion

Residents value the in-training exam as a marker of their academic progress and for their ability to gain familiarity with the qualifying exam. They prefer question-based preparation over text-based learning, as long as there is a detailed explanation of each answer. Educators creating structured in-training review may want to focus on question-based material with detailed explanations.

Introduction

Emergency medicine (EM) residents take the American Board of Emergency Medicine (ABEM) in-training exam annually. Performance on this test has been shown to correlate with performance in the ABEM qualifying exam, which is required for board certification.¹ Optimizing performance on the in-training exam may therefore be of value to resident physicians and program directors alike.

The best method of preparation for in-training exams is currently unclear, and programs to aid in preparation have reported mixed results.^2–6 Cheng found that an EM residency-developed review course did not affect performance on the in-training exam.² Gillen also failed to demonstrate improved exam performance after implementation of a program that utilized structured reading and graded quizzes.³ However, there is some evidence from other specialties that preparation can in fact improve scores on in-training exams.^4–6

To better direct future attempts at exam preparation, it is worth considering the views of the trainees. Residents’ opinions regarding the exam itself and strategies for preparation are unknown. The aim of this investigation is to characterize resident perspectives on the exam, with the hope that information can be obtained that may aid in the development of more effective review methods.

Material and methods

Research design

Semi-structured interviews of senior EM residents at one institution were performed. Data analysis used qualitative methodology in order to allow generation of hypotheses. The study was reviewed by The Office of Responsible Research Practices (ORRP) at The Ohio State University (OSU) and it was deemed exempt from Institutional Review Board review.

Study setting and population

Study participants were second- and third-year EM resident physicians at OSU. Residents at this program are required to take the exam annually. The study took place between March 2012 and May 2012. These groups of residents, and time of year, were selected to ensure that all participants had taken the exam at least two times. Residency education at OSU follows the requirements of the Accreditation Council for Graduate Medical Education.

Twenty-four residents (12 per year) were asked to participate by an email request for an interview. Participation was voluntary, and this was made clear in the email, as required by ORRP. Prior to beginning each interview, written consent to participate was obtained, and further assurances of confidentiality were provided.

Study protocol and data analysis

An interview guide was created by the investigators in advance of this study. The first version was reviewed and revised, based on feedback from other staff of the EM department with experience in residency education. The final version consisted of five questions that were intended to stimulate conversation (Table 1). Interviewers were encouraged to ask additional follow-up questions to maximize response and to promote discussion. One-on-one interviews with residents were conducted. These interviews were performed by one of two investigators, based on schedule compatibility with the resident. Interviewers recorded notes during and immediately after the conversation. On a separate page, descriptive data were also recorded, including resident’s year in training, and sex.

Table 1.

Guide used in interviews

1. How do you feel about the in-training exam?

2. Discuss the importance of the in-training exam and preparing for it.

3. What methods have you used in the past to prepare for the exam? Why?

4. What is important to you when choosing a method for exam preparation?

5. What would be your ideal method of preparation for the in-training exam?

Interview notes, and separate descriptive data, were stored securely until all interviews were completed, at which point they were analyzed using basic descriptive statistics and qualitative methodology.^7,8 The investigators independently reviewed the interview notes (both the notes they took and the notes the other investigator took). As themes and sub-themes emerged during reading, they were recorded. After completion of thematic analyses, the two investigators met to compare results. Differences were discussed and a final consensus thematic analysis was agreed upon. The investigators then went back to review all the notes to determine the frequency of themes and sub-themes collaboratively. If the investigators agreed that a single response directly related to more than one theme, it was coded as such. In some instances, responses clearly fit within a major theme, but not a sub-theme, and were simply counted as relating to the major theme.

Results

Thirteen residents agreed to participate in this study. Six of the participants were third-year residents (two female, four male) and seven were second-year residents (four female, three male). The mean USMLE Step 1 score for the thirteen participants was 220. The mean USMLE Step 2 CK score was 235. One investigator (TRE) conducted six interviews and seven were conducted by the other (AWB).

A total of eight major themes were noted. Within these major themes, eighteen sub-themes were noted. Major themes and sub-themes, as well as the frequencies of each, are shown in Tables 2 and 3. The major themes could be further characterized within domains of attitudes regarding the exam, and opinions regarding preparation. We further describe the themes and sub-themes below.

Table 2.

Thematic content of attitudes and opinions regarding the in-training exam

Major theme	Sub-theme	Frequency
Value examination		21
	Motivation to study	3
	Provide assessment of current knowledge/deficiencies	9
	Monitor progression over training years	3
	Compare results to classmates/peers	2
	Improved familiarity with ABEM exam	5
Weaknesses of exam		2
	Score reporting not immediate	1
	No ability to review questions	1

Abbreviation: ABEM, American Board of Emergency Medicine.

Table 3.

Thematic content of opinions regarding in-training exam preparation

Major theme	Sub-theme	Frequency
Question format optimal		20
	Similar to ABEM format	4
	Patient centered/clinically relevant	2
	Answers provide enough detail	5
	Provides assessment of strengths/weaknesses	1
	Text-based learning less valuable	4
Practical aspects important		11
	Portable	3
	User-friendly	3
	Affordable	1
	Ability to set pace, small increments, affect method selection	4
Preparation timing		8
	Closer to ABEM exam more preparation	4
	Study weeks/months before exam	4
Preparation should be relevant		7
	To exam performance	5
	To clinical practice	2
Peer/fellow resident recommendations highly regarded	No sub-themes	6
Longitudinal study ideal	No sub-themes	2

Abbreviation: ABEM, American Board of Emergency Medicine.

Resident attitudes about the in-training exam

The two major themes in this category were “value examination” and “weaknesses of exam”. There were 21 responses in the value examination theme and two responses in the weaknesses of exam theme. Within the theme value examination, five sub-themes emerged, the most common of which was “provides assessment of current knowledge and deficiencies”.

Value examination

The most commonly coded theme was that residents value the in-training examination. Within this major theme, five sub-themes were identified. Nine of the thirteen participants (69%) noted that the exam “provides an assessment of their current knowledge and deficiencies”. Thirty-eight percent mentioned an “improved familiarity with the ABEM qualifying exam” (the “real” exam) as a reason for valuing the in-training exam. Other sub-themes included “motivation to study”, “monitoring progression over the course of training”, and “the ability to compare him/herself to peers”.

Weaknesses of examination

Mention of in-training exam weaknesses was noted in two interviews. One participant noted that “score reporting was not immediate” and another noted the “lack of ability to review the questions” after the exam.

Resident opinions regarding in-training exam preparation

In this category, six major themes emerged: “question format is optimal”, “importance of practical aspects”, “preparation timing”, “relevance of preparation”, “fellow resident recommendations are highly regarded”, and “longitudinal study is ideal.” Of the major themes, “question format is optimal” (five sub-themes) was most common, with twenty such responses. The next-most common theme was “importance of “practical aspects” (four sub-themes), followed by “preparation timing” (two sub-themes).

Question format optimal

Regarding resident opinions about in-training exam preparation, the most common major theme was that residents found studying materials in question form to be ideal. Five participants noted that this was true especially when the study materials included explanations with detail sufficient to enhance learning. Four participants preferred a question format, similar to the format to the in-training and qualifying exams. Four residents specifically noted that text-based learning was less valuable. Other sub-themes included that question-based studying was patient-centered or clinically relevant (two residents), and that this format provided a method of assessing strengths and weaknesses (one resident).

Practical aspects important

The second most common theme regarding exam preparation was that there are practical issues that residents consider when choosing a study method. Among these practical concerns, the ability to set their own pace and to study in small increments was noted by four participants. The portability and user-friendliness of materials were each noted by three residents as being important factors in choosing a method of preparation. One resident mentioned that the affordability of the study tool was a consideration.

Preparation timing

The timing of in-training exam preparation emerged as a major theme and was noted eight times. Within this major theme, two sub-themes were mentioned. Four residents noted that they devoted more time to in-training exam preparation when they were closer to taking the ABEM qualifying exam (ie, they studied more as senior residents than they did as interns). Four residents also identified that they focused on in-training exam preparation in the weeks and months prior to the exam, rather than throughout the year.

Preparation should be relevant

Five residents noted the importance of study practices that are relevant to performance in the in-training exam. Additionally, two residents mentioned the need to have study methods that are relevant to clinical practice.

Fellow resident recommendations are highly regarded

Six residents stated that they used the recommendations of other residents when deciding how they would prepare for the in-training exam.

Discussion

The aim of this investigation was to characterize EM resident attitudes regarding the in-training exam and opinions regarding optimal preparation strategies. The hope was to uncover information that could be used to develop more effective test preparation programs. The two most notable findings of this investigation are that residents generally perceive value in the annual examination and that they prefer question-based preparation.

Not surprisingly, residents cited improved familiarity with the ABEM qualifying exam as a benefit of the in-training exam. However, this was not the most common sub-theme noted. The majority of residents mentioned using the results of the in-training exam as an assessment of their current knowledge and deficiencies. (It should be noted that exam results are accompanied by a detailed breakdown of performance in multiple topics.) Additionally, participants noted that the in-training exam allows them to monitor their progress as they advance through residency. This indicates that residents, undoubtedly motivated to pass the qualifying exam, place high value on this objective measure of their professional growth. Presumably, this is related to a desire to improve, which is also highlighted in other major themes and sub-themes. One resident noted that a question-based study method was preferred because it provided an assessment of strengths and weaknesses. Additionally, one person mentioned the inability to review questions after the exam as a weakness of the in-training exam.

Regarding preparation for the exam, residents overwhelmingly indicated that they preferred a question-based format. This may be somewhat intuitive, as it mirrors the qualifying exam itself, and four residents mentioned this as a reason for choosing this manner of study. However, it seems that this is only true if the study material provides explanations that are detailed enough to enable learning, indicating that not all question-based materials are necessarily valued. It is noteworthy that almost half of the residents in this study used the recommendations of other residents in selecting their method of preparation. While it is possible that this contaminated our results somewhat, we believe that it is more likely that the promotion of question-based study aids has trickled down from graduating residents over several years. This may relate to the work of Bull et al, who found that resident-led review sessions were superior to faculty-led sessions.⁴

Interestingly, four residents specifically mentioned that text-based study was less valuable as it pertains to exam preparation. Previous work by Gillen showed that a structured reading program had no effect on exam scores across all classes in an EM residency, though it did demonstrate improvement in postgraduate year 1 (PGY-1) scores, compared to historical controls.³ However, this study included a question-based component with faculty-led explanations. This finding may be consistent also with prior work that showed no improvement in in-training exam scores following a didactic review course.² Accordingly, attempts by residency administration to develop structured in-training review perhaps should be question-focused. This would allow not only for the review of relevant material, but also for discussion of test taking strategies that may improve exam performance.

The timing of exam preparation was identified as being an important factor in exam preparation, and was specifically mentioned by several residents. Not only does it appear that residents study more as they near the date of the exam, but also they seem to view the in-training exam more seriously as they near the qualifying exam. This may indicate that, as residents progress through the training program, they become more aware of the importance of the qualifying exam, and, as noted above, of the need to use their in-training exam results as a predictor of their future performance on the qualifying exam. Lastly, though four residents noted preparing in the weeks leading up to the in-training exam, two cited longitudinal study as being ideal. Due to the design of the study, it is unclear if this is simply a difference in approach between residents, or the difference between “ideal” and “actual” preparation.

Born from 1981 onwards, the millenial generation should now make up a large portion of medical trainees. Though we did not collect this data, it is likely that a majority of our sample was composed of members of this generation. Millenials are widely accepted as multi-taskers who are comfortable with technology, so it is no surprise that portability and user-friendliness of materials, and the ability to set their own pace were all mentioned with similar frequency by participants. More than one resident preferred to use electronic, question-based materials for exam preparation. For example, recently presented work utilizes the Twitter social media format as a method for disseminating board-review questions to residents, although no outcome data is available.⁹ The preference for electronic format will likely become more prevalent as the millennial generation enters residency training en masse, and as more electronic study aids become available.

While we identified several themes related to resident attitudes and opinions regarding exam preparation, we do not have data on the effectiveness of the preferred study methods. Some prior work in other specialties has shown that directed study can influence in-training exam performance and board pass rates.^4–6 To our knowledge, no previously published studies in EM have shown any specific exam preparation technique to be superior.

Limitations

As this was a single-center study, it is possible that resident attitudes and opinions simply reflect the local culture, and are not generalizable. This is especially relevant, since residents cited recommendations from other residents as a key component to selecting a method of study. Additionally, some residency curricula emphasize in-training exam preparation more than others; the degree to which our program emphasizes this may limit the generalizability of our findings.

We utilized a semi-structured interview with pre-determined questions intended to stimulate conversation with subjects. To ensure completeness, EM faculty staff vetted these questions, and efforts were made during interviews to allow subjects to freely discuss any aspect of their exam preparation they wished to. However, it is possible that the questions themselves may have biased subjects by making them believe that only topics raised by interviewers were important. Our methodology called for the authors to code interviews and to discuss differences of opinion. The lack of other reviewers to serve as “tiebreakers” may have introduced bias, although there were only minor discrepancies in coding.

Finally, just over half of the eligible residents participated in the study. It is possible that residents with stronger opinions of the in-training exam were more likely to agree to participate. If the remaining eligible residents had participated, it is conceivable that this would have affected the results. However, the authors believe that theoretical saturation was achieved in the work.^7,8

Conclusion

To our knowledge, this is the first study to identify themes concerning residents’ attitudes and opinions about in-training exam preparation. Residents generally value the exam as a marker of their academic progress, and as an opportunity to gain familiarity with the qualifying exam. They prefer question-based preparation to text-based learning, as long as there is a detailed explanation of each question. Residency leaders who develop structured in-training review programs can focus on question-based material with detailed explanations. Further work is required to overcome the limitations of this study, such as a multi-centered qualitative study, or outcomes-based quantitative evaluation.