Class-Wide Access to a Commercial Step 1 Question Bank During Preclinical Organ-Based Modules: A Pilot Project

Abstract

Purpose

The authors examined the usefulness of a commercially available Step 1 question bank as a formative academic support tool throughout organ-based modules in an integrated preclinical medical curriculum. The authors also determined the extent to which correlation between question bank utilization and academic metrics varied with Medical College Admission Test (MCAT) scores.

Method

In 2015, a cohort of 185 first-year medical students at University of Alabama School of Medicine were provided with 18-month full access to a commercially available Step 1 question bank of over 2,100 items throughout organ-based modules, although there were no requirements for use. Data on student use of the question bank were collected via an online administrative portal. Relationships between question bank utilization and academic outcomes including exams, module grades, and United States Medical Licensing Examination (USMLE) Step 1 were determined using multiple linear regression.

Results

MCAT scores and number of items attempted in the question bank significantly predicted all academic measures, with question bank utilization as the stronger predictor. The association between question bank utilization and academic outcome was stronger for individuals with lower MCAT scores.

Conclusions

The findings elucidate a novel academic support mechanism that, for some programs, may help bridge the gap between holistic and mission-based admissions practices and a residency match process that places a premium on USMLE exam scores. Distributed formative use of USMLE Step 1 practice questions may be of value as an academic support tool that benefits all students, but particularly those entering with lower MCAT scores.

The Medical College Admission Test (MCAT) and Step 1 of the United States Medical Licensing Exam (USMLE Step 1) have been identified by Gliatto and colleagues¹ as key points of constraint that define much of the medical education process. During the admissions process, holistic approaches have made an effort to avoid exclusive focus on metrics such as the MCAT and better incorporate students’ broader attributes and experiences that may contribute to their potential for a promising career in medicine.^2,3 This has eased some concerns that overemphasis on the MCAT may disadvantage applicants from diverse backgrounds who may be the focus of institutional mission.^4,5 It has also facilitated greater inclusion of students from a wider range of academic backgrounds and majors, not all of whom may have had equivalent premedical advising or who may come from smaller educational systems that did not emphasize standardized testing.³

In contrast, USMLE Step 1 remains a critical metric used by residency selection committees to “triage” the high volume of applications they receive. Aside from not being the original intent of the test,⁶ this may inadvertently disadvantage students who do not excel at standardized testing but are likely to be competent physicians.^4,5,7,8 Performance on standardized testing has been specifically cited as an ongoing barrier to progress for the types of underrepresented students who may be targeted by mission-based admissions practices.^9,10

Given that performance on the MCAT has long been recognized as an independent predictor of USMLE Step 1 performance,^11–13 students scoring below the national average on the MCAT are at a higher risk of poor performance on the USMLE Step 1.¹⁴ One of the current challenges of medical student academic support is to help students of increasingly diverse academic backgrounds succeed academically and prepare for a residency application process that is inherently metric focused.¹⁵ In addition to content mastery, tools are needed to help students develop test-taking competencies tailored to the type of testing they will encounter on USMLE exams.

Not surprisingly, factors predicting better USMLE Step 1 performance include exposure to National Board of Medical Examiners (NBME)-style questions during test preparation.¹⁶ Although active retrieval strategies have long been associated with enhanced learning and retention,¹⁷ the concept of “testing effects” as described by Roediger and colleagues^18,19 suggests that active retrieval specifically in a test question format may be particularly beneficial, either through reinforcement of content or by learning within a framework that more readily transfers to actual testing demands (i.e., “transfer-appropriate processing”). Although question bank use is common during dedicated periods of USMLE Step 1 preparation, the value of exposure to practice questions throughout preclinical course work has not been fully explored. There is, however, evidence that testing effects may be more pronounced with exposure to material at intervals, rather than massed exposure (such as may be typical in a dedicated Step 1 prep period).¹⁸ Many institutions have sought to use more “board-style” questions on exams developed by instructors, although this approach may not capture the full stylistic range of questions and may underrepresent important testable concepts. Access to and use of NBME Customized Assessments²⁰ may address these problems, but the exposure to such items is still fairly limited, and formative feedback is not possible. Item exposure without feedback, while potentially beneficial, will not necessarily yield the same testing effect on recall as exposure with feedback.¹⁸ As such, this is not an optimal platform for distributed formative learning or development of test-taking skills for students needing to develop such skills.

In this study, we examined the relationship between medical students’ use of a Step 1 question bank and preclinical academic performance. As part of a pilot project, in 2015 one class cohort at the University of Alabama School of Medicine was offered 18-month access to a commercially available Step 1 preparatory question bank (over 2,100 items) throughout organ-based modules in an integrated curriculum. We compiled and tracked data regarding question bank use in each module to address the hypothesis that, controlling for academic risk/aptitude (as measured by MCAT score), greater use of the question bank during organ-based modules would be associated with higher scores on a variety of academic metrics: instructor-designed exams, NBME Customized Assessments, module final grades, and USMLE Step 1. On the basis of early observations and anecdotal student feedback when the question bank was initially offered, we also hypothesized that the association between question bank use and higher scores on outcome measures would be stronger for students with lower MCAT scores (i.e., an interaction between MCAT scores and number of questions attempted).

Method

Participants

The study was subject to review and approval by the Institutional Review Board for Human Use at the University of Alabama at Birmingham. Participants were an undergraduate medical school class cohort of 185 preclinical students, of whom 79 (43%) were women and 106 (57%) men. We examined academic performance data that are collected as part of routine administrative tracking.

Question bank access

We provided students with 18-month unlimited access to an NBME-style commercial individual question bank account (the Kaplan USMLE Step 1 Qbank), beginning as they transitioned from introductory fundamental science courses in the fall of their first year to integrated organ-based modules in the spring. Access extended throughout all organ-based modules as well as through a five-week Step 1 dedicated preparation period and the deadline to sit for the test itself. We provided instructions on how to access the account, along with an optional orientation to features and use of the question bank. Otherwise, we gave no directives as to how the question bank should be used, and its use was not required in any module.

We collected data via a Web-based administrative portal that provided tracking of question bank use and performance for all students. During the course of nine organ-based modules, we recorded the number of items attempted and correctly answered by each student for that particular organ or system during the time period of the module. Students were able to customize their use of the questions in a number of ways including timed or untimed, with or without immediate feedback (i.e., tutor mode). They could also generate blocks of questions from individual or multiple disciplines and systems. Data were not available regarding use of these customized settings. Concurrent use of other resources was not tracked. Anecdotally, students reported that the question bank was used in a variety of ways and at varying times during a given module, although students generally tended to use it for the organ system corresponding to a given module.

Data analysis

Data analyses focused on four outcome measures: scores on instructor-designed exams (typically one in each module); scores on exams developed by module directors using NBME Customized Assessment Services (one in each module); final module grades; and USMLE Step 1 score. One module (neuroscience) administered weekly quizzes in lieu of an instructor-designed exam, and thus we used the average of all quiz scores in place of an instructor-designed exam score. We generated linear z score composites across all modules for each outcome measure. Each was the criterion in a multiple linear regression analysis with predictors that included MCAT score, total number of items attempted in the question bank across all nine modules, and an interaction term (i.e., MCAT × number of items attempted). Given that the average of multiple MCAT attempts has been shown to best predict academic outcomes,²¹ this metric was used in cases of multiple MCAT scores. Statistical analyses were conducted using Stata statistical software, version 14.1 (StataCorp, College Station, Texas), α = 0.05, two tailed.

Results

We obtained complete data for all participating students, although question bank data for one individual constituted an extreme univariate and bivariate outlier. Excluding this individual, analyses included data from 184 students (79 women, 105 men). Of these, 117 (63.6%) took the MCAT once, another 60 (32.6%) took the MCAT two or three times, and 7 individuals (3.8%) took the MCAT more than three times. Mean of the highest MCAT scores was 30.0 (SD = 3.5), and the mean of participants’ average MCAT score was 29.2 (SD = 3.9). Three individuals completed the preclinical curriculum but had not taken USMLE Step 1 at the time of data analysis. Thus, the Step 1 analysis sample consisted of 181 first-time test takers (mean = 223.7, SD = 22.3).

Use of the question bank by organ module is summarized in Table 1. Module-specific use depended on the number of questions available for that module, which varies substantially. Question bank use by module varied from 22.3% of students in the neuroscience module to 92.4% of students in the reproductive module. By the end of the nine modules, 97.3% of students had accessed and used the question bank to varying degrees. The number of items attempted across all nine modules ranged from 0 (five participants) to 1,470 (mean = 655.6, SD = 327.7). Excluding the five participants who did not use the question bank, students attempted a mean of 673.9 (SD = 313) questions.

Table 1.

Use of the Question Bank by Module, Presented Chronologically, for 184 Preclinical Medical Students, From a Study of Student Use of a Commercial Step 1 Question Bank (the Kaplan USMLE Step 1 Qbank) and Academic Metrics, University of Alabama School of Medicine, 2015

Module	% of 184 students attempting any items	Mean (SD) items attempteda	Median items attempteda	Median % correcta
Cardiology	68.5	84.0 (78.7)	58	45.0
Pulmonary	78.8	88.5 (46.8)	97	52.8
Gastrointestinal	89.1	143.5 (56.1)	173	58.8
Renal	87.5	85.4 (24.6)	98	62.0
Musculoskeletal/skin	88.0	118.4 (54.9)	133	61.0
Neuroscience/behavior	22.3	24.5 (31.1)	10	40.6
Hematology/oncology	70.1	51.9 (28.6)	55	60.0
Endocrine	84.2	86.2 (51.2)	88	65.2
Reproductive	92.4	115.6 (68.3)	132	64.3
All modules	97.3	673.9 (313.0)	665	59.3

Abbreviation: USMLE indicates United States Medical Licensing Examination.

^aFor students who attempted items.

Correlations among outcomes and predictors are presented in Table 2. Individual regression models for the four primary performance outcomes are presented in Table 3. Question bank use was measured as number of items attempted per student. Number of items correct was highly correlated with number of items attempted and thus not included as a separate predictor. As previously established, average MCAT score emerged as a significant predictor of all outcomes, although it was only modestly correlated with number of question bank items attempted. The data represented in Table 3 also show that the number of question bank items attempted was a significant independent predictor of all four outcomes (controlling for MCAT). The interaction between items attempted and MCAT was likewise statistically significant for all outcomes. The nature of this interaction is presented visually in Figure 1, with the relationship between items attempted and outcome presented for three selected MCAT values (25, 30, and 35). As hypothesized, lower MCAT scores were associated with a stronger positive relationship between question bank use and academic outcomes. Therefore, it is evident that participants with lower MCAT scores who seldom used the question bank were more likely to score below the class average on all preclinical performance outcomes; however, performance of students entering with below-average MCAT scores who highly used the question bank performed closer to their peers with higher MCAT scores on all outcomes, including on the USMLE Step 1 examination.

Table 2.

Correlations Among Academic Measures and Question Bank Use for 184 Preclinical Medical Students, From a Study of Student Use of a Commercial Step 1 Question Bank (the Kaplan USMLE Step 1 Qbank) and Academic Metrics, University of Alabama School of Medicine, 2015

Academic metric	1.	2.	3.	4.	5.	6.	7.
1. MCAT	1.00	—	—	—	—	—	—
2. Question bank items attempted	0.21a	1.00	—	—	—	—	—
3. Question bank items correct	0.25b	0.96b	1.00	—	—	—	—
4. Instructor-designed exams	0.41b	0.36b	0.46b	1.00	—	—	—
5. NBME Customized Assessments	0.47b	0.42b	0.53b	0.90b	1.00	—	—
6. Module final grade	0.45b	0.39b	0.50b	0.96b	0.97b	1.00	—
7. USMLE Step 1 examinations	0.56b	0.39b	0.49b	0.79b	0.87b	0.84b	1.00

Abbreviations: USMLE indicates United States Medical Licensing Examination; MCAT, Medical College Admission

Test; NBME, National Board of Medical Examiners.

^aP < .01.

^bP < .001.

Table 3.

Multiple Regression Results for Primary Outcome Measures for 184 Preclinical Medical Students, From a Study of Student Use of a Commercial Step 1 Question Bank (the Kaplan USMLE Step 1 Qbank) and Academic Metrics, University of Alabama School of Medicine, 2015

Model	Adjusted R2	Standard coefficient	P value
Instructor-designed exams (z score composite)	0.26	—	—
MCAT	—	0.61	< .001
Question bank items attempted	—	1.32	.005
MCAT × items attempted	—	−1.13	.027
NBME Customized Assessments (z score composite)	0.33	—	—
MCAT	—	0.63	< .001
Question bank items attempted	—	1.21	.007
MCAT × items attempted	—	−0.96	.047
Module final grade (z score composite)	0.31	—	—
MCAT	—	0.68	< .001
Question bank items attempted	—	1.46	.001
MCAT × items attempted	—	−1.26	.01
USMLE Step 1 examinationa	0.40	—	—
MCAT	—	0.79	.001
Question bank items attempted	—	1.42	< .001
MCAT × items attempted	—	−1.25	.008

Abbreviations: USMLE indicates United States Medical Licensing Examination; MCAT, Medical College Admission

Test; NBME, National Board of Medical Examiners.

^aTotal of 181 first-time test takers.

Figure 1.

Relationship between items attempted and academic outcomes at selected MCAT values, from a study of preclinical student use of a commercial Step 1 Question Bank (the Kaplan USMLE Step 1 Qbank) and academic metrics, University of Alabama School of Medicine, 2015. Regression lines illustrating the MCAT × items attempted interaction at selected MCAT values (25, 30, 35) illustrated for (A) instructor-designed exams, (B) NBME Customized Assessments, (C) module final grades, and (D) USMLE Step 1 scores. Abbreviations: MCAT indicates Medical College Admission Test; NBME, National Board of Medical Examiners; USMLE, United States Medical Licensing Examination.

Discussion

These findings support the idea that use of an NBME-style question bank throughout preclinical organ-based modules is associated with stronger outcomes. The interaction between question bank use and MCAT score is of particular interest from the standpoint of academic support. Few academic resources exist that benefit all students while conferring relatively greater benefit to students who may enter medical school with greater academic risk or less well-developed standardized test-taking skills. Although there is some cost involved, provision of question bank access does not require additional academic support staff and may be a means of leveraging limited academic support budgets.

Although our findings suggest a beneficial testing effect associated with question bank use, the nature of that effect is difficult to ascertain given the limited data relating to exactly how and when the bank was used (e.g., timed vs. tutor mode, focused vs. mixed blocks, throughout the module vs. for exam prep). An important distinction warranting further exploration is whether the observed testing effect is due to reinforcement of content memory or transfer-appropriate processing (i.e., test-taking skill). The former may be more likely to generalize to other training settings (e.g., clerkships), whereas the latter would likely be of more circumscribed benefit to USMLE testing. The fact that both MCAT score and question bank use were more strongly associated with NBME exams and USMLE Step 1 scores than scores on instructor-designed exams suggests that an element of NBME test-taking skill underlies this association, rather than simply content knowledge. It also suggests that instructor-generated “board-style” questions do not fully capture the nuances of NBME testing.

Possible causality in the relationship between question bank usage and academic performance is likewise unclear. The fact that MCAT score was only modestly associated with question bank use suggests that it is not exclusively a matter of high-performing students using a resource more extensively (although this is likely a contributing factor). There may be qualitative differences in how students with higher and lower MCAT scores use the question bank, and they may differ in other study habits as well. Of particular interest may be the idea of deliberate practice, which may involve wider use of active study strategies that may be more broadly indicative of self-reflective capacity, planning, motivation, and endurance. Not surprisingly, deliberate practice is more evident in high-achieving students and for students with higher MCAT scores.²² Question bank use may be one of multiple ways in which deliberate practice is applied in their study habits. Further examination of the benefit of the question bank in the context of broader study habits would help clarify whether MCAT score is a true independent predictor or a proxy for other effective study habits.

Conclusions

For participating students, distributed practice of NBME-style questions was associated with stronger performance on module-specific academic outcomes as well as USMLE Step 1, with a relatively stronger effect for individuals with lower MCAT scores. This may be due in part to strengthening of test-taking skills (i.e., transfer-appropriate processing) in addition to enhanced content mastery. Although potential causal relationships remain to be fully explored, distributed practice of NBME-style questions throughout preclinical course work may be a valuable academic strategy, particularly for students entering with lower MCAT scores. For institutions, providing question bank access may also offer a rare opportunity to maximize academic support for all students, while still conveying proportionally stronger benefit to students who may be at elevated academic risk.

Limitations and future directions

Primary limitations of the study include the correlational nature of the analyses and the fact that it reflects a single cohort at one institution. Although data were captured for all students in the cohort, the design was essentially observational in nature, with limited information available regarding motivations for using the question bank, methods and strategies used, and concurrent use of other resources. The fact that no parameters were placed on use of the question bank does, however, bolster the generalizability of the findings to other typical medical student cohorts who are understood to use a variety of resources in a variety of ways. Future research might examine more structured use of the question bank, perhaps by integrating its use more formally within preclinical modules. Our institution is also examining incorporation of the question bank into peer tutoring and group review activities, as well as incorporating data regarding question bank use into routine academic advising.