Unlocking Medical Student Success: A Systematic Review and Meta-Analysis of Third-Party Resources Used for Medical Education and USMLE Board Preparation

Abstract

An increase in third-party commercial resources targeted towards medical students has led to substantial changes in the way students prepare for medical assessments. A systematic review and meta-analysis was conducted to identify the third-party resources most used by medical students. A total of 1056 citations were screened with 19 full-text studies included. Twenty-two unique third-party resources were examined, with meta-analysis revealing the three most used resources to be First Aid, UWorld, and Pathoma. Medical faculty should consider curriculum adaptation to better cater to the learning needs of students and institute systems that ensure equitable access to verified commercial resources.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-024-02116-7.

Introduction

The United States Medical Licensing Exam (USMLE) is a series of standardized exams designed to assess the foundational and clinical competencies of medical trainees [1]. Demonstrating proficiency in these exams is not only a crucial benchmark to progress from medical school to residency, but performance on these exams is also used to stratify applicants applying to residency [2]. Given increasing competition for residency positions and emphasis on USMLE performance in residency application evaluation, medical students often tailor their studying to the USMLE exams, using third-party resources both as a supplement to their traditional medical school curriculum and as preparation resources for these standardized board exams [3, 4].

Due to this demand, there has been an explosion in the number and availability of these test preparation resources available to medical students over the past 20 years. Many of these resources are now considered necessary for medical student success and are recommended by medical educators [4]. Navigating the realm of third-party resources can oftentimes be overwhelming and expensive given the large number of resources available. Furthermore, it is generally not a part of a standardized medical curriculums to introduce students to these third-party resources and explain which ones to use and/or how to use them [3]. Rather this is knowledge students are forced to pick up through the “hidden” curriculum or through conversations with other students [5–7].

There have been several studies examining medical student use of third-party resources. However, many of these have occurred within isolated, individual school settings and as a result, their findings may not be able to be applied to medical students across the United States (US) in general [3, 8]. The aim of this study is to perform a systematic review and meta-analysis on all available studies investigating the use of third-party resources among medical students in order to identify and quantify the most prevalent resources used by medical students. It is our hope that the insights obtained from this comprehensive literature review inspire medical curriculum reform to provide more equitable access to highly utilized commercial resources and provide valuable guidance to future students in deciding what third-party resources to use.

Methods

The published literature was searched using strategies designed by a medical librarian for the concepts of medical students (including both allopathic and osteopathic medical students), the UMSLE or curriculum, and third-party resources, with related synonyms. The strategies were created using a combination of controlled vocabulary terms and keywords and were executed in Embase.com, Ovid-Medline All, Scopus, Web of Science, and ERIC (ProQuest). Results were limited to January 1, 2000 to April 13, 2023. Complete search strategies are in the Supplemental Methods. A total of 1056 results were retrieved, of which 473 duplicate citations were identified and removed. The remaining 582 citations were uploaded to Covidence software for further screening analysis (Fig. 1) [9]. Four reviewers (M.R.K, G.H, S.P, D.C.) performed title and abstract screening, ensuring that each manuscript title and abstract was screened twice. In cases of disagreement between the initial two reviewers, a third reviewer resolved the conflict. Full text screening was completed using inclusion criteria: 1) English manuscript, 2) original research, 3) about United States (US) medical students, 4) quantification of third-party resource usage for USMLE or curricular preparation. Any reviews, conference abstracts, or non-original research were excluded. Data extraction included information regarding study design, population, percent of students using resources, purpose for resource use, outcomes measures, and a description of main findings.

Fig. 1.

Flowchart showing the study selection process for a systematic review of the literature from January 1, 2000-April 13, 2023 executed in Embase.com, Ovid-Medline All, Scopus, Web of Science, and ERIC (ProQuest) databases on medical student use of third-party commercial resources as supplements to the medical school curriculum or as aids for USMLE board exam preparation

The meta-analysis utilized a forest plot, graphically presenting the consistency and reliability of results from selected studies. Following the recommended steps by Neyeloff and Fuchs [10], the forest plot was designed using Microsoft Excel Version 16.1.1. Heterogeneity among the results was detected by using the chi-square test together with calculating the I² statistic. I² was calculated using the formula I² = (Q-df)/Q × 100%, where Q is the Cochran heterogeneity statistic and df the degrees of freedom [11]. A value of I² greater than 50% represents high heterogeneity and a value below 50% represents low or moderate heterogeneity [11–13]. A fixed-effect model was used when heterogeneity was low and a random-effect model was utilized when heterogeneity was high to calculate the effect summary. Ultimately due to high heterogeneity, a random effects model was employed across all meta-analysis as a more conservative assessment that accounted for both within study and between study variability [14, 15]. Effect size was computed for each study and a pooled effect size with 95% confidence intervals was calculated to summarize across studies. The meta-analysis was performed exclusively for resources that had a minimum of five studies investigating student use of that resource as sample sizes lower than five may lead to unreliable estimates [14, 15]. Studies were excluded from meta-analysis that implemented school programs that may have promoted the use of a specific third-party resource to avoid potential overinflation of the natural usage of these resources by medical students [16].

Results

Of the 582 unique studies identified, 483 were excluded during title and abstract screening and 80 were excluded during full-text review (Fig. 1). The final 19 studies included in data extraction and analysis were published between 2013 and 2022. Among these, 18 (94.7%) utilized student surveys and 1 (5.2%) utilized a web-based administrative portal to capture quantitative information on student use of commercial third-party resources. The study population was comprised of allopathic medical students in 15 (78.9%) studies, osteopathic medical students in 4 (21.1%) studies, and both osteopathic and allopathic medical students in 1 (5.2%) study. The study aims, participant population, reported student usage of third-party resources, outcome measures, and main findings of the 19 studies are described in Table 1. The specific commercial third-party resources examined across all studies were UWorld (8), First Aid (7), Anki (6), Pathoma (5), Kaplan Qbank (5), Firecracker (5), SketchyMedical (4), Osmosis (4), Goljan Audio Podcasts (3), USMLE Rx (2), USMLE Practice Exams (2), Board Review Series (2), Boards and Beyond (2), SecondLook (1), Amboss (1), Doctors in Training (1), Master the Boards (1), OMT review (1), Step Up to Medicine (1), Complex Qbank (1), Draw it to Know it (1), and Youtube (1). A brief description of these resources, including price, unique features, and information coverage, is provided in Table 2.

Table 1.

Characteristics of 19 selected articles about medical student usage of commercial third-party resources

Author, Year	Aim of Study	Participant Population	Use of Third-Party Resources	Outcome Measure	Description of Main Findings
Vora et al. [17]	• Explore the association between scoring at least 600 on COMLEX-USA Level 1 and GPA, MCAT score, and different test preparation strategies	• 113 third-year osteopathic medical students at Nova Southeastern University College of Osteopathic Medicine	• Review Books: 96% (106/113); Practice Exams: 50% (56/113); Question Banks: 91% (103/113); Lecture Videos: 44% (49/113); Class Notes: 3% (3/113)	• Performance on COMLEX-USA Level 1	• Scoring 600 or higher was associated with a higher GPA (p < .02), a higher MCAT score (p < .05), earlier preparation initiation (p < .05), and not ranking the Comprehensive Osteopathic Medical Self-Assessment Examination (COMSAE) as the most helpful practice examination (p < .04)
Tackett et al. [18]	• Examine student engagement with medical education videos incorporated into a preclinical Cardiovascular Systems course	• 359 students who took the first-year 8-week Cardiovascular Systems course in 2019 and 2020 at University of Miami Miller School of Medicine	• Osmosis: 100% (359/359)	• Student feedback	• Most students found Osmosis videos to be helpful for learning (204/232, 87.9%; p = .001) and preferred Osmosis videos to the traditional lecture format (134/232, 57.8%; p < .001)
Seal et al. [19]	• Examine the predictive value of the NBME and UWorld on Step 1	• 466 anonymous Reddit users who had previously taken USMLE Step 1	• UWorld: 82% (383/466); UWSA 1: 77% (360/466); UWSA2: 79% (369/466); NBME Practice Tests 13–24: 14% (64/466)-71% (333/466) depending on which practice test	• Performance on Step 1	• Students who took at least one UWorld practice test did significantly better on Step 1 (p = .001) compared to those who took none. Students who took both UWorld practice tests did significantly better on Step 1 (p = .04) compared to those who only took one. Students who took at least one of the five newest NBME practice tests did significantly better (p = .0003) than those who took none
Schwartz et al. [20]	• To describe the development, evaluation, and effects of a student-initiated USMLE Step 1 preparatory program. The program provided second year students with First Aid for the USMLE Step 1, a six month subscription to the USMLE World question bank, and optional peer review sessions covering basic sciences and organ systems taught by high-performing upperclassmen	• 5 cohorts of students at University of Illinois College of Medicine at Chicago from 2010–2014. The 2010–2012 student cohorts did not use the preparatory program while the 2013–2014 cohorts did	• The survey on third party resource use was only distributed to the 2012 cohort (N = 159) • First Aid: 98% (156/159); Uworld: 80% (125/163); Kaplan Qbank: 2% (4/163)	• Step 1 pass rate and score	• After implementing the USMLE Step 1 preparatory program, the average step 1 score increased by 8.82 points (p < 0.001) and there was a higher first-attempt pass rate by 8% (p < 0.001)
Parry et al. [21]	• This study sought to examine preparatory resources and their association with USMLE Step 1 performance	• 170 medical school students completing their second year in June of 2016 and 2017 at a state-funded, public medical school in the Midwest	• UWorld: 92% (157/170) • First Aid: 91% (154/170)	• Performance on step 1	• For UWorld, there was a significant effect of completion rates on USMLE Step 1 scores (p = 0.001). For First Aid, there was no statistically significant effect of completion rates on exam scores
Lu et al. [22]	• The objective of this study was to identify Anki use and its association with USMLE Step 1 performance	• 201 second-, third-, and fourth- year medical students at the University of North Carolina School of Medicine in March 2020	• Significant Anki Use: 66% (132/201)	• Performance on step 1	• Anki users step 1 score (mean = 241.10, standard deviation = 13.17) was significantly higher (p = 0.12) than that of nonusers (mean = 235.51, standard deviation = 17.68)
Kind et al. [23]	• To describe medical students’ reading habits and resources used during clinical clerkships, and to assess whether these are associated with Step 2 performance outcomes	• 130 third- year clerkship students from The George Washington University School of Medicine, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, and The University of Illinois College of Medicine at Urbana Champaign	• Question Banks: 92% (120/130); Online Databases: 81% (105/130); Test Prep Books: 80% (104/130); Other Online Resources: 61% (79/130); Primary Literature/Review Articles: 58% (75/130); Textbook: 55% (72/130); Essentials/Pocket-sized books: 35% (45/130); Online cases: 28% (36/130); Notes/Materials from prior coursework: 21% (27/130)	• Performance on Step 2	• Step 1 score (p < .001), Shelf-exam percentile score (p < .001), and number of hours spent reading for exam study (p = .01) were significantly associated with an increased Step 2 score. Students rated question banks and board review books as most useful for exam preparation and rated online databases and pocket-sized textbooks as most useful for patient care. 80% of students ranked classmates as being most influential for recommended resources
Kauffman et al. [24]	• Mixed-methods approach correlating student attendance and access to the formal curriculum in a second-year pathophysiology course to performance on Step 1. Additionally, survey and focus group data evaluated the usage and importance of both the formal curriculum and third-party resources	• 48 second-year medical students enrolled in the GI/Renal Module in the 2017–2018 academic year at the University of Central Florida College of Medicine	• Uworld*: 96% (46/48); Pathoma*: 98% (47/48); First Aid*: 90% (43/48); Sketchy*: 79% (38/48); Boards and Beyond*: 60% (29/48); Osmosis*: 52% (25/48); Firecracker*: 25% (12/48); Goljan*: 23% (11/48); Kaplan*: 19% (9/48); Board Review Series*: 17% (8/48); Anki*: 8% (4/48); Board Vitals*: 4% (2/48); Khan Academy*: 4% (2/48); PreTest*: 2% (1/48); Becker Pharmacology*: 2% (1/48)	• Performance on Step 1	• Survey and focus group data show that students place high importance on faculty-developed materials that they can use on their own, but not attendance. The third-party resources are highly used as an adjunct to the formal curriculum and to focus on Step 1 study. Attendance and access to the formal curriculum do not predict Step 1 performance, whereas performance on high- and low stakes internal assessments do
Hudder et al. [25]	• Analyze the first year of use of the Osmosis platform among osteopathic medical students and to identify barriers to its use	• 1135 first- and second-year students at Lake Erie College of Osteopathic Medicine during the 2016–2017 academic year across all 3 campuses	• First-year students Osmosis use: 73% (416/573) created account; Second-year students Osmosis use: 27% (151/562) created account	• Student use and feedback	• Osmosis adoption and use varied by campus, educational pathway, and individual students. Barriers to use reported by students in surveys and in feedback included limited time to learn new study skills, technical issues integrating the platform with other applications they use, and preferences for other study habits
Graff et al. [26]	• Examines perceptions between students and faculty regarding the efficacy of lecture-based teaching and learning strategies used by students overall	• 155 medical students and 81 faculty members from Medical College of Wisconsin	• Sketchy Medical: 74 (114/155); First Aid: 64% (99/155); YouTube: 63% (98/155); Pathoma: 54% (84/155); Academic Enhancement (student led tutoring program): 52% (80/155); Firecracker*: 12% (19/155); Draw-it-to-Know-it*: 15% (23/155); Anki*: 22% (34/155); Evidence based journals*: 21% (33/155)	• Student and faculty feedback	• Student and faculty perceptions regarding student learning needs were significantly different. Students use lectures extensively, but additionally add to the financial burden of medical school by personally funding supplemental resources
Gely et al. [27]	• Determine the effect of test preparation resources and race on test scores	• 192 Rush University Medical Center first-year to fourth-year medical students enrolled during the 2020–2021 academic year	• Step 1 resource: 69% (132/192) • Step 2 resource: 40% (76/192)	• Step 1 and Step 2 performance	• No differences between test scores existed between races and there was no association between test scores and the use of test preparation resources
Finn et al. [28]	• Explore medical student use of e-learning devices and resources during phases M1 and M3 when preparing for Step 1. The ultimate purpose of this study was to help medical and other educators optimize their use of e-learning resources	• 108 first-year and 119 third-year medical students at University of Michigan Medical School	• SecondLook: M1-64.8% (70/108), M3-1.7% (2/119); Amboss: M1-22.2% (24/108), M3-10.1% (12/119); Anki: M1-59.3% (64/108), M3-47.1% (56/119); Boards and Beyond: M1-44.4% (48/108), M3-17.6% (21/119); Firecracker: M1-0.0% (0/108), M3-0.0% (0/119); Goljan Audio podcasts: M1-5.6% (6/108), M3-15.1% (18/119); Lecturio: M1-0.9% (1/108), M3-0.0% (0/119); MD Cases: M1-0.0% (0/108), M3-0.0% (0/119); MEDBULLETS: M1-2.8% (3/108), M3-0.8% (1/119); Osmosis: M1-51.9% (56/108), M3-10.9% (13/119); Pathoma: M1-29.6% (32/108), M3-82.4% (98/119); Physeo: M1-0.9% (1/108), M3-0.0% (0/119); Picmonic: M1-0.9% (1/108), M3-5.9% (7/119); Sketchy Medical: M1-63.9% (69/108), M3-73.9% (88/119); USMLA-Rx: M1-12.0% (13/108), M3-14.3% (17/119); UWorld: M1-1.9% (2/108), M3-96.6% (115/119)	• Student feedback	• M1 students often preferred resources that were tailored to the specific courses in their curriculum while M3 preparing for Step 1 shifting almost exclusively to a solitary learning strategy using commercial e-learning resources. Peer advice was the major determinant influencing e-learning resource selection and videos were the most popular e-learning modality. Students cited efficient acquisition of knowledge and preparation for examinations as major reasons for e-learning tool utilization. Current e-learning tools are used in rather specific ways and more effort by educators might be needed to enhance their wider appeal and appropriate use by medical students
Emahiser et al. [29]	• Gain insight into the reasons students skipped class, identify the type of study materials they were using, and determine what they thought would motivate them to come to class	• 92 first- and 53 second-year medical students at Touro University Nevada's College of Osteopathic Medicine	• Percent of students who listed resource in their top 3 • Pathoma*: M1-5% (5/92), M2-51% (27/53); Sketchy Ultimate*: 0% (0/92), M2-40% (21/53); Anki*: M1-37% (34/92), M2-36% (19/53); Boards and Beyond*: M1-31% (29/92),M2-22% (12/53); First Aid*: M1-4% (4/92), M2-22% (12/53); UWorld*: M1-0% (0/92), M2-21% (11/53); Board Review Series ractice questions*: M1-4% (4/92), M2-5% (3/53); Youtube videos*: M1-33% (20/92), M2-4% (2/53); Firecracker*: M1-0% (0/92), M2-0% (0/53); Google Scholar*: M1-0% (0/92), M2-0% (0/53)	• Student feedback	• Students perceive that lectures may not prepare them for Board exams and would be more motivated to go to lecture if faculty addressed Board related content more often in class and on assessments. Students also heavily utilized outside study materials to prepare for Board exams
Deng et al. [30]	• Examined the prevalence of specific self-directed methods of testing, with or without spaced repetition, among preclinical students and assessed the relationship between these methods and licensing examination performance	• 72 students at Washington University School of Medicine in St. Louis who completed the two-year preclinical curriculum in 2014 and had taken Step 1	• Anki: 31% (22/72) • Firecracker: 49% (35/72)	• Performance on Step 1	• In bivariate correlations with Step 1 score, the variables that were significantly related included MCAT score, number of honors grades, number of boards-style multiple choice questions, and number of Anki questions (p < 0.05). In a multivariate regression model, significant independent predictors of Step 1 score included boards-style multiple-choice question completed (B = 0.0022, p < 0.001), unique Anki flashcards seen (B = 0.00059, p = 0.024), second-year honors (B = 1.198, p = 0.002), and MCAT score (B = 1.078, p = 0.003)
Harris et al. [31]	• Determine the usage of Anki among first-year medical students in an integrated first-year module consisting of anatomy and physiology	• 60 first-year medical students at the University of Central Florida College of Medicine	• Anki*: 70% (42/60)	• Student use and feedback	• The majority of first-year medical students used Anki to help them study for the Structure and Function module. The number of Anki cards and daily use of Anki cards varied among users. Students perceived the need to supplement their learning with external resources as opposed to replace, students preferred to use already made cards as opposed to creating their own, and students preferred to do Anki after watching a lecture
Burk-Rafel et al. [3]	• Determine medical students study behaviors when preparing for the USMLE Step 1 and how these behaviors are associated with Step 1 scores	• 274 medical students at University of Michigan Medical School who completed their first Step 1 attempt within 2014 or 2015	• First Aid 99.3% (271/274); Uworld Qbank: 99.3% (271/274); Pathoma: 90.5% (247/274); Goljan audio: 53.1% (145/274); USMLE-Rx Qbank 25.6% (70/274); Firecracker 19% (52/274); Kaplan Qbank 12.1% (33/274); UWSA: 81% (221/274); NBME CBSSA: 60.9% (167/274)	• Performance on Step 1	• Initiating study prior to the designated study period, increased review book usage, and attempting more practice questions were all associated with higher Step 1 scores, even when controlling for Medical College Admission Test scores, preclinical exam performance, and self-identified score goal (adjusted R2 = 0.56, P < .001)
Bonasso et al. [8]	• Provide medical educators with data on which USMLE Step 1 resources correlate with higher exam scores	• 164 medical students at West Virginia University School of Medicine	• First Aid: 97.6% (160/164); UWorld: 88.4% (145/164); Doctors in Training: 51.8% (85/164); Kaplan Qbank: 15.9% (26/164); Board Review Series: 42.1% (69/164) used	• Performance on Step 1	• Students who used USMLE World Question bank had higher USMLE Step 1 scores than students who did not use this resource (p < 0.05). No other statistically significant differences were found between groups
Bhatnagar et al. [32]	• Quantify the financial burden experienced by medical students from board preparation and examination	• 290 fourth-year osteopathic medical students from 38 different osteopathic medical schools	• COMBANK COMPLEX Level 1 Qbank: 26.9% (78/290); UWorld Step 1 Qbank: 26.6% (77/290); Pathoma: 25.2% (73/290); COMBANK COMPLEX Level 2 Qbank: 23.8% (69/290); UWorld Step 2 Qbank: 20.7% (60/290); Sketchy: 20.0% (58/290); OMT Review: 20.0% (58/290); Step-Up to Medicine: 14.5% (42/290); Master the Boards: 12.1% (35/290); First Aid for the USMLE Step 1 CK: 10.7% (31/290)	•N/A	• Osteopathic medical students spent $4,129 on average for the cost of board preparation materials
Banos et al. [16]	• Examined the usefulness of a commercially available Step 1 question bank as a formative academic support tool throughout organ-based modules in an integrated preclinical medical curriculum. Determined the extent to which correlation between question bank utilization and academic metrics varied with MCAT scores	• A cohort of 184 first-year medical students at University of Alabama School of Medicine provided with 18-month access to Kaplan starting in the spring of their first year	• Kaplan Qbank: Use ranged from 22.3% (41/184) to 92.4% (170/184) depending on the module. 97.3% (179/184) of students used the kaplan qbank to varying degrees across all modules	• Instructor exam scores • NBME Customized Assessment Services exam scores • Final module grades • Step 1 score	• The number of question bank items attempted was a significant independent predictor of instructor designed exams (p = 0.005), NBME customized assessments (p = 0.007), module final grade (p = 0.001), and USMLE Step 1 Exam (p < 0.001). Distributed practice of NBME-style questions was associated with stronger performance on module-specific academic outcomes as well as USMLE Step 1 with a relatively stronger effect for individuals with lower MCAT scores

^*Exact values were not provided in publications so approximations were made based on bar graphs presented within the publications

NBME National Board of Medical Examiners, M1 first year medical student, M2 second year medical student, M3 third year medical student, M4 fourth year medical student, Qbank Question bank, USMLE United States Medical Licensing Examination, MCAT Medical College Admission Test

Table 2.

Commercial third-party resources description

Resource	Information Coverage	Unique Features	Price
SecondLook	• Self-review tools for college and professional level-students. Subjects covered include Histology, Neuroanatomy, Musculoskeletal Anatomy, Basic Radiology, EKG, Fetal Heart Rate Tracing, Oral Radiology, Dental Anesthesia	• Mobile applications developed by the University of Michigan function as self-review tools, providing a series of images with questions and answers to self-test level of knowledge. Compatible with iOS and Android	• Free of cost
Amboss	• USMLE Step 1, Step 2, Step 3, NBME Shelf Exams	• Amboss Library: > 25,000 textbooks, medical journals, guidelines, and papers distilled into 1 digital platform. > 1,000 radiographic images, videos, illustrations, and flowcharts. Anki add-on that automatically links terms to the Library for efficient learning. Mobile app compatible with iOS and Android • Amboss Qbank: > 2,700 Step 1, > 3,900 Step 2 and Shelf, and > 2,000 Step 3 practice questions with explanations that link to the Amboss Library. Unique tools such as Attending tips and Key info highlights to improve test taking. Mobile app compatible with iOS and Android • Offers 2 free self-assessments per year, 1 for Step 1 and 1 for Step 2, with performance breakdown and 3-digit score	• For students, membership is $129/year or $14.99/month for unlimited access to Amboss library, anki-add on, 50 free Qbank questions per month, and free mobile app use • Choice to upgrade for full Qbank access: $149 for 1 month access, $189 for 3 month access, $229 for 6 month access, $299 for 1 year access
Goljan Audio Podcasts	• USMLE Step 1	• High yield content provided by Dr. Goljan at Oklahoma State University with a unique sense of humor making his lectures fun and entertaining • Available on multiple internet platforms such as YouTube, Spotify, and more	• Free of cost
USMLE Rx	• USMLE Step 1, Step 2	• > 2,300 USMLE style questions for Step 1, > 2,300 USMLE style questions for Step 2. Detailed progress reports with predictive scoring. Two 160 question self-assessments. Mobile app compatible with iOS and Android • Subscription to Rx360 + includes: Step 1 question bank, First Aid integration, Flash Facts (3,500 case-based flash cards), Express Videos (1,300 videos), Rx Bricks (library with 800 interactive modules) to aid studying	• Rx 360 + is $149/1 month, $199/3 months, $329/6 months, $399/12 months, $449/24 months • Can purchase individual items separately or bundles to save money • Step 1 question bank is $129 for 1 month, $299 for 12 months
Board Review Series	• USMLE Step 1	• Textbook series that covers all core basic science material. Offers diagnostic images, figures, over 4,500 multiple choice questions with explanations, and options to customize quizzes via subject	• Available on amazon and kindle, prices vary • Institutional subscriptions vary
Doctors in Training	• USMLE Step 1, Step 2, Step 3, COMLEX Level 1, Level 2, PANCE, PANRE	• Primer videos are available for each exam. Includes PDF study guides with board-style questions and answers	• Currently out of business. “New products are coming soon” promoted on their website
Master the Boards	• USMLE Step 2, Step 3	• High yield review utilizing the “Fischer Method” which highlights best initial test, most accurate test, most likely diagnosis, and best initial treatment • Includes algorithms for patient management, diagnostic images, decision trees, flow charts, and comparison tables	• Available on amazon and kindle, prices vary
OMT Review	• COMLEX Level 1, Level 2, USMLE Step 1, Step 2	• > 10 hours of osteopathic principles and practice COMLEX video content, > 4,500 comlex level 1 and 2 practice questions, > 300 OMM review questions, NBOME question bank, 700 OMT review flashcards, performance analytics • USMLE Step 1/COMLEX Level 1 Masterclass includes 94 hours of lecture videos. USMLE Step 2/COMLEX Level 2 Masterclass includes 130 hours of lecture videos	• COMLEX Level Qbank only: $159 for 1 month, $399 for 12 months • Bundle Pack includes Qbank, Masterclass, flashcards, and additional videos: $297 for 1 month, $797 for 12 motnhs
Step Up to Medicine	• USMLE Step 2, NBME Shelf Exams	• Textbook with high-yield review of medicine with enhanced high-yield outline format including “Quick hits” and clinical pearls for efficient studying. 100 question practice exam. Includes algorithms, outlines, charts, tables, graphs and mnemonics to facilitate retention of material. Covers every essential area of medicine. eBook available for purchase	• Available on amazon, prices vary
Combank Qbank	• COMLEX Level 1, Level 2, Level 3, COMAT Shelf Exams, USMLE Step 1, Step 2, PANCE, PANRE	• 2,000 + practice questions, up to date First Aid references, simulated Prometric Testing interface, access to hundreds of osteopathic principles and practice (OMM) videos, access to mnemonic videos by picmonic, national benchmarking, first-time pass guarantee	• Multiple subscription and bundle packages exist, starting from $199 for 30 days to $569 for 545 days
Draw it to Know it	• MCAT, USMLE Step 1, Step 2, Step 3, COMLEX Level 1, Level 2, Level 3, PANCE, PANRE, Neurology RITE Exam	• Use of whiteboard tutorials with interactive learning exercises, flashcards and drills, thousands of board style questions, and practice guidelines • Web-based active learning tools work across all devices. Mobile app compatible with iOS	• USMLE/COMLEX preparation materials: $34.99/month
Firecracker	• USMLE Step 1, PANCE, PANRE, medical school and osteopathic school courses	• Personalized tool and essential digital tutor with ability to tailor study plans based on your knowledge and goals. Utilizes spaced repetition learning for long-term memory retention	• Free 7-day trial, price not available
Boards and Beyond	• USMLE Step 1, Step 2, Step 3 COMLEX Level 1, Level 2, Level 3, NBME Basic Science Exams, NBME Shelf Exams	• Step 1/Preclinical: 447 videos and 2325 practice questions, emphasize understanding of basic and preclinical sciences • Step 2, Step 3/Clinical: 265 videos, 1395 practice questions	• Multiple subscription plans exist, between $24 for 1 week to $399 for 2 years
Youtube	• Users can search for videos covering any topic of their choosing	• Students and professionals can find video lectures, tutorials, and explanations on various medical topics and clinical skills • Channels run by medical education and institutions often provide high-yield content, mnemonics, and visual aids. Accessible to everyone with internet connection. Mobile app compatible with iOS and Android	• Free of cost
Pathoma	• USMLE Step 1	• Based on the Fundamentals of Pathology textbook by Dr. Husain Sattar. Includes 35 + hours of online videos covering the textbook and over 300 images. iOS support (beta), Android support	• Option for 3, 12, or 21 month subscriptions starting at $84.95
SketchyMedical	• MCAT, USMLE Step 1, Step 2, PANCE, NAPLEX, NCLEX	• Transforms must-know concepts into fun, memorable stories. High-yield content review using memorable visual mnemonics. Includes 4795 + board-style practice questions and custom quizzes. Mobile app compatible with iOS and Android	• Multiple subscription plans exist, between $50/month for 6 months to $25/month for 24 months
Anki	• Completely customizable to the individual depending on the decks/flashcards used	• Utilizes spaced repetition to help users learn and retain information more efficiently. It allows users to create digital flashcards or use pre-made decks to study and review information in a structured and efficient manner. Mobile app compatible with iOS and Android	• AnkiWeb is free of cost • iOS application costs $24.99
Osmosis	• USMLE Step 1, Step 2, COMLEX Level 1, Level 2, Foundational Sciences, Organ Systems, Clerkships, PANCE, NCLEX, Dentistry, Pharmacy	• 1,700 + videos, 15,000 + flashcards powered by spaced repetition, 2,700 + board-style practice questions, high-yield notes. Mobile app compatible with iOS and Android	• Multiple subscription plans exist, between $318 for 6 months to $618 for 2 years
UWorld	• MCAT, USMLE Step 1, Step 2, Step 3, COMLEX Level 1, Level 2, UKMLA, Internal Medicine ABIM, Family Medicine ABFM, PANCE, PANRE, NCLEX-RN, NCLEX-PN, FNP, NAPLEX, MPJE, CPJE	• Question banks with questions at or above exam-level difficulty and modeled after actual exams. Includes self-assessments, content-rich explanations for correct and incorrect answer choices, ability to flag questions, customizable exams, performance graphs, and flash cards. Mobile app compatible with iOS and Android	• Multiple subscription plans exist, between $319 for 30 days to $559 for 360 days
First Aid	• USMLE Step 1, Step 2, Step 3	• Textbooks annually updated to provide a complete framework for USMLE preparation.1,400 must-know topics with mnemonics, 1,200 photos and illustrations, Rapid Review section for last-minute preparation	• Available on Amazon, prices vary
NBME	• USMLE Step 1, Step 2, Basic science, NBME Shelf Exams	• Offers official practice exams and assessments that closely mirror the USMLE format, provides valuable diagnostic reports and clinically relevant questions. Exams can be administered at a local institution or prometric testing center to mirror test day	• Local exam fees range from $46-$59, Remote exam fee $51, Prometric Testing exam fees range from $103-$149
Kaplan Qbank	• USMLE Step 1, Step 2, Step 3, COMLEX Level 1, Level 2	• 3,300 + board-style questions, 1,000 + images, tables, videos, and 3D interactive anatomical models, two full-length practice exams, one diagnostic exam, detailed answer explanations and progress reports. Mobile app compatible with iOS and Android	• Multiple subscription plans exist, between $159 for 1 month to $499 for 24 months

USMLE United States Medical Licensing Examination, COMLEX Comprehensive Osteopathic Medical Licensing Examination, MCAT Medical College Admission Test, PANCE Physician Assistant National Certifying Exam, PANRE Physician Assistant Recertification Exam, COMAT Comprehensive Osteopathic Medical Achievement Tests, RITE Residency In-service Training Examination, NAPLEX North American Pharmacist Licensure Examination, NCLEX National Council Licensure Examination, UKMLA United Kingdom Medical Licensing Assessment, ABIM American Board of Internal Medicine, ABFM American Board of Family Medicine, FNP Family Nurse Practitioner, MPJE Multistate Pharmacy Jurisprudence Examination, CPJE Californian Pharmacy Jurisprudence Exam

UWorld and First Aid

Eight studies reported on medical student use of UWorld, a learning platform and comprehensive question bank for USMLE board preparation, and 7 studies reported on medical student use of First Aid, a concise and comprehensive textbook covering a range of topics on the USMLE Step 1 exam. Many of these studies explored the use of UWorld and First Aid in the context of their association with student Step 1 performance. These were two of the most widely used resources across all studies, with 157 of 170 second-year students at a Midwest public medical school using UWorld and 154 using First Aid [21], 145 of 164 students at West Virginia University (WVU) School of Medicine using UWorld and 160 using First Aid [8], and 271 of 274 students at University of Michigan Medical School (UMMS) using UWorld and 271 using First Aid as supplemental Step 1 preparatory resources [3]. Within a survey administered in 2012 at University of Illinois College of Medicine (UICOM), 125 of 163 students who had taken Step 1 used UWorld and 159 used First Aid [20], and in a survey study of 466 anonymous Reddit users who had previously taken Step 1, 383 reported using UWorld [19]. Likewise, a study at the Medical College of Wisconsin (MCW) examining the differing perceptions between students and faculty in terms of effective learning strategies found that 99 of 155 survey respondents spanning across first-, second-, and third-year students used First Aid as a supplemental education resource [26]. A similar study at the University of Central Florida (UCF) College of Medicine exploring students decreased engagement with the formal medical curriculum reported that approximately 96% and 90% of 48 surveyed students used UWorld and First Aid, respectively [24]. An investigation into the cost burden of board preparation materials for osteopathic medical students found that among 290 fourth-year osteopathic students across 38 osteopathic medical schools, 77 used UWorld as a Step 1/Level 1 resource, 60 used UWorld as a Step 2/Level 2 resource, and 31 used First Aid for USMLE Step 2 as a Step 2/Level 2 resource [32]. UWorld usage can change drastically across years in medical school as students transition from preclinical into the clinical curriculum and dedicated Step 1 preparation time, with one study reporting only 2 of 108 first-year medical students at UMMS using UWorld while 115 of 119 of third-year students used UWorld to prepare for Step 1 [28]. Employing a random effects model yielded an effect summary of 75.96% (53.61%-98.31%) for UWorld use and an effect summary of 78.22% (39.20%-117.23%) for First Aid use (Fig. 2a and b). Excluding osteopathic medical student specific studies yielded an effect summary of 82.80% (69.01%-96.59%) for UWorld and an effect summary of 89.57% (77.11%-102.04%) for First Aid.

Fig. 2.

Forest plot analysis illustrating the percent of medical students that use A) UWorld, B) First Aid, C) Anki, D) Firecracker, and E) Pathoma as supplements to the medical school curriculum or as aids for USMLE board exam preparation

Anki and Firecracker

Although there are key difference in terms of customizability, pre-structured content, user interface, and community support, Anki and Firecracker are both learning softwares that utilize spaced repetition algorithms to optimize learning and retention. Six studies reported on medical student use of Anki and 5 studies reported on medical student use of Firecracker. The study at MCW found that approximately 22% of 155 first-, second-, and third-year students used Anki and 12% used Firecracker [26], and the study at UCF College of Medicine found that approximately 25% of 48 students used Firecracker and 8% used Anki [24]. Anki use changed slightly as medical students progressed through medical school, with 64 of 108 first-year students at UMMS using Anki while 56 of 119 third-year students used Anki to prepare for Step 1 [28]. An investigation into the association between spaced repetition self-directed learning methods and USMLE board performance at University of Washington in St. Louis School of Medicine found that of 72 students who had previously taken Step 1, 22 used Anki and 35 used Firecracker [30]. Anki use was also explored specifically as a supplement to the anatomy and physiology curriculum at UCF College of Medicine where approximately 70% of 60 surveyed students used Anki [31]. One study showing a positive association between Anki usage and Step 1 performance found that 132 of 201 second-, third-, and fourth-year medical students at the University of North Carolina School of Medicine reported using Anki [22]. Likewise, while the UMMS study investigating student Step 1 preparatory behaviors did not report on Anki, it found that 52 of 274 students used Firecracker [3]. Employing a random effects model yielded an effect summary of 40.75% (21.42%-60.01%) for Anki use and an effect summary of 19.22% (6.50%-31.93%) for Firecracker use (Fig. 2c and d).

Pathoma and Sketchy

Pathoma and Sketchy are both comprehensive medical video libraries that cover high-yield topics; however, they differ in their subject focus. Pathoma's content centers on pathology while Sketchy is primarily dedicated to microbiology and pharmacology. Five studies reported on medical student use of Pathoma and 4 studies reported on medical student use of Sketchy. At MCW, 54% (84/155) and 74% (114/155) of survey respondents used Pathoma and Sketchy as a supplemental education resource, respectively [26]. At UCF College of Medicine, approximately 98% of surveyed students used Pathoma and 80% used Sketchy [24]. Likewise, 247 of 274 surveyed students used Pathoma to prepare for Step 1 at the UMMS [3]. Pathoma and Sketchy use both increased as medical students progressed through medical school, with 32 and 69 of 108 first-year students at UMMS using Pathoma and Sketchy, respectively, while 98 and 88 of 119 third-year students used Pathoma and Sketchy, respectively [28]. Among 290 osteopathic students, 73 used Pathoma and 58 used Sketchy as Step 1/Level 1 resources [32]. Employing a random effects model yielded an effect summary of 63.64% (36.86%-90.42%) for Pathoma use (Fig. 2e).

Alternative Question Banks

The USMLE Kaplan Qbank and USMLE Rx provide students with exam-like test questions and integrated test taking strategies within an interface that mimics the Step 1 exam. Five studies reported on student use of the Kaplan Qbank to prepare for Step 1 and 2 studies reported on student use of the USMLE Rx question bank. The use of the Kaplan Qbank and USMLE Rx ranged across studies. Thirty-three of 274 UMMS students, 4 of 163 UICOM students, 26 of 164 WVU School of Medicine students, and approximately 19% of 48 UCF College of Medicine students reported using the Kaplan Qbank as a Step 1 preparatory resource or a supplement to the medical school curriculum [3, 8, 20, 24]. Likewise, when a cohort of medical students at University of Alabama were provided free 18-month access to the Kaplan Qbank, 179 of 184 students accessed and used the Qbank to varying degrees across preclinical modules [16]. In two separate studies, 70 of 274 students at UMMS used USMLE Rx but only 13 of 108 first-year students and 17 of 119 third-year students reported using USMLE Rx [3, 28].

Osmosis

Four studies reported on student use of Osmosis, an educational platform that integrates a variety of medical content including videos, board style practice questions, flashcards, and reference articles. During the implementation of a new medical educational technology at Lake Erie College of Osteopathic Medicine, which involved granting first-year students complimentary access to Osmosis, 416 of 573 students created an Osmosis account. Similarly, among second-year students who were required to pay for their subscriptions, 151 of 562 students created an account [25]. Likewise, a study at Miller School of Medicine incorporated engagement with Osmosis into the grading structure of their preclinical cardiovascular system course. All 359 students who took the course engaged with Osmosis in some manner. Among these students, 232 completed a survey on their experience with Osmosis, 134 of whom preferred Osmosis to traditional lectures [18]. At the UCF College of Medicine, approximately 52% of 48 students used Osmosis [24] and when exploring the use of Osmosis as medical students progressed through medical school, 56 of 108 first-year students at UMMS used Osmosis while only 13 of 119 third-year students used Osmosis [28].

Goljan Audio Lectures

Three studies reported on student use of Goljan Lectures, a series of audio lectures and podcasts created by Dr. Edward Goljan that cover a range of topics including cellular pathology, systemic pathology, and various organ systems. One UMMS study showed that 145 of 274 students who had taken Step 1 used Goljan Audio [3], while a separate UMMS study showed use of Goljan Audio changed as medical students progressed through medical school with 6 of 108 first-year students and 18 of 119 third-year students using Goljan Audio [28]. Likewise, approximately 25% of 48 UCF College of Medicine students reported using Goljan Audio [24].

Practice Exams

As students embark on their preparations for the USMLE Step 1 exam, it is common for them to utilize practice exams to optimize their readiness for test day. Two prominent providers of such practice exams are the National Board of Medical Examiners (NBME) and UWorld. The NBME offers multiple Comprehensive Basic Science Self Assessments (CBSSA) and UWorld offers UWorld Self Assessment 1 (UWSA1) and UWorld Self Assessment 2 (UWSA2). Among UMMS students, 167 of 274 and 221 of 274 used NBME CBSSA and UWSA1 to prepare for Step 1, respectively [3]. Likewise, in the survey of 466 anonymous Reddit users who had previously taken Step 1, 360 reported using UWSA1, 369 reported using UWSA2, and at least 333 reported using at least one NBME CBSSA [19].

Other Common Commercial Resources

Although they each have been investigated in a limited number of studies, Amboss, Boards and Beyond, and the Board Review Series were also prominent third-party resources used by medical students. Amboss is an online learning platform providing a wide range of study materials and resources to support medical education, Boards and Beyond is a video-based learning platform offering education videos on a range of medical topics, and the Board Review Series is a collection of review books used by medical students for board exam preparation. The UMMS study was the only one to report on Amboss student use, in which 24 of 108 first-year students and 12 of 119 third-year students used Amboss [28]. In this same study, 48 of 108 first-year students and 21 of 119 third-year students were found to use Boards and Beyond [28]. At UCF College of Medicine, approximately 60% and 17% of 48 surveyed students used Boards and Beyond and the Board Review Series, respectively [24]. Likewise, 145 of 164 surveyed students at WVU School of Medicine used the Board Review Series [8].

Discussion

In this systematic review and meta-analysis, we synthesized findings from 19 studies investigating the use of commercial third-party resources by medical students either as a supplement to their medical school curriculum or as preparation resources for board exams. We found that the three most commonly used resources were First Aid, UWorld, and Pathoma. While previous reviews have examined the relationship between commercial resources and USMLE performance [33], this review is the first to comprehensively synthesize information from a diverse range of studies regarding the number of students utilizing each commercial resource as a supplementary aid to their medical curriculum or as a means of preparation for Step exams. As the use of e-learning and commercial resources becomes more common amongst medical students, there is an increasing push for medical educators to further integrate and adapt the use of these resources into the traditional medical school curriculum [28, 34–36]. In order to successfully do so, what is first required is a thorough identification of the resources currently employed by medical students, raising awareness among fellow students, medical faculty, and academic administrators about these valuable tools. This systematic review and meta-analysis successfully achieves this goal, offering a comprehensive overview of the third-party resources most widely used by medical students and thereby laying a solid foundation for future educational enhancements.

Commercial test preparation courses and educational resources have been ingrained in the academic landscape for decades, with one of the first reviews evaluating the efficacy of commercial test preparation resources published in 2004 [37]. As in person courses began to transition towards self-directed learning platforms, the 2000’s and 2010’s witnessed a surge in the availability of online educational platforms, video libraries, and comprehensive question banks catering to the needs of medical students. Prominent examples include the introduction of Anki in 2006, UWorld in 2003, Boards and Beyond in 2014, and Amboss in 2012, among others. Self-directed learning resources are so deeply engrained into medical education that an increasing number of medical students have started to utilize and even rely on these platforms as valuable supplements to their medical curriculum [3, 7, 19, 28, 38, 39]. In fact, a recent study revealed that, on average, students utilized a total of seven distinct study resources while preparing for exams related to various organ-based systems [40]. Although the specific third-party resources that resonate best with each individual student vary based on their unique learning style and preferences, this review has underscored certain resources such as First Aid, UWorld, Pathoma, and Sketchy that have achieved widespread utility across US medical students [41]. Students take into consideration various factors when choosing what third-party resources to use, including cost, peer advice, convenience, and familiarity with the educational modality [28, 42, 43]. First Aid and UWorld are both highly vetted, long-standing medical educational resources utilized as far back as the 1990’s and early 2000’s [44]. Their current popularity can likely be attributed to strong recommendations from peers, the high quality and comprehensive consolidation of high yield information, and their widespread recognition and familiarity within the medical community. Sketchy and Pathoma, on the other hand, are relatively new third-party resources introduced in the early 2010’s [45, 46]. Pathoma’s current success may lie in its multimedia approach that may appeal to a wider variety of student learners [46]. As a result, this resource has gained a strong reputation within the field of medical education. Likewise, Sketchy’s unique and innovative visual and auditory approach likely underpins its extensive popularity among medical students [45]. Additionally, its convenience and easy online accessibility, coupled with strong endorsements from peers, potentially further contribute to its widespread usage among medical students.

While the extent of evidence may vary depending on the specific study design and resource under investigation, the utilization of numerous platforms like Anki, Kaplan Qbank, UWorld, and USMLE Rx, among others, has demonstrated a positive correlation with improved performance in both classroom exams and USMLE board exams [3, 8, 16, 19, 21, 22, 30, 40, 47]. The effectiveness of these resources may be attributed to their strategic utilization of fundamental learning theories, such as interim testing, active learning, and spaced retrieval [48–50]. Question banks offer students a valuable tool to simulate exam conditions, allowing them to practice consolidating their existing knowledge and effectively guiding them in identifying their weaknesses [3, 21, 48]. Likewise, resources like Anki that encourage learning through spaced repetition improves retention of medical knowledge, which is a key aspect of performance on the board exams [22, 30, 49]. A further benefit of third-party resources that may contribute to their role in enhancing board exam performance is their ability to consolidate high-yield information into a single location, enhancing studying efficiency for students [3]. As the popularity of these resources continues to rise, a notable shift has occurred from conventional classroom-based learning to a student self-teaching approach reliant on commercial third-party platforms. Consequently, this transition has widened the gap and fostered a disconnect between students and faculty in terms of comprehending and addressing the learning needs of students [16]. It has been observed that students consistently expand their knowledge through self-directed learning and the utilization of third-party resources, a finding that emphasizes the urgency for medical school processes to prioritize learner-centered education [16]. In light of the widespread utilization of third-party resources documented in the literature, it is imperative for faculty members throughout the US to begin to embrace and integrate these commonly used resources into their classrooms. Medical educators are an indispensable source of knowledge for medical students due to their extensive experience in the field and long-term familiarity with standardized board exams. Medical students and medical educators collaborating together to identify specific third-party resources that students find useful and educators carefully validate can allow third-party resources to be effectively leveraged alongside medical educators' expertise to enhance student knowledge retention, preparation, and performance. In fact, integration of Osmosis into a Cardiovascular Systems course at the University of Miami was met with overwhelmingly positive feedback from students who found it to be immensely beneficial in their learning experience [18].

Given that third-party commercial resources are commonly employed as supplementary materials alongside the formal medical curriculum, students are burdened with additional financial expenses in acquiring these materials. According to a study conducted among osteopathic medical students, the average cost of supplementary board preparation materials was estimated to be US$4,129 [32]. As there is a benefit to incorporating third-party resources into board exam preparation, access to these resources creates a disparity in medical education by providing an unfair advantage to those students who can afford them [26]. This disparity can have a significant downstream impact on an individual student’s medical career as affluent students with better access to financial resources may perform better on medical school exams or national board exams, providing them with an enhanced residency application or future career options. As these resources continue to transition from optional to essential in order to succeed in medical school, students may find themselves compelled to allocate their loan funds towards the purchase of these resources or risk potentially placing themselves at a disadvantage in comparison to their peers. To cultivate an equitable educational environment that prioritizes the needs of every student, it is imperative for medical schools to consider funding the expenses linked to board preparation materials or frequently used supplemental third-party resources such as those presented in this review. By doing so, medical schools can actively contribute to reducing disparities and ensuring equal access to essential learning materials for all students. Notably, it has been shown that when supplemental resources were provided to students free of charge, there was a remarkable surge in their utilization and a downstream positive correlation with enhanced performance in both school and national board exams. For example, when class wide access was provided for the Kaplan Qbank at the University of Alabama School of Medicine, nearly 98% of students used the Qbank to some degree and the number of Qbank questions completed was associated with higher scores on instructor-designed exams, final module grades, and USMLE Step 1 scores [16]. Similarly, at UICOM, an impactful student-led initiative was undertaken to develop a Step 1 Preparation course in which second-year students were provided with First Aid and a six-month subscription to UWorld. Although the course encompassed additional elements, it is noteworthy that there was a significant improvement of 8.82 points in student performance on Step 1 after the implementation of this program [20]. In addition to the out-of-pocket cost associated with the purchase of third-party resources, students must dedicate time and energy to locate, review, and evaluate all the resources available to them on the market. Similarly, students often find themselves overwhelmed by the sheer number of available third-party resources and may attempt to use too many without much success. As these resources integrate into the fabric of medical education, medical faculty may be able to decrease the burden put on medical students by educating them about the advantages and limitations of each third-party resource while also validating the resources that prove to be the most effective for exam preparation. The decision to fund third-party resources raises several financial and ethical considerations that medical educators must carefully evaluate. For instance, it is crucial to weigh the advantages and drawbacks of allocating educational funds to third-party resources, as this may divert funds from other important educational initiatives within each medical school or enhance tuition costs. Additionally, questions arise regarding which resources to fund and whether these decisions should be made collectively by medical educators or individually by medical students. By highlighting the most popular and widely utilized commercial resources among their peers, we hope this review offers both medical students and medical educators a strong starting point as they begin to explore and evaluate potential commercial resources to support medical education.

Limitations

Although our study conducted an exhaustive review of the literature encompassing key terms we deemed relevant, our search strategy may have inadvertently overlooked studies that could have offered additional insights. Additionally, as our review only included studies based in US medical schools, the application of the findings presented in this review cannot be generalized to international medical students. Furthermore, our inclusion criteria focused on publications that explicitly quantified the usage of third-party resources and therefore, we did not include studies that only broadly mentioned their use. While most studies provided precise numbers regarding student usage of each investigated third-party resource, a few studies only presented data in the form of bar charts without accompanying number values. In such instances, our best estimations were made based on the available bar graphs to approximate student usage of specific resources. Furthermore, the results reported in many of the studies included in this review were obtained through self-reported medical student surveys. These optional surveys can be subject to bias due to the inherent motivation of students who choose to complete them. Additionally, the data obtained from these surveys may be unreliable if students inaccurately estimate their usage or engagement with individual third-party resources.

With the exception of one study, the majority of included studies did not report on the specific timing of resource usage across different stages of medical education [28]. Further investigation into this temporal aspect of third-party resource usage is a promising avenue for future research as the patterns of resource utilization can undergo substantial changes from the first year to the fourth year of medical school. Similarly, as our study was limited by the relatively low number of studies included in our meta-analysis for each resource and the short time range in which these studies were conducted, we were unable to conduct a comprehensive analysis of how the use of third-party resources has evolved over time. This is an important avenue for future research as it could reveal which resources have become more or less popular since their inception. Additionally, while this review highlights the most commonly utilized third-party resources by US medical students, it does not explicitly investigate the reasons why these resources are so popular. Future investigation on the specific factors of third-party resources that drive medical student preference could offer valuable insights to medical educators. Lastly, the NBME decided to change Step 1 scoring to a pass/fail format in January 2022. Our search strategy did not yield enough publications to provide a comprehensive analysis of how the use of third-party resources changed following this transition. Future research into how the use of third-party resources has evolved since this decision could offer valuable insights for current medical students and educators.

Conclusions

Over the past two decades, there has been a remarkable surge in the abundance of medical education third-party commercial resources. This review identified First Aid, UWorld, and Pathoma through meta-analysis as the three most commonly used resources by medical students. As the medical education landscape continues to evolve, it is imperative for medical school faculty and administrators to adapt their curriculum to better cater to the changing learning needs of students. This may involve embracing the integration of commercial resources to promote better alignment with student preferences. Likewise, considering the substantial financial and time investment that medical students must allocate towards selecting appropriate commercial resources, it is our hope that the insights derived from this review will inspire medical schools to contemplate implementing systems that provide equitable access to these resources for all students.

Supplementary Information

Below is the link to the electronic supplementary material.

Author Contributions

MRK, SP, and NZ contributed to study conception and design. AH performed the literature search. MRK, GH, DC, RK, and SP performed literature screening, data extraction, and data analysis. MRK, GH, RK, and SP drafted the manuscript. MRK, GH, DC, SP, and NZ critically revised the manuscript. NZ provided supervision. All authors read and approved the final manuscript.

Funding

No funding was received to assist with the preparation of this manuscript.

This researched received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Data Availability

The datasets generated during and analyzed during the current study are available from the corresponding author on reasonable request.

Declarations

Ethics Approval

As this study is a review of previously published literature, no ethics approval is required.

Conflicts of Interest

The Authors declare that there is no conflict of interest.