A Curriculum Ignored? The Usage of Unofficial Commercial and Peer Learning Resources in Undergraduate Medical Education at a Canadian Medical School

Abstract

Medical students often seek educational resources outside of the formal curriculum to support their education. Commercial and student-developed educational resources are readily available, and may supplement or even replace aspects of the formal curriculum. Since this has the potential to impact what students learn and ultimately impact patient care, gaining a better understanding of the prevalence and patterns of use of these resources is of great interest. The purpose of this study, therefore, is to explore the use of non-traditional resources in undergraduate medical education. We surveyed University of Ottawa medical students to determine the usage, rationale, and perceived benefits and drawbacks of these educational resources. The response rate was 57/342 (17%). Of 57 respondents (first year = 21, second year = 34, MD/PhD = 2), 98.2% report using non-traditional resources, with Upper Year Notes, Student Developed Anki decks (a spaced repetition flashcard tool), and Student Developed Question Banks being most common. Although 75% of students reported using official lecture slides and practice tests, they reported spending more of their time (63%) using third-party resources compared to official curriculum offerings. Reported advantages of non-traditional resources listed were faster, more efficient, and easier to understand/search. Disadvantages were difficulty aligning unofficial resources with local exams, determining the level of detail required, and cost. These results suggest a need to examine curriculum design and delivery and further investigate the role of non-traditional resources developed by medical students and the role of medical students as peer educators.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01899-5.

Introduction

In order to become competent physicians, medical students must assimilate a large amount of data. In this technological era, students often seek educational resources to supplement or replace traditional learning materials. Enabled by the rise of the internet, there has been a proliferation of third-party resource options designed to supplement the traditional or formal undergraduate medical education curriculum offered by medical schools. These commercial and medical student–developed resources, which may be labelled as “non-traditional” resources have changed how and what medical students choose to study [1] and claim to be more time efficient in contrast to traditional lectures and textbooks [2]. In addition, the COVID pandemic has led to significant changes to curriculum delivery thus providing an opportunity to reexamine how education is delivered [3].

The use of non-traditional supplemental online learning materials appears widespread. In 2021, a majority (96.1%) of American medical students reported using non-traditional online resources with 35% reporting utilizing this content daily [4]. A separate US study reported that 75.8% of students used non-traditional resources for the majority of their learning [5]. A recent review confirms a large portion of current medical students use commercial-off-the-shelf learning platforms developed for medical education (referred to as MedED-COTS) to prepare for national licensing examinations in North America [6].

Several studies report the utilization of specific non-traditional resource platforms. For example, Anki, a free spaced repetition flashcard tool was utilized in multiple schools [1, 7] with utilization exceeding 80% at some American medical schools [7]. Other platforms such as uWorld, UpToDate, and Online Question Banks have also been reported [6]. Other non-traditional resources include Osmosis, a primarily video-based online library [8], and local student-developed resources such as flashcards [9].

Most of the studies point to these resources being selected and purchased by students themselves [1]. However, in Europe, some faculty have integrated MedED-COTS non-traditional curriculum resources into the formal curriculum. For example, Amboss, a content library and question bank, was formally integrated into a portion of a medical school’s digital learning platform to provide clinical cases and content libraries [10]. Other schools in North America have also reported integrating these MedED-COTS resources into the curriculum, with potential benefits of saving faculty time and improving student performance on national licensing exams [1, 11].

So, it appears that the majority of students are using multiple non-traditional resources, but there are only a few studies that actually provide details regarding patterns of use. For example, the very broad AAMC survey in the USA [4] reports the high prevalence of use but does not delineate types of resources used. Many studies are focused on the use of these resources to prepare for national licensing examinations but these and other studies do not fully explore how and when these resources are used [6, 12]. Our study seeks to bridge this gap through exploration of not only the “what and how many” of non-traditional resource utilization, but to begin to explore “how and why” students use these resources beyond preparing for national examinations.

In summary, despite the seemingly widespread use of these “unofficial” resources, hereafter referred to as “non-traditional resources,” in undergraduate medical education, we have limited understanding of how widespread the use of these resources is, what resources are used, and how they are integrated into the existing curriculum, including the perceived benefits and risks from the students’ perspective. Since these may impact what our students learn, ultimately impacting patient care, developing a better understanding of this phenomenon is important.

The purpose of this study is to explore the use of non-traditional resources in undergraduate medical education. Specifically, we are interested in exploring the use of non-traditional curricular resources, utilization patterns, and the rationale given for using these resources, as well as the perceived benefits and drawbacks, in comparison to the usage of more traditional curricular resources offered such as lectures. Since what and how students learn has critical implications for patient care, this study addresses an important issue.

Methods

For the purposes of this study, “non-traditional resources” refers to any materials that are not developed by the medical schools as part of the formal curriculum. They may include local peer-created resources or commercial products. MedED-COTS are a form or non-traditional resource that are commercially developed.

Survey Development

The online survey was developed based on the recommendations from the Association for Medical Education in Europe guide [13]. A literature review (PubMed) informed the survey, as well as the gray literature including the Canadian Federation of Medical Students website’s list of subsidized resources [14], and the knowledge from three current medical student authors.

The first draft of the survey was developed by DM and revised iteratively by the other co-authors (AK, ML, SHM). Subsequently, the survey was piloted with three third-year medical students to ensure clarity and that the questions were interpreted as intended. The questionnaire gathered demographic data such as age, gender, language of study (French or English), and prior educational background.

It then explored current use of both official (lecture attendance) and non-traditional curricular resources and advantages and disadvantages of each, as well as current study habits including impact of non-traditional resources (time spent studying, and satisfaction with current method).

For the purposes of this study, attendance for the formal curriculum focused on optional non-mandatory activities, since mandatory lectures are required for all students. The final questionnaire included 24 questions and took approximately 10 min to complete in the pilot (Appendix 1).

Setting and Participants

The University of Ottawa is a bilingual (French/English) medical school and has approximately 166 students in each year with a breakdown of ~ 118 studying in English and ~ 48 studying in French. In English-speaking Canada, students enter medical school after completing at least 2 years of undergraduate studies. The medical school training is 3-4 years in total with 1.5–2 years dedicated to primarily class work (pre-clerkship) followed by clinical rotations (clerkship). Since pre-clerkship has a clearly defined standardized curriculum with multiple resources (slides, suggested readings) provided from faculty, the students from these years were the focus of our inquiry. At the University of Ottawa, the formal curriculum includes both mandatory lectures and non-mandatory lectures. Optional traditional style core curriculum lectures exist alongside mandatory twice weekly small group Case Based Learning, large group Team-Based Learning, small and large group Physical Exam teaching, and Social Determinants of Health lectures. A typical week would have a total of ~ 25 h of instruction, with ~ 15 h of optional learning activities included in that total. Preclerks are tested midway and at the end of each 10-week block on all content utilizing a combination of multiple choice and written answer core knowledge and clinical decision-making questions mirroring the Medical Council of Canada Qualifying Exam Part 1. Social determinants of health content is either tested directly or via an essay. While occasionally mentioned by individual faculty, there was no organized promotion of non-traditional resources (such as Anki, Amboss, and others) by the medical school.. During COVID-19 restrictions (March 2020–June 2022), the MD2024 completed preclerkship entirely in a hybrid online model via web conferencing software (lectures, some small-group activities and some physical exam teaching) and in-person (some small group and physical exam teaching although in 2020–2021 these were mostly virtual). The MD2025 cohort experienced the same but with more in-person components. Neither class had in-person traditional lectures offered during this study’s time-span.

All medical students in years 1–2 (pre-clerkship) and students in the PhD phase of the combined MD-PhD program who had just completed year 2 were approached in the weekly class email through the local student association, the Aesculapian Society. A reminder was sent on 4 occasions, 1 week apart. Participants were entered into a draw for a $50 gift card upon completion of the survey. The survey responses were gathered between April and May 2022, during the end of the academic year for both cohorts.

Data Analysis

Descriptive statistics were analyzed using the R statistics platform [15]. Free-form response questions were reviewed independently by three co-authors (DM, ML, AK) and discussed during several team meetings to reach consensus on the common themes following thematic analysis principles [16]. Ethics approval was provided by the University of Ottawa research Health Sciences and Science Research Ethics Board and approved by the Vice Dean Undergraduate Medical Education.

Results

In total, there were 57 responses (first years = 21, second years = 34, PhD = 2) out of 342 eligible pre-clerkship students for a 17% response rate. The majority, 72% (n = 41) were in the English stream and 28% (n = 16) were in the French stream, matching the proportion in the student population. Most respondents identified as female, 70% (n = 40) and 30% (n = 17) identified as male. This is broadly reflective of the cohorts at the University of Ottawa who are 55–59% female [17]. Most students reported a biomedical (n = 19, 23%) or health science (n = 16, 28%) background. There were no discernable differences between responses due to these identifying factors.

Use of Non-traditional Curriculum Resources

As demonstrated in detail in Table 1, 56 (98%) report using non-traditional curriculum resources. The most popular resources were mostly student-developed, such as Upper Year Anki decks and Upper Year Notes (these are comprehensive resources covering all aspects of the curriculum created by medical students in previous years), as well as Upper Year Question Banks (a student run project at the University of Ottawa to create questions for all areas of the curriculum). MedED-COTS resources such as Osmosis, Amboss, and UptoDate were next in popularity.

Table 1.

Breakdown of all non-traditional resources including MedED-COTS, used by students (total n = 57). Note all resources are in the English language. Resources with fewer than 10% total utilization are grouped into classes based on the source. Note that some resources may be based on primarily one geographic location while allowing students or physicians from different countries to also contribute. Costs do not reflect frequent discounts and were obtained using an American proxy internet connection in April 2023

Resource	Number of students using (n) the resource	Percentage of students utilizing the resource (%)	Source/developer	Cost per year ($ USD 2023)	Description
Upper Year Anki decks (local)	33	58	Multiple student developed	Free	Premade flashcard deck to cover content found in the official curriculum
Upper Year Notes (local)	32	56	Sole student developed	Free	Note sets created by a single previous student
Upper Year Question Banks (local)	27	47	Multiple student developed	Free	Collaborative question banks created by previous students from University of Ottawa
Osmosis (American)	25	44	Faculty physicians	378	Online videos, articles, and questions
Amboss (German, in English language)	18	32	Faculty physicians	429	Online videos, articles, and questions
UpToDate (American)	16	28	Faculty physicians	219	Clinical decision-making tool
Self-made Anki decks (local)	13	23	Self-made	Free	Self-made flashcards
Step 1 Med Bullets (American)	13	23	Students, residents, and faculty physicians	250	Question bank
External Anki Decks (e.g., Lightyear) (American)	12	21	Multiple student developed	Free	Premade flashcard decks typically focusing on USMLE content
Merck Manuals (American)	12	21	Faculty physicians	Free	Medical information source
Boards & Beyond OnlineMedEd Bate’s Guide to Physical Examinations LearningRadiology (All American)	6–10	10–20%	Faculty physicians	249 (Boards & Beyond)	Video libraries and question banks
				429 (OnlineMedEd)	Online videos and summaries with some physical resources
				135 (Bates’s)	Book
				50 (LearningRadiology)	Website and book
Toronto Notes (Canadian)	9	16	Faculty physicians and students	150	Medical reference and clinical handbook
uWorld Sketchy First Aid for Step 1 Visible Body Stanford Medicine 25 Firecracker Life in the Fast Lane (American except Life in the Fast Lane which is Australasian)	3–5	5–9	Faculty physicians	559 (uWorld)	Question banks for USMLE
				399 (Sketchy)	Medical infographics
				50 (First Aid for Step 1)	Reference book
				Free (Visible Body)	Anatomy website
				Free (Stanford Medicine 25)	Physical exam skills videos and written content
				199 (Firecracker)	Online medical reference and online tutor
				Free (Life in the Fast Lane)	ECG and imaging cases and articles
Orthobullets (American)	5	9	Students, residents, and faculty physicians	200	Medical reference, question banks, and video library
“Question/Case of the Day” style email newsletters or social media pages Podcasts (Variable nationality)	3–5	5–9	Variable, faculty physicians typically	Usually free
YouTube (Variable nationality)	1–2	2–4	Variable, students, faculty physicians	Free	Educational videos
Lecturio (German, in the English language) Med Mastery (Austrian) Healio (American) Teaching Medicine (Canadian) Picmonics (American) Pathoma (American) Doctors in Training (American) MEDSKL (Canadian) Giblib (American) Dr. Najeeb Lectures (American)	2	4	Faculty physicians and medical students	299.88 (Lecturio)	Video library and clinical case questions
				468–564 (MedMastery)	Online medical reference library
				Variable (Healio)	Online medical reference library
				Free (Teaching Medicine)	Online cases
				180 (Picmonics)	Medical infographics
				99.95 (Pathoma)	Pathology images and reference teaching
				550 (Doctors in Training)	Online medical reference library
				Free (MEDSKL)	Online medical reference library
				360 (Giblib)	Online medical reference library
				99 (Dr. Najeeb’s Lectures)	Online lectures
Calgary Blackbook (Canadian) Geeky Medics (British)	1	2	Multiple students, resident physicians	Free, paid content available for GeekyMedics	Clinical flowcharts (Calgary Blackbook) Physical exam teaching and cases (GeekyMedics)

Students generally learned about resources from other medical students, as shown in Table 2.

Table 2.

Who or where students learned about the resources they currently use. Upper year students are medical students at least 1 year ahead of the student answering the survey (n = 57)

Source of recommendation for resource	Count (n)	Percentage (%)
Upper year students	53	93
Classmates	49	86
Searching online	23	40
Student-compiled resource list	16	28
Online advertisements	5	9
Faculty-compiled resource list	2	4
Faculty members	1	2

Students provided both selected responses and free-text responses regarding perceived advantages and disadvantages of non-traditional/external resources, shown below. The quantitative and qualitative data are presented according to the themes of advantages and disadvantages. Of note is that almost all students utilize many different resources limiting the ability to generalize about any specific resource.

Advantages of Non-traditional Resources

For selected response questions, 55 (96%) of students feel that the usage of non-traditional resources is an efficient use of their time and 53 (93%) report that non-traditional resources are somewhat or significantly more efficient than the official curriculum. This data, and all quantitative data presented, is attached in Appendix 2.

Free-text responses demonstrated several themes related to advantages of non-traditional resources. This is encapsulated by one student who describes non-traditional resources collectively as “Faster, more efficient, and easier to understand/search” (student 1). These three core themes were frequently mentioned together.

Non-traditional resources were touted as more time efficient by almost all of the students. The high quality of non-traditional material led to time savings by limiting the need to make notes during lectures and availability anytime, such as to help prepare the week before formal lectures. Ready availability on electronic devices provided flexibility in when and at what pace to study as stated by the student; the ability to “study using these resources (student made & Osmosis) to go at my own speed” (student 8) and “when I’m in a time crunch Anki is in a format that I can easily use on the go” (student 42), such as when waiting in line, are illustrative examples. Others expressed the time savings “It’s (Anki) also effective and I don’t feel like I spend an insane amount of time studying” (student 17).

The highly organized format and ease of use of non-traditional resources on the whole was noted by many. “Presented in a friendly format, asynchronous, self-paced, organized for optimal learning” (student 43). Students also suggested the consistent structuring of non-traditional resources was an advantage “It’s easier to understand and presented in the same sort of ways consistently” (student 38).

The resources offered flexibility along a spectrum of being very comprehensive covering all aspects of a topic, to providing more “bite sized” and easy to assimilate small amounts of data (student 19).

The interactive nature of non-traditional resources was noted. This included a well-developed number of test questions to practice as well as individualized repetition strategies. The usage of Anki, a spaced repetition flashcard tool, was specifically noted as a “better learning strategy” (student 28).

Notably, Anki’s use of a scheduling algorithm was identified as a positive as it “motivates me to work every day.” Interactive video formats were also praised repeatedly (student 30), as were integrated mnemonics (student 49).

Students described the non-traditional resources as a means to address perceived shortfalls of the formal curriculum. Students frequently expressed frustration with the disconnect between tested concepts and lectures. “I would estimate that I spend 98% of my study time using non-traditional resources. This is because I do not find the resources [multiple student resources and MedED-COTS] provided by the school to be useful or sufficient to succeed” (student 14). This dissatisfaction extends to the lecturer’s content preparation, with one student noting that online resources (specifically utilizing UpToDate and OnlineMedEd) “are more up to date, usually” (student 25).

For some students, non-traditional resources were used to focus for the future. “While studying medicine it’s no longer about simply doing well for a test but rather preparing for clerkship. Naturally our curriculum will vary slightly from other schools, so I think it’s important to consume info from third-party resources that gives you a bigger picture of what you may be expected to know during clerkship (even if it wasn’t covered in class)” (student 22). This idea that these resources are better at providing clinically relevant information was mentioned repeatedly.

Disadvantages of Non-traditional Resources

Several key disadvantages were consistently identified. The cost of MedED-COTS resources was highlighted in the free-text responses. The amount of money (CAN) spent for MedED-COTS resources varied with the following breakdown: $0 spent by 11 students (19%); $0–50 spent by 11 students (19%); $51–200 spent by 14 students (25%) and $201–300 spent by 14 students (25%), and 7 students spending more than $301 (12%). Comments from participants were noted, “I have to pay to use them (Osmosis, Boards & Beyond, Amboss), on top of already very high tuition fees” (student 32). However, others note this and still view them as worth it, “I would say cost is the biggest factor, but given how much we invest in our med school education, I’d say the resources (Osmosis, Amboss) I have been using are more than worth the money” (student 22).

The second most prominent disadvantage was a sense that the resources do not align with the official curriculum in relation to the tested content. As plainly stated, “[I] have to map it (Anking and Amboss) onto the curriculum myself” (student 5). As one individual notes, “I’m never entirely sure if they cover the exact same content at the level expected. To counter this, I simply go beyond what I think we are expected to know” (student 1). Some view this over-studying requirement as “overwhelming” (student 11). Interestingly, some report appreciating this over-studying; “I don’t see disadvantages other than sometimes they give information not from the objectives but still very important” (student 4).

Concerns about errors are prominent. One form is the usage of, typically American, resources that feature different information “the guidelines may not be up to date or consistent with local practices” (student 9).

Some less common concerns included questionable clinical relevance. As one individual (student 16) noted “The disadvantages include that the notes (Upper Year Student Notes) may not be as clinically relevant to things like CDMQ [Clinical Decision-Making Questions], they provide more memorization-based learning rather than applicability or usefulness in clerkship for the future…Great for test writing, less so for long-term memory.” Several students also expressed concern that if they missed the formal lecture, they would miss important details, could not ask questions, or miss a “clinical pearl” (student 41).

Sheer quantity is the final negative, with some noting, “There are way too many Anki slides, and I spend a lot of time suspending them” (student 26).

Use of Formal Curriculum and Resources

Overall, lecture attendance is listed in Table 3 where it can be seen that 26 (46%) of students attended more than 90% of these lectures and 15 (26%) reporting skipping all voluntary lectures. Of the 26% who do not attend, 60% (9 students) do not use recordings either.

Table 3.

Lecture attendance for non-mandatory lectures overall and by year of study in percentages

	All (%)	First year (%)	Second year (%)	Anglophone (%)	Francophone (%)
I attend greater than 90% of all non-mandatory lectures in real time	45.62	40.19	48.71	39.02	62.5
I attend 60–89% of all non-mandatory lectures in real time	19.3	27.27	14.29	24.39	6.25
I attend less than 60% of all non-mandatory lectures in real time	8.77	9.09	8.57	9.76	6.25
I do not attend non-mandatory lectures in real time	26.32	22.73	28.57	26.83	25

Usage of official uOttawa resources is shown in Table 4 with 43 (75%) using lecture slides, and official practice tests. The Student Mentorship Center slides were the third most common officially endorsed resource with 42 participants (74%). Twenty-one students (37%) used their own notes.

Table 4.

Breakdown of official uOttawa resources and usage

Resource name	Number of students using this resource (n)	Percentage (%) of all students using resource
Lecture slides	43	75
Official uOttawa practice tests	43	75
Student Mentorship Center slides (student-created slides summarizing the week of lectures)	42	74
Official uOttawa websites (e.g., Brightspace)	31	54
Self-made Mind Maps or flowcharts/diagrams based on the official curriculum	24	42
Own lecture notes	21	37
Textbooks or readings	9	16
None (no official resources used)	2	3.5

Self-reported Study Time and Time Spent Using Non-traditional Resources

For all students, the average study time was 38 h per week, including time spent in lectures. The range was 5–100 h. The mean weekly study time was 33 h for second year, and 46 h for first years. The English stream spent 37.4 h and the French stream spent 39.9 h weekly. Due to the overwhelming utilization of non-traditional curriculum resources (98% utilization), no comparison in total time spent studying between the two groups was possible.

On average, pre-clerkship students reported spending 63% of their study time using non-traditional resources, and 37% using official curriculum resources. The English stream spent a mean of 57% and the French stream spent 77% of their study time utilizing non-traditional resources respectively.

Just over half of students (56%, n = 32) in both streams report no desire to write the USMLE Step 1 exam, 32% are undecided (n = 18), and 12% are planning on writing it (n = 7).

Satisfaction with current studying methods was generally positive, with 40 (70%) reporting they were satisfied. This was 28 (82%) for second years and 12 (57%) for first years.

Discussion

This study set out to explore the use of non-traditional resources in undergraduate medical education. Our results demonstrate that the vast majority (98%) of students use non-traditional resources, with 63% of study time being spent on these resources with generally high satisfaction (70%). Students used peer-generated and MedED-COTS products, and they selected resources based on word of mouth from near-peers, echoing earlier findings [12]. These findings lend support to previous research that found very high levels of use in US medical schools [4, 7]. Our data from one Canadian medical school seems to lend further support to this very high non-traditional resource use. Our results expand on previous work by demonstrating that resources developed by students are also popular alongside MedED-COTS products, at least in our context. This contrasts with previous studies where student-made flashcards were less popular (31%) than MedED-COTS (49%) flashcards [1], and student-interactive resources were never used by 39.5% of students in comparison to online never being used by 8.98% [11].

Peer-assisted learning is well established in medical education [18] and this represents a new area for education resources, as students are creating their own resources such as Anki flashcards [7, 9]. This could be due to the perception of local student-made resources being “high yield” for examinations. Prior studies of clerks [19] noted clerks used medical apps such as UpToDate for reference purposes, improving knowledge/test taking, and saw accessibility and interactivity as the major benefits. Motivation for using these resources has largely focused on USMLE Step 1 scores as a motivating factor in the USA [20], which is not applicable in this Canadian context where only 12% of students indicated strong interest in writing the test. Despite this, other resources targeting USMLE content such as Osmosis (44%) and Amboss (32%) were popular with Canadian students in our sample. Past work providing a single non-traditional resource found that convenience and ease of use were important as motivating factors for that resource [21]. With the now pass/fail USMLE Step 1 having less of an impact on residency matching, the importance of examining motivation aside from national licensing examinations grows in importance as students seeking resources to improve clinical performance instead of exam scores represents a different objective.

Another significant finding is that a quarter of students do not attend lectures or use faculty-generated lecture slides at all, for non-mandatory lectures. Most students in the free-form and quantitative responses (93%) viewed non-traditional curriculum resources as more efficient than the official curriculum; this may be one reason for the low rate of attendance. Previous work has found that 14.38% of medical and dental students never used instructor-guided learning resources to study physiology, while the vast majority (91.02%) used online resources [11]. A 2021 survey at an American medical school found that 84% used lectures, and the same percentage (84%) used the most popular MedED-COTS resource, Boards & Beyond [22]. Lecture attendance patterns with (74%) attending lectures in our study are within range of other medical schools, with 73.8% using instructor-guider learning resources at least half of the time [11].

Several types of non-traditional resources were used. Student-developed resources such as upper year note sets, question banks, and Anki decks appeared to be used as frequently (58% for the most popular) by our respondents as did MedED-COTS resources (44% for the most popular). In particular, the Anki usage (58% for local student-made decks in our study) is very similar to other reported studies of patterns of making Anki decks for their local curriculum with a result of 57.4% [7]. Utilization of student-made resources is similar to past studies where 56% of students used student-generated resources at least weekly [12]. In our study, Anki was used for multiple purposes: for local curriculum review, and to prepare for future examinations, as previously noted in American contexts [7].

There is evidence to support the validity of student-developed resources, for example, usage of student-made Anki flashcards but not MedED-COTS Firecracker flashcards, was positively correlated with USMLE Step 1 performance [1].

Evidently, students are developing and seeking out their own resources [9, 22]. Non-traditional resource use, therefore, may support self-regulated learning (SRL) [23].

SRL is a cyclical process where learners actively control their thoughts, actions, and motivation to achieve their goals [23]. Several models of SRL exist but most suggest that the student is responsible for setting, planning, executing, monitoring, and reflecting on their learning goals. Here we observed pre-clerkship medical students setting goals, frequently related to medical school exam performance in this study, which then drives the utilization of non-traditional resources in a bid to better their exam performance with improved efficiency. Students then self-monitor the effectiveness of these resources via their exam performance relative to the time investment. In a study exploring the use of learning plans to support SLR in clerkship, web-based non-traditional resources were an integral resource for students [24].

In our study, students described several benefits to using these resources. Several factors were inter-related. Non-traditional resources on the whole were considered more efficient. This was related to several features: the actual format, with students noting that non-traditional resources have easier to use interfaces and present information in easier to understand formats, the ability to access from anywhere/anytime, and the testing and spaced repetition methods to improve retention. All of these features have previously been noted [21, 25]. In addition, some studies described reduced exam anxiety with the use of Anki [26], or other online resources [25]. Our participants did not share reduced anxiety as a benefit but were not focused on use for national exams.

The interactive nature of these resources is also something that was frequently noted as an advantage, with the actual platform’s utilization of spaced repetition (Anki), or the abundance of questions to test knowledge, being noted. Students also identified the benefits of gaining extra information not in the traditional curriculum and allowing for improved clinical performance. This is in contrast to the American focus on USMLE performance [6], and the indication that pre-clerkship students are thinking of clinical performance, something previously seen in clerks [19]. In part, this could represent future oriented studying or the increased focus on clinical performance over standardized test scores for residency selection in Canada.

Previous studies have sought student opinions regarding cost, moreso for MedED-COTS. Although a concern, with costs ranging from $0 to $300 + , most students felt it was worth the investment. Notably, this is far less than what the typical American student spends on MedED-COTS resources of $4129 [27] which may contribute to feeling the cost was worth the investment.

Non-traditional resources are used extensively by medical students and may have several advantages for both students and faculty. We have listed several advantages for students, yet faculty could benefit as well, for example from question banks that are readily made, thus saving faculty time [8].

We certainly would support the suggestion from a recent BEME guide [6] that faculty formally consider integrating these resources into the formal curriculum as was already seen in a German medical school [10].

Limitations

This study has several limitations. First, the sample size represents a limitation of generalizability. Second, there is the potential for availability bias where individuals motivated to respond are more likely to be motivated to seek out non-traditional resources. However, our findings generally support current findings in the literature. Lastly, the findings may not generalize to other medical schools with different curriculum designs.

Conclusion

The finding of overwhelming usage of MedED-COTS and student-developed non-traditional resources, while a sizable minority of students report not using lecture slides, points to a need to rethink curricular design. If students are rejecting faculty-developed resources in favor of student-developed or MedED-COTS resources, this represents a mismatch that must be addressed. Is it time to “rethink our current curricular offerings” as past commentaries have suggested? This is especially important given both the cost in terms of faculty time and student payment for commercial resources. A portion of university resources might wisely be directed to co-construction of learning materials by students and faculty together, or alternatively, faculty could review and approve currently existing non-traditional resources. Medical students are evidently invested and interested in participating actively in their own learning and faculty can play a valuable role in guiding students towards sound educational principles as they create their own learning resources.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Appendices

Appendix 1. The Survey Instrument

Attached as supplemental file.

Appendix 2. Survey Quantitative Data for Time Efficiency

Commercial and Community Third-Party Non-Curricular Resource Usage in Pre-clerkship Undergraduate Medical Education
Do you feel that your usage of third-party resources is an efficient use of your time?
Answer Choices	All Responses		First Year Medical Student Responses		Second Year Medical Student Responses
Yes, it is significantly more efficient than the official curriculum	54.39%	31	50.00%	11	57.14%	20
Yes, it is somewhat more efficient than the official curriculum	38.60%	22	45.45%	10	34.29%	12
Yes, it is as efficient as the official curriculum	3.51%	2	4.55%	1	2.86%	1
No, it is less efficient than the official curriculum	3.51%	2	0.00%	0	5.71%	2
Other (please specify or clarify your answers if needed)		5		3		2

Author Contribution

Donovan Makus: all aspects, writing — original draft. Project administration. Anshu Kashyap: all aspects. Mark Labib: methodology, validation, formal analysis, investigation, writing — review and editing Susan Humphrey-Murto: all aspects, supervision.

Funding

This work was supported by an Ontario Medical Student’s Association (OMSA) Medical Student Educational Research Grant (MSERG).

Data Availability

Data is available upon reasonable request from the corresponding author.

Declarations

Ethical Approval

The questionnaire and methodology for this study was approved by the University of Ottawa Research Ethics Board [H-02–22-7616] and the Vice Dean — Undergraduate Medicine in the Faculty of Medicine. The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Informed Consent

Participants provided consent when they agreed to complete the survey instrument.

Conflict of Interest

The authors declare no conflict of interests.