Implication Statement
The development of multiple-choice questions (MCQs) for undergraduate medical education study purposes is resource intensive. Commercially available question banks are typically expensive, only available in English, and may not be aligned with medical school learning objectives. Here, we introduce The Ottawa Question Bank: a student-led, bilingual study resource curated to a Canadian undergraduate medicine curriculum (www.theottawaquestionbank.ca). In total, 205 medical students wrote and edited 4438 original MCQs linked to objectives from the University of Ottawa undergraduate medical education curriculum. The project has received positive feedback from both developers and users. Our experience suggests that involving medical students in MCQ development is feasible and can result in the rapid creation of a low-cost, high-quality study resource curated to a program’s learning objectives. The platform outlined here can be used as a model for other medical schools and professional degree programs to develop their own question banks, including pharmacy, dentistry, nursing, and physiotherapy. Interested programs are encouraged to contact our team for collaborative opportunities.
Énoncé des implications de la recherche
L'élaboration de questions à choix multiples (QCM) dans le cadre de l'enseignement médical de premier cycle exige beaucoup de ressources. Les banques de questions disponibles dans le commerce sont généralement coûteuses, disponibles uniquement en anglais et ne correspondre pas forcément aux objectifs d'apprentissage des facultés de médecine. Nous présentons ici la Banque de questions d'Ottawa : une ressource d'étude bilingue dirigée par des étudiants et adaptée à un programme d'études de médecine de premier cycle au Canada (www.theottawaquestionbank.ca). Au total, 205 étudiants en médecine ont rédigé et édité 4438 QCM originaux liés aux objectifs du programme d'enseignement médical de premier cycle de l'Université d'Ottawa. Le projet a reçu des commentaires positifs de la part des développeurs et des utilisateurs. Notre expérience suggère qu'il est possible d'impliquer des étudiants en médecine dans le développement de QCM et de créer rapidement une ressource d'étude peu coûteuse et de haute qualité, adaptée aux objectifs d'apprentissage d'un programme. La plateforme décrite ici peut servir de modèle à d'autres facultés de médecine et programmes professionnels pour développer leurs propres banques de questions, y compris la pharmacie, l'odontologie, les soins infirmiers et la physiothérapie. Les programmes intéressés sont encouragés à contacter notre équipe pour des opportunités de collaboration.
Introduction
Multiple-choice questions (MCQs) continue to be a widely-used examination modality to assess medical trainee competency worldwide.1 To assess their knowledge, many medical students use faculty-supplied formative questions or commercially available item banks.2 However, question development is a resource intensive endeavor, and third-party resources are often expensive, unilingual, and are not necessarily tailored to the learning objectives of the Canadian undergraduate medical education (UGME) curriculum.3,4 Additionally, the exercise of MCQ development has been found to increase knowledge comprehension in both classroom and clinical settings.5 There is, therefore, a growing need for bilingual, affordable, high-quality MCQ-based study resources curated to the Canadian UGME curriculum, and current medical students may represent an untapped resource to help fill this need.
To address this need, we oversaw the development and implementation of The Ottawa Question Bank (www.theottawaquestionbank.ca). This was a crowdsourced initiative whereby 205 medical students from the University of Ottawa 2022, 2023, and 2024 cohorts wrote, edited, and reviewed 4438 original MCQs all linked to at least one learning objective from the University of Ottawa’s UGME curriculum, all of which are available in both English and French.
Innovation
The entire development of the question bank, including writing, editing, and French translation occurred over two 10-week periods during summer 2020 and 2021. Recruited medical students signed a non-disclosure form to ensure the integrity of existing faculty examination questions. They self-selected their preferred subject areas and were provided with a template to guide question development. To help standardize question quality, recruited medical students were asked to complete the “Primer on Writing or Editing Quality MCQ Items,” a virtual one-hour module prepared by McMaster University
(https://machealth.ca/programs/mcq_and_cdm_prep/).
Medical students were given a five-week timeline to submit their MCQs. Once MCQs were submitted to the oversight team, all MCQ writers were provided with editing assignments to be completed within a three-week period. Several medical students were selected as “Chief Editors” to oversee the editing process of a specific subject matter area. Bilingual medical students were recruited to review French translations performed by DeepL (www.deepl.com) over a two-week period. DeepL was the selected machine translator since its translation quality has been shown to be superior to other services, including Google Translate.6 The overall development effort was almost entirely student-run; faculty members were not directly involved in the development, editing or translation of the submitted questions, however one faculty member did serve in an advisory role. Completed MCQs were categorized by subject matter area and offered to the incoming undergraduate medical school cohort for a nominal fee, with all proceeds donated to a local charity. Following its original rollout, The Ottawa Question Bank has been digitized and is available online
Evaluation
The Office of Research Ethics and Integrity at the University of Ottawa Research Ethics Board (REB) gave us an exemption to perform the initial evaluation of this resource. We developed a survey to evaluate the preliminary perceptions and experiences of MCQ writers and users of The Ottawa Question Bank. A total of 64 medical students responded to the survey, representing a response rate of 19% (64/340). The majority of respondents believed that using this resource improved their content comprehension, reduced anxiety, helped to prepare for formal examinations, and increased examination grades (Table 1). MCQ writers also reported that their involvement in the project facilitated their review and consolidation of content (data not shown). The cost to develop the question bank was minimal; all students volunteered their time to the project. For translation, we purchased a DeepL Pro account for CAD$75.00.
Table 1.
Early user perceptions of The Ottawa Question Bank. Sample size: N=64.
| Strongly disagree | Disagree | Neutral | Agree | Strongly Agree | ||||||
|---|---|---|---|---|---|---|---|---|---|---|
| The Question Bank: | N | % | n | % | n | % | n | % | n | % |
| Was easy to use | 2 | 3 | 0 | 0 | 3 | 5 | 19 | 30 | 40 | 63 |
| Improved my comprehension | 2 | 3 | 2 | 3 | 10 | 16 | 27 | 43 | 22 | 35 |
| Helped me prepare for exams | 1 | 2 | 2 | 3 | 5 | 8 | 22 | 34 | 34 | 53 |
| Eased my anxiety about exams | 3 | 5 | 6 | 9 | 13 | 20 | 18 | 28 | 24 | 38 |
| Was similar difficulty to exam | 2 | 3 | 15 | 23 | 17 | 27 | 21 | 33 | 9 | 14 |
Future directions
Our experience suggests that a large volume of study questions can be developed by crowdsourcing medical student efforts, and that medical students benefit from their involvement.
Acknowledging that every undergraduate medical program is unique, we are excited to invite medical students across Canada to submit new questions and review existing items. This would contribute to the expansion and utility of the current resource. The platform outlined here can also be used as a model to develop study questions across other professional degree programs, including pharmacy, dentistry, nursing, and physiotherapy. By working closely with interested programs, our team can provide guidance to inform similar initiatives, and can upload questions onto our existing online platform for easier accessibility and shareability. Interested programs are invited to contact the corresponding author for further information.
Acknowledgements
The authors would like to thank all the medical student writers and editors from the University of Ottawa 2022, 2023, and 2024 cohorts who contributed to The Ottawa Question Bank. For more information, please visit www.theottawaquestionbank.ca.
Funding Statement
Funding:
BJMB is supported by a Frederick Banting and Charles Best Canada Graduate Scholarship (Doctoral Award) from the Canadian Institutes of Health Research (CIHR; FRN: FBD-175922).
Conflicts of Interest
The authors have no conflicts of interest to declare.
Edited by
Sarah Ibrahim (section editor); Anita Acai (senior section editor); Marcel D’Eon (editor-in-chief).
References
- 1.Epstein RM. Assessment in medical education. N Engl J Med. 2007;356(4):387-96. 10.1056/NEJMra054784. [DOI] [PubMed] [Google Scholar]
- 2.Wynter L, Burgess A, Kalman E, Heron JE, Bleasel J. Medical students: what educational resources are they using? BMC Med Educ. 2019;19(1):36. 10.1186/s12909-019-1462-9 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Ripp K, Braun L. Race/ethnicity in medical education: an analysis of a question bank for step 1 of the United States medical licensing examination. Teach Learn Med. 2017;29(2):115-22. 10.1080/10401334.2016.1268056 [DOI] [PubMed] [Google Scholar]
- 4.London DA, Kwon R, Atluru A, Maurer K, Ben-Ari R, Schaff PB. More on wow USMLE Step 1 scores are challenging academic medicine. Acad Med. 2016;91(5):609-10. 10.1097/acm.0000000000001159 [DOI] [PubMed] [Google Scholar]
- 5.Foos PW, Mora JJ, Tkacz S. Student study techniques and the generation effect. J Educ Psychol. 1994;86(4):567-76. 10.1037/0022-0663.86.4.567 [DOI] [Google Scholar]
- 6.Yulianto A, Supriatnaningsih R. A quantitative evaluation of close-language pair translation (French to English). AJELP. 2021;9(2):109-27. 10.37134/ajelp.vol9.2.9.2021 [DOI] [Google Scholar]