Student-Written Multiple-Choice Questions—a Practical and Educational Approach

Abstract

The purpose of our student-led project was to fulfill junior medical students’ demand for instructive, curriculum-specific practice questions while providing a learning experience and teaching opportunity for participating senior students. Eleven second-year students were taught how to write high-quality multiple-choice questions through an interactive workshop. Subsequently, they were instructed to write questions with detailed explanations for their assigned lecture topics. Thirty-four student-written and faculty-reviewed questions were combined with 16 purely faculty-written questions to create a 50-question exam. No significant difference was found in question difficulty between the student-written (79.5%) and faculty-written (84.0%) questions (p = 0.37). The discrimination index and point biserial correlation were higher for student-written (0.29, 0.32) vs. faculty-written (0.17, 0.25) questions (p < .01, < .05). The test-takers learned key course topics, while the test-writers reviewed key first-year objectives and refined their test-taking strategies. The project provided a model for feasibly developing comprehensive, high-quality, and curriculum-specific questions.

Background

Multiple-choice question exams are found ubiquitously throughout a medical trainee’s career as they can easily, objectively, and reliably assess medical knowledge and judgment [1]. An important and often unrecognized benefit of the multiple-choice question is its usefulness in learning and reviewing medicine [2]. In order to prepare for the various exams, students routinely utilize question banks. The educational benefit is reaped not only from answering the questions themselves, but also from reviewing the feedback [3]. Students can review their performance on practice questions to assess their strengths and weaknesses in various domains and analyze their thought processes.

At our institution, the preclinical curriculum is divided into nine courses, and students must pass a 100-question multiple-choice exam at the culmination of each course. This division of preclinical materials yields an extremely high volume of content to be assessed in each final exam. Given the breadth of the content, students often request practice questions that they can use to assess their knowledge and prepare for the summative exams. Although there are several third-party question banks available, they are geared towards board exams with content, learning objectives, and topic categorization that are not neatly aligned with those of our curriculum. Since constructing high-quality multiple-choice questions de novo is an arduous process, the high demand for a curriculum-specific question bank at our institution has been unmet.

We have also faced another unmet demand, albeit from a much smaller group of students, for opportunities to engage in medical education and curriculum development. While we have multiple “pathways,” such as global health, clinical research, and primary care, there are few experiences available for students interested in medical education and teaching.

Our goal was to meet those two student requests: (1) develop practice questions for junior medical students to learn and review course content and (2) provide opportunities in teaching and assessment for interested senior medical students. We developed an elective course into which 11 second-year medical students self-enrolled due to their interest in education and teaching. The elective was one of over 20 offered that semester and fulfilled one of two required elective credits. In addition to various workshops, the course featured a capstone project of creating a practice exam for the first-year cardio-pulmonary-renal course.

Activity

In order to train the second-year medical students, we held a two-hour workshop, led by a faculty trained in medical education and assessment, in which we discussed the guidelines [4] for writing high-quality, content-rich, multiple-choice questions and used the examples provided by the National Board of Medical Examiners [5].

Several key concepts were taught to students during this workshop:

1. Properly constructing question stems to avoid inadvertently assessing the test-takers’ language skills: For example, we reviewed common confusing syntactic features of question stems to avoid (e.g., double negatives, phrases such as “all of the following are true except”). Additionally, students were instructed to restrict the length of question stems and answers to avoid giving an unfair advantage to fast readers.

2. Appropriately choosing answer options to mitigate the benefit of “good” test-taking strategies and maximize assessment of medical knowledge: We discussed the importance of choosing appropriate distractor options that test the student’s knowledge rather than their skills of eliminating unlikely options. For example, students were instructed to avoid writing options that are nearly synonymous (which may allow the strategic test-taker to eliminate both options) or options that are mutually exclusive (which may suggest to the strategic test-taker that one of the two must be correct).

3. Creating robust answer explanations: Students were instructed to write detailed answer explanations for the correct and incorrect answer choices for each of their questions. Test-takers would therefore be able to use the explanations to review key pathophysiology and understand why each of the distractors seemed like good options but were ultimately incorrect. This not only helped in covering more course content, but also ensured that faculty reviewing the questions would be able to more accurately and efficiently validate the question items.

At the end of the workshop, the second-year students practiced writing sample boards-style, clinically integrated, multiple-choice questions under the direct supervision of the faculty and in collaboration with peers. Each student shared the question that they wrote and received positive and constructive feedback on the quality of their newly formed question based on the guidelines discussed during the workshop.

After the workshop, students wrote 1–2 questions and detailed answer explanations for each of their selected topics and e-mailed the completed question items to the corresponding faculty member for review and feedback. Faculty were instructed to provide feedback that discussed the positive factors of the test item, how the item could be revised, and/or why the item would not be selected. After making any necessary modifications based on the feedback, the students submitted their items to the chair of the cardio-pulmonary-renal course. Out of 41 submitted student-written, faculty-reviewed questions, the course chair selected 34 questions and combined them with 16 purely faculty-written questions to generate a 50-question practice midterm exam. The final selection of questions was made to ensure appropriate content distribution. The newly composed practice exam was administered to the entire first-year class of 175 medical students. After the midterm was administered, a reflective workshop was held with the student test-writers to review how their question items performed based on item analysis results to provide and receive post hoc feedback about their questions. A class-wide, mandatory survey was also given to the first-year students to elicit their feedback regarding their experience with and perceived utility of the practice exam.

Results

We found no statistically significant difference in question difficulty (% of the students who answered correctly) between the student-written (79.5%) and faculty-written (84.0%) test items (t = 0.91, p = 0.37). The discrimination index based on Kelley’s (1939) derivation (the mean difference in the percent correct response on a test question between the upper and lower 27% of test-takers) [6] was significantly higher for student-written (0.29) vs. faculty-written (0.17) questions (t = 3.19, p < 0.01). The discrimination index using the mean point biserial correlation coefficient was also significantly higher for the student-written (0.32) vs. faculty-written (0.25) questions (t = 2.37, p < 0.05).

Overall, the new practice midterm received a rating of 4.5/5 (SD = 0.73) on an anonymous course evaluation survey completed by all 175 first-year students. Of the 46 comments from the survey, 32 comments were positive. The first-year students indicated that the practice midterm was “helpful in solidifying the block’s topics,” that the questions “fairly addressed the various course objectives,” and that taking the exam helped them “assess [their] grasp on the course material thus far.”

Meanwhile, anonymous written feedback from the student question-writers revealed that the process of constructing the questions and answer explanations helped them develop their exam-writing skills, deeply review key first-year topics, and refine their own test-taking strategies. Two students noted that the question-writing helped prepare them for the Step 1 exam, to be taken at the end of second year, by helping them reconceptualize how to approach questions. Notably, they appreciated receiving personal feedback from faculty and indicated that they would participate in similar projects in the future. Additional comments indicated that the exercise helped “elucidate how knowledge can be presented in the form of a question” and that the activity was “not only helpful to the faculty” by saving their time and reducing their workload in constructing test items, but also a good “exercise in critical thinking” for the student test-writers. They reported that writing 3–5 questions was not a significant time burden (3–6 hours in total per student). Constructive criticism primarily centered around the variability in timeliness and quality of feedback from faculty lecturers.

Discussion

This project helped meet junior medical students’ demand for curriculum-specific practice test questions while not only alleviating the workload on faculty, but also providing a positive learning experience and teaching opportunity for the participating senior students. Admittedly, given that this project was conducted with only one cohort, we cannot provide evidence that the first-year students performed better on their final assessment as a result of taking the practice midterm. Additionally, the questions generated through this student-written approach may not be appropriate for summative assessment due to potential concerns regarding question item security. It may be difficult to generate enough excitement or interest amongst students to voluntarily participate in such a project multiple times or across cohorts.

This project provides a practical foundation for the goal of feasibly developing a comprehensive practice question bank tailored to our curriculum. By applying this methodology across the preclinical curriculum over the years, it would be possible to generate hundreds of practice multiple-choice questions covering the key objectives for each course. Since performance on student-written practice multiple-choice questions may predict performance on final examinations [7], course directors would be able to use these formative examinations in order to identify students who may benefit from additional academic support prior to the summative examination.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

IRB exempt

Informed Consent

NA