Before it becomes completely random

Selecting our future colleagues for postgraduate training programs is one of residency program’s greatest responsibilities. After waves of change to the nature of assessment in undergraduate medical education, medical graduates’ application files have become increasingly bare and ranking candidates seems more difficult than ever.¹

The benefits of pass/fail grading systems for students are very well documented and include reduced stress surrounding examinations and increased emphasis on long-term learning without any detrimental effects on licensing examination performance.² Unfortunately, when undergraduate medical programs in Canada universally adopted them, it rendered the ranking of candidates based on an objective measure of their foundational and clinical knowledge much more difficult. A handful of postgraduate programs have adapted by administering home-made, subject-specific, examinations during their pre-interview assessment of candidates. While this is a quick fix, it is only viable because it is the exception.

The reduced emphasis on summative assessment in medical curricula has also had another unexpected consequence. To keep some agency on their applications, it seems students are increasingly investing time in research and volunteering activities. Assessment often drives learning, and this shift may indirectly under-emphasize the importance of studying. As an unintended consequence of pass/fail grading, students may feel justified in dedicating less time towards studying for clinical practice since they cannot be differentiated on this basis. Overall, this new paradigm may divert attention away from clinical performance, which is arguably the most important skill to foster in clerkship years.

The collection of evaluations in the medical student performance record (MSPR) provides multiple snapshots of students’ clinical skills ranging from medical expertise to professionalism and communication. This should have compensated for the loss of information from pass/fail grading systems. Unfortunately, the usefulness of the information made available in the MSPRs is also in question.³ Programs commented on its lack of uniformity, questions regarding the source of narrative comments and the relative lack of negative information.³ It is notable that some faculties have even begun to forgo narrative comments altogether or any negative or constructive comments from clinical rotations. In a competitive match process where interview offers are limited these issues incorporate an element of uncertainty which is necessarily detrimental to certain applicants. If faculties are providing different information on their students, it is difficult to assess students from faculties other than their own. Elective rotations used to very helpful in assessing “away” candidates. Unfortunately, various reforms have limited their availability. The AFMC, rightfully, capped the number of electives in a single entry-level discipline.⁴ Students can now visit fewer programs in one field and must be more selective with their electives. This was a foreseen consequence of a well-received and thought-out policy. A completely unforeseen change was the COVID-19 pandemic. Away elective rotations have been on hold for two years now meaning no faculty has been able to perform first-hand assessments of some (if not most) of their residency candidates. Programs need to exclusively rely on the information provided from candidates’ home-school, which as mentioned may be filtered in some ways. The effect on the pre-interview and final ranking is unclear but is concerning for applicants’ mobility.

Interviews may become increasingly important to differentiate candidates given the changes to applicants’ files. This is a problem for both applicants and programs. For applicants, it creates a high-stakes interview process where the margin for error is small. For programs, it begs the question of reproducibility. If more information were available to them, would they invite the same sample of candidates? Essentially, if there is any doubt to the reproducibility of this process, it is likely that the changes discussed here have incorporated a certain element of randomness that is out of the control of postgraduate programs and candidates alike.

In 2020, following the cancellation of in-person interviews due to the COVID-19 pandemic, McMaster University’s Faculty of Medicine made headlines by announcing they would be selecting a portion of their incoming medical school cohort randomly. McMaster based their decision on years of admissions data that, in their eyes, suggested mobility of students following an initial ranking was not as widespread as commonly thought. While this can be debated, it provides an extreme example of how programs might select applicants when the data available to rank them is either missing, incomplete or uninformative.⁵ Postgraduate residency programs may already be wrestling with this question. A broader discussion on how to correct for the loss of information in applicant files needs to be had before we perform one of our greatest responsibilities completely randomly.

Conflicts of Interest

The authors have no conflicts of interest to disclose.