Survey of Canadian urology programs: Which aspects of the Canadian Residency Matching Service (CaRMS) application are the most important?

Abstract

Introduction

For medical students, determining which aspects of the Canadian Residency Matching Service (CaRMS) application are the most important when applying to residency programs can be challenging. Due to the lack of current and reliable information on the selection criteria of Canadian urology residency programs, we surveyed each program about which criteria are the most important when selecting future residents in order to provide medical students with more transparency and programs with a better idea of how their criteria compare to those of others.

Methods

An electronic survey was sent to all 13 Canadian urology residency programs (both program directors and selection committee members). It asked respondents to rate each aspect of the application on a five-point Likert scale. Following a 100% response rate from program directors, the same survey was sent to selection committee members. A numeric mean score was calculated for each individual aspect surveyed to create an overall rank list of the components. Independent samples t-tests (two groups) were used to compare the scores of program directors vs. program committee members and of francophone programs vs. anglophone programs.

Results

Forty-three urologists involved in the application process answered. The three most important aspects were rotation performance at the respondent’s institution (4.95±0.21), quality of reference letters from a urologist (4.60±0.62), and interview performance (4.49±0.63). There were no statistically significant differences between program directors and committee members for mean score of any aspect surveyed. Compared to anglophone programs, francophone programs gave statistically more significant importance to French proficiency (p<0.001) and pre-clinical academic performance (p=0.0272), while giving less importance to English proficiency (p<0.001).

Conclusions

Canadian urology residency programs are similar in that they rank “clinical performance during a rotation at their school” as the most important selection criteria when choosing a future urology trainee. Graduate degrees, career plans, and reference letters from non-urologists have less impact when choosing future urology residents. Francophone schools and anglophone schools differ in the importance of language proficiency and preclinical grades as selection criteria for urology residency. This study will provide future urology applicants with more information and transparency when applying to urology programs in Canada and be of use to urology residency programs that must now publish their selection criteria.

Introduction

In Canada, all urology residency programs use the Canadian Residency Matching Service (CaRMS) to select future postgraduate trainees. The selection process involves the submission of the CaRMS application package, review of the documents by programs, and an in-person interview. The CaRMS application package includes reference letters, personal letters, curriculum vitae, medical student personal record (MSPR; also known as the Dean’s letter), and transcript(s).¹

For the medical student applicants, determining which aspects of the application are the most important drivers to be selected as a urology trainee is difficult. Students must often rely on career advisors who may not always have accurate information, mentors who do not always share the same views as program selection committees, or third-party websites not managed by the programs themselves. Urologymatch.com, a website created with the purpose of informing medical students about the American Urological Association match process, has received over 1 million visits since its inception, as applicants are eager to learn which factors will help to predict a highly successful application to urology.² In addition, different residency training programs across the country may have differing views on the importance of various aspects of each application.

In 2013, to increase the transparency of the Canadian residency match, the Association of Faculties of Medicine of Canada (AFMC) released the Best Practices and Selection (BPAS) report. Among its recommendations, residency programs must publicly share their selection process goals.³ This implies identifying and publishing online the skills and attributes of a successful applicant for their program. This practice will be implemented in the coming years.

To further provide transparency to urology applicants, we sought to survey Canadian urology residency program selection committees on which aspects of a medical student’s application are the most important when selecting future residents. By doing so, our objective is to provide urology hopefuls with information on which aspects of their application hold the most weight for selection committees of Canadian urology residency programs. In the context of BPAS’ implementation, this study also offers Canadian urology programs an idea of how their current criteria compare to those of other programs. We hypothesized that clinical performance would be highly regarded, while career aspirations and pre-clinical grades would rank more poorly.

Methods

Survey construction

In May 2018, a brief electronic survey constructed with Google Form (Google Form, Mountain View, CA, U.S.) was developed using the CaRMS website and feedback from residents and staff urologists involved in medical education from a single center. Collaborators carefully reviewed the survey to determine if the questions effectively captured the breadth of possible urology resident criteria. While certain items were added following the feedback, others were removed due to overlap with other items. For example, the MSPR, which varies from school to school and encompasses academic performance, extracurricular involvement, and other aspects already surveyed, was removed.

Survey content

Respondents were asked to rate 30 items on a five-point Likert scale with 5 being “very important” and 1 being “unimportant.” These aspects related to the interview, academics, electives, letters, language proficiency, extracurricular activities, and research, as well as career aspirations. The survey was available in both French and English. All 30 items can be found in Table 1.

Table 1.

Rank, mean score given, and standard deviation of the importance of aspects of the application

Rank	Aspect of application	Mean score	SD
1	Performance during a rotation at respondent’s school	4.95	0.21
2	Quality of reference letters from urologists	4.60	0.62
3	Quality of/performance at the interview	4.49	0.63
4	Doing a rotation at respondent’s school	4.47	0.91
5	Clerkship academic performance	4.42	0.63
6	English proficiency	4.28	0.77
7	Stress management during the interview	4.26	0.69
8	Reputation of the urologist writing the reference letter	4.07	0.91
9	Extracurricular activities	3.81	0.76
10	Pre-clinical academic performance	3.74	0.93
11	Appearance at interview	3.74	0.79
12	Quality of the letter of intent	3.65	0.78
13	Quality of research publications in urology	3.63	0.82
14	Athletic endeavors	3.60	0.73
15	Number of presentations in urology	3.58	0.79
16	Community involvement and service	3.56	0.70
17	Number of reference letters from non- urologists	3.51	0.91
18	Total length of rotations in urology	3.51	0.96
19	Entrepreneurial endeavors	3.51	0.74
20	Number of publications in urology	3.49	0.91
21	Artistic and musical endeavors	3.40	0.69
22	Political and student leadership endeavors	3.19	0.88
23	Quality of research publications in fields other than urology	3.17	0.85
24	Presentations in fields other than urology	3.12	0.96
25	Number of publications in fields other than urology	3.12	0.88
26	Desire to work in a university center	2.63	1.07
27	Desire to work in a community setting	2.58	1.03
28	Having a higher degree (Masters, PhD)	2.53	1.10
29	Reference letters from non-urologists	2.47	0.88
30	French proficiency	2.33	1.57

SD: standard deviation.

While not initially surveyed for, an additional item looking at the importance of a student’s secondary language proficiency was created. This variable was assigned the scores that anglophone programs gave to French proficiency and that francophone programs gave to English proficiency. This item is not included in Table 1.

Survey dissemination

The survey was sent in March 2018 to all 13 program directors of each Canadian urology program offering residency positions. Following a 100% response rate from program directors, the same survey was sent to each program’s residency selection committee members (including residents) in June 2018. The composition of the selection committees and the emails of their members were disclosed by the program directors. The survey was closed after at least two committee members from each program answered the survey or if no response was received after five email reminders sent every two weeks.

Statistical analysis

A numeric mean score was calculated for each aspect surveyed. This was used to create an overall rank list of the components. Independent samples t-tests (two groups) were used to compare the scores of the following groups: program directors vs. program committee members and francophone programs vs. anglophone programs. Statistical analysis was performed with Stata MP 14 (StataCorp LLC, College Station, TX, U.S.).

Results

Response

Forty-three urologists involved in the application process answered the survey (12 in French and 31 in English). All 13 program directors answered the survey. All institutions had at least one committee member respond, with only one institution not attaining the two committee-member responses after five email reminders.

Most and least important aspects of the application

The three most important aspects of the application as perceived by all programs were the rotation performance of the applicant at their institution (mean score of 4.95, standard deviation [SD] 0.21), quality of reference letters from urologists (mean score of 4.60, SD 0.62), and quality of/performance at the interview (mean score 4.49, SD 0.63). The three least important aspects of the application were French proficiency (mean score of 2.33, SD 1.57), reference letters from non-urologists (mean score of 2.47, SD 0.88), and having a higher degree, such as masters or doctorate degree (mean score of 2.53, SD 1.10). Table 1 provides an overview of all items surveyed with the mean score, standard deviation, and rank of the item.

Program directors vs. committee members

Student t-test revealed no statistically significant differences between program directors and committee members for the mean score of any aspect surveyed.

Francophone programs vs. anglophone programs

Student t-test revealed statistically significant differences between francophone programs and anglophone programs for mean score of English proficiency (p<0.001) and French proficiency (p<0.001), as well as pre-clinical academic performance (p=0.0272). Compared to anglophone programs, francophone programs gave more importance to French proficiency (mean score of 4.58 vs. 1.45) and pre-clinical academic performance (mean score of 4.17 vs. 3.58), while giving less importance to English proficiency (mean score of 3.50 vs. 4.58).

When looking at language proficiency in terms of a second language, differences between anglophone and francophone programs persisted (p<0.001). Francophone programs gave more importance to second language proficiency than anglophone programs did (mean score of 3.50 vs. 1.45).

Discussion

In this study, the most and least important aspects of the CaRMS application from the program’s perspective were identified. Previous Canadian studies examining residency selection criteria did not specifically look at urology,⁴ were outdated,⁵ or focused on specific components of the CaRMS application.⁶ The only urology-specific study was conducted in the U.S. and demonstrated the importance of the United States Medical Licensing Examination^® Step 1 examination,⁷ which is not required in the Canadian medical graduate application. Literature has also highlighted the stark differences between the CaRMS process and the American process.⁸ As expected, the most important criterion when evaluating an applicant was the medical student’s performance during an elective at the respondent’s school. This criterion is supported by other highly regarded items, such as clerkship academic performance and a reference letter from a urologist — both testaments of a student’s clinical performance. It is also supported by “doing a rotation at the respondent’s school,” ranking fourth overall. This finding can be interpreted as urology programs wishing to directly assess students and/or get a sense of commitment from a student. Between 2013 and 2019, out of 220 students matching to a Canadian urology residency program, only 25 (11%) matched without an elective in urology at their matched school. In the last two years, only one student out of 61 did so¹ (Fig. 1). With the new AFMC initiative to reduce the number of electives that a student can do in a single specialty, the importance of this criterion in the future remains unknown. Starting with the class of 2021, the AFMC has implemented an eight-week limit on electives in a single specialty to favor parallel planning and a more diversified medical education.⁹ As previously discussed in a recent commentary, while having several benefits, this cap limits the number of institutions a medical student can visit in the context of a urology rotation, limiting their competitiveness in the residency match at unvisited schools if the selection culture doesn’t change.¹⁰

Fig. 1.

Canadian medical graduates (CMGs) matched to urology with or without elective in urology at matched school (Source: carms.ca, publicly available).

Additionally, SDs were the smallest for the most important selection variables identified in the survey. This demonstrates that most programs across the country are similar in their rankings of important aspects of the application. Major differences included those between francophone and anglophone programs for English proficiency, French proficiency, and pre-clinical academic performance. While French proficiency in francophone programs is self-explanatory, preclinical academic performance can be explained by the different grading policies — Quebec francophone schools using a numeric grading system rather than the pass/fail system. However, these programs are moving towards the pass/fail system and, therefore, this difference with anglophone programs should progressively disappear. Other than the differences between the francophone and anglophone programs, items surveyed with high SDs included research presentations in fields other than urology and specific career aspirations. These may represent subtle differences in residency training cultures and phenotypes, such as academic culture, values, caseload, etc.

Program directors and committee members agreed on all items. Little disagreement among the committee bodes well for applicants. However, we do not know the internal dynamics of each committee. Some may have program directors making the final decision with advisement from the committee, while others look for majority or consensus. Respondents may simply be relaying information that their committee considers important.

This study will hopefully be useful to those who aspire to a urology specialization by providing them with more transparency when applying to a Canadian urology residency program. In the context of the upcoming implementation of the AFMC’s BPAS report, this study will also be of use to programs that will soon have to publish their selection criteria and processes.³ Future studies should explore students’ perception of which aspects of the application are most important. Diving further into the most important selection criteria identified in this study can provide more insight to medical students. For example, while this study demonstrates that clinical performance is an important criterion, specifically identifying attitudes, behaviors, and the level of knowledge needed to have a good clinical performance may be of additional use to medical students. A more significant question would be to assess which qualities and achievements at the medical student level make for a good urology resident.

This study is not without limitations. While the survey was reviewed by experts, it was not validated. This may limit the reproducibility of the study. Intangibles, such as a student’s fit with the team and commitment to a program, were not explicitly assessed. Programs were not asked if they had demographic preferences, such as gender or medical school attended preferences. While these factors may or may not influence the selection process,¹¹ they were outside the scope of this study. Institution-specific assessments were not surveyed, such as the assessment of basic skills at the interview — a practice that might increase considering the cap on urology electives limiting the possibility to observe candidates locally. Finally, given recent changes to the AFMC electives policy and considering that it is only a one-year sampling, the applicability of this study may be limited for students in the class of 2021 and thereafter.

Conclusions

Canadian urology residency programs are similar in ranking “clinical performance during a rotation at their school” as the most important selection criteria when choosing a future urology trainee. Conversely, graduate degrees, career plans, and reference letters from non-urologists have less impact. Francophone schools and anglophone schools differ in the importance of language proficiency and pre-clinical grades as selection criteria for urology residency. This study will provide future urology applicants with more information and transparency when applying to urology programs in Canada. Our findings also have implications for residency programs, which must now publish their criteria and selection processes online.