Residency Applicant Perceptions of Multiple Mini Interviews in Post-Graduate Year 1 Residency Interviews

Abstract

Introduction: Residency interviews offer an opportunity for both candidates and programs to evaluate whether the pairing is a good fit. Multiple mini interviews (MMIs) have been incorporated into interviews for medical training as a way to evaluate non-cognitive abilities.

Objectives: To determine how candidates perceived the interview process at our institution, specifically related to the MMIs.

Methods: This retrospective review evaluated candidates’ perceptions of traditional interviews and MMIs through post-interview surveys over a 3-year period. Candidates evaluated the interview activities, time allowed for MMIs, and overall impression of the program during the 2-week period between submitting rank lists and the release of match results. Survey results are reported using descriptive statistics. Candidate perceptions on their ability to showcase skills in different types of interviews was evaluated with chi-square test.

Results: The interview day increased the desire to pursue residency at our institution for 88% of candidates. Candidates reported similar ability to showcase skills developed during pharmacy school through the clinical and patient education MMIs compared to traditional interviews; however, they did not feel the collaboration MMI showcased their abilities as well as traditional MMIs.

Conclusion: The introduction of MMIs to the residency interview day were perceived positively by most candidates and allowed candidates to showcase abilities in a different manner from traditional interviews.

Introduction

Each year, the pharmacy residency recruitment process brings excitement and anxiety to pharmacy students and residency program leadership. While programs seek strategies to streamline application review and identify candidates who will be high performing residents, they must also provide an interview experience that highlights the program’s strengths and is engaging to candidates. Several methods of assessing clinical knowledge and problem-solving have been described in the literature including the use of clinical questions, development of care plans or SOAP notes, and formal presentations.¹

Although many studies report means of evaluating residents during the application and interview process, there is no one-size fits all interview structure for pharmacy residencies. This is not surprising as the characteristics that make for successful residents are likely different for each institution.² One example of this is the importance many programs place on fit of the resident for the program; however, fit will have a unique definition at each program.^3-8 Aligning the characteristics evaluated during the interview day with those desired in residents is key to a successful interview day for a residency program.^9,10

Multiple mini interviews (MMIs) are short, standard interviews that have been described in medical and pharmacy school admission interviews. The incorporation of MMIs into medical school admission interviews has been found to measure non-cognitive skills and differentiation of candidates.^11-16 Pharmacy schools have also incorporated MMIs with favorable perceptions from those who participated. ^{11, 12, 16, 17} The University of Toronto pharmacy program reported MMI scores were the only evaluation tool used at pharmacy school admission process that predicted performance on the OSCE (objective structured clinical examination) portion of pharmacy licensure in Canada.¹⁸ There is less literature discussing the use of MMIs in the interview process for pharmacy residency programs. The post-graduate year one (PGY1) pharmacy residency program at the University of Kentucky implemented MMIs into their interview process in 2013. Both candidates and interviewers felt the MMI structure was successful in demonstrating and assessing specialized knowledge.¹⁹

We implemented MMIs into our PGY1 pharmacy residency interview process beginning January 2015 in an effort to improve the ability to identify candidates who would succeed in our program. In conjunction with the changes in the interview structure, we initiated a continuous quality improvement process in order to identify successes and challenges of the interview structure each year and ensure our interview structure did not deter candidates from ranking our program. The primary goal of this study was to determine candidates’ perceptions of the interview process at our institution, specifically related to the MMIs.

Methods

In January 2015, our PGY1 Pharmacy Residency Program incorporated four MMIs into the residency interview day designed to assess specific skills deemed to be important in our PGY1 residency program (Table 1). Each MMI was created by the PGY1 program leadership including the residency program director (RPD) and residency coordinators (RC) and reflected scenarios a resident may encounter at our institution (Table 1). For each MMI, a grading rubric was designed to increase the objectivity of the evaluation and tested as has been previously described.²⁰ Most MMIs were completed one-on-one with the candidate and evaluating preceptor, with the exception of the collaboration MMI, which was conducted in small groups of candidates paired with an evaluating preceptor. In addition to the MMIs, candidates were introduced to the program, completed 4 traditional interviews, toured the hospital, attended a departmental activity, and had lunch with the current residents. In order to minimize anxiety during the MMIs, all candidates were provided information about the philosophy of MMIs via email, and the RPD explained the MMI process at the beginning of the interview day. Based on feedback received from the survey and the lack of ability to distinguish between candidates, one MMI was removed after the first year and replaced with a preceptor roundtable.

Table 1.

MMI classification, examples, and skills evaluated

MMI Classification	Example Scenario	Characteristics Evaluated
Clinical case	Acute stroke evaluation, treatment of sepsis	Critical thinking, thought process, problem solving skills
Collaboration activity	Group presentation on role of pharmacist, mini teaching session for pharmacy learners	Collaboration, leadership skills, teamwork
Patient education	Inhaler use, insulin teaching	Communication
Time management activity	Creation of a project timeline	Time management, project management, planning abilities

A survey was created using Survey Monkey® assessing impressions of the interview day and timing of the MMI scenarios. Survey questions addressing impressions of the interview day are included in Tables 2, 3, and 4; of note, questions in Table 4 were added in 2016. Candidates were also able to leave comments about their impression of the interview day. The survey was distributed annually via email to all of the candidates following the final rank list submission deadline from the American Society of Health-Systems Pharmacists (ASHP) Resident Matching Program and before the release of match results. All candidates who completed MMIs during the PGY1 interview day at our institution from January 2015 through March 2017 were included. Candidates were able to complete the survey anonymously. Feedback was reviewed by the PGY1 RPD and residency coordinators (RCs) immediately following the Match. In conjunction with observations from the RPD and RCs, the feedback was used to implement changes in the interview schedule for the following years.

This study was approved by the University of Chicago’s Institutional Review Board. Statistical analysis was conducted with StataCorp. 2013 (College Station, TX: StataCorp LP). Descriptive statistics were used to describe survey responses. Chi-square test was used to compare categorical.

Results

The post-interview survey was completed by 126 of 192 candidates over a three year period (66%). A majority of the candidates reported a positive experience with the interview day, with 88% reporting that the interview day increased the desire to pursue residency training at our institution. Candidate responses to questions regarding overall impression of the program for each year are described in Table 2. When asked to rank the top three interview activities that left a positive impression of the program, 41%, 49%, and 66% of candidates ranked MMIs in their top three activities for each year from 2015 to 2017 (Figure 1). The percentage of candidates ranking the MMI as the activity that left them with the strongest positive impression increased from 10% in 2015 to 29% in 2017.

Table 2.

Survey results pertaining to overall impression of the program

	2015 (n =49)	2016 (n=39)	2017 (n =38)
The interview day increased my desire to pursue residency training at [the institution]	41 (84%)	35 (90%)	35 (92%)
I had adequate time with current residents	44 (90%)	38 (97%)	35 (92%)
I met a sufficient number of preceptors	37 (76%)	23 (59%)	33 (87%)
The interview day was a good length	27 (55%)	31 (79%)	27 (71%)
I had adequate time to ask questions	n/a	n/a	38 (100%)

Figure 1.

Percentage of Candidates Reporting a Positive Impression for Activities Conducted during the Interview Day†

In 2016 and 2017, candidates were asked if the type of interviews during the interview day allowed them to showcase the skills and experiences they gained over the four years of pharmacy school. Results are reported in Table 3. There was no difference between the clinical MMI or patient education MMI and traditional interviews with regards to the candidates’ assessment of how they showcased their skills and experiences (p=0.68 and p=0.86, respectively). Candidates felt the collaboration MMI did not demonstrate their skills and experiences as well as the traditional interviews (p<0.01).

Table 3.

Candidates’ perceptions of ability to showcase skills developed during pharmacy school

Interview	Yes, n (%)	Somewhat, n (%)	No, n (%)
Clinical MMI	62 (81%)	14 (18%)	1 (1%)
2016	29 (74%)	10 (27%)	0 (0%)
2017	33 (87%)	4 (11%)	1 (3%)
Collaboration MMI	31 (40%)	34 (44%)	12 (16%)
2016	14 (36%)	18 (46%)	7 (18%)
2017	17 (44%)	16 (42%)	5 (13%)
Education MMI	61 (79%)	12 (16%)	4 (5%)
2016	33 (85%)	5 (13%)	1 (3%)
2017	28 (74%)	7 (18%)	3 (8%)
Traditional Interview	59 (77%)	17 (22%)	1 (1%)
2016	29 (74%)	9 (23%)	1 (3%)
2017	30 (79%)	8 (21%)	0 (0%)

During the first year, candidates reported wanting more time to ask questions and more time with the programs’ current residents. Almost half of the candidates reported that the interview day was not a good length. For the 2016 interview cycle, a question and answer session with the programs current residents was added, and the interview day was slightly reorganized to result in an overall length decrease of 30 minutes. After implementation, few candidates ranked the resident interview session in their top three during the interview day. Additionally, several candidates reported they did not have enough time with preceptors and would like to ask more questions of the preceptor group. To address this, the resident question and answer session was subsequently switched to a preceptor round table.

Discussion

While residency interviews are known as an opportunity for programs to identify strong candidates, it’s also an opportunity for candidates to get a feel for the program and determine if they will gain the experiences desired during residency. Much of the literature surrounding pharmacy residency interviews focuses on aspects RPDs desire or evaluate; however, candidates are the drivers for the Match process and must determine which program is the best fit for them based on interactions prior to and during the interview. As changes were made to our interview structure including the addition of MMIs which are relatively new to pharmacy residency, we wanted to ensure these changes did not negatively impact candidates’ perceptions of our program.

As described by Powers Shaddix et al, the right “fit” for a candidate in a program can include many components such as patient population, preceptors, and program culture.²¹ In order for residents to obtain a better understanding of the programs and determine what programs fits their needs, it is important for programs to provide candidates with a realistic picture of what residency might look like in the program. The MMI scenarios used during the interview cycles provided examples of what a resident might experience at our institution when interacting with a preceptor or patient during a learning experience. Candidates were provided institutional protocols or guidelines, as appropriate for the scenario, and were able to utilize them as would be expected if they were to match with our program. Candidates were therefore able to engage in these activities prior to coming here as a resident and use this information in determining if the environment was one they desired in residency training.

Although we cannot compare to years before MMIs were incorporated, the percentage of residents reporting that the interview day increased their desire to pursue residency training at our institution increased each year. Candidates offered feedback via the narrative comments in their survey responses that the combination of traditional interviews and MMIs made the day engaging and allowed different skills to be showcased. The interview experience was described by two candidates as one that set the bar very high for other programs and gave them an opportunity to judge how their experience as a resident would be. While the increased desire to pursue residency training at our institution was based on the full interview experience, the percentage of candidates reporting that the MMIs had a strong positive impact leads the authors to believe that the MMIs contributed to this significantly.

Over the three-year period, the MMIs became the activity with the highest proportion of candidates reporting that it left a positive impression of the program. This change may have been driven by increased comfort of candidates with MMIs or an improvement in the programs’ execution of the MMIs over the three-year period. Regardless of the reason, the MMIs represented a positive addition to the interview day for most candidates. Most residency candidates reported that the clinical MMI, the patient education MMI, and the traditional interviews offered the ability to showcase their skills and experiences from their time in pharmacy school; whereas, less than half of the candidates felt that the collaboration MMI allowed them to showcase their skills. The PGY1 RPD and RCs felt that the MMIs gave the program a chance to evaluate the characteristics listed in Table 1 more effectively than traditional interview questions. This is similar to the responses received by candidates and interviewers when MMIs were incorporated into PGY1 interviews at University of Kentucky.¹⁹ The RPD, RCs, and preceptors involved in the interviews felt that compared to the behavioral questions asked in traditional interviews, MMIs forced candidates to use the skills they had developed during school instead of simply describing scenarios.¹⁹ Similarly, interviewers at Emory University Hospital felt that MMI implementation allowed for better comparison between candidates.²² Additionally, candidates may not be aware of the skills being evaluated in a MMI and may, therefore, not realize that the extent to which they are showcasing skills developed during pharmacy school. For example, the collaboration MMI was not considered highly reflective of skills learned during school; however, leadership and collaboration skills developed or refined during school are essential skills for all residents and were perceived to be more objectively evaluated in the collaboration MMI than in behavioral questions by the interviewers.

As schools of pharmacy prepare current students for residency interviews, students are trained to answer traditional interview questions and often receive coaching prior to interview days.^23-26 Because candidates cannot prepare for a specific MMI scenario in advance, the use of MMIs may level the playing field and allow candidates to show case their abilities instead of providing canned responses to interview questions. For candidates who cannot access residency interview preparation through their schools of pharmacy, the traditional behavioral questions used in interviews may be more challenging than for those who received coaching. MMIs also provide a more objective evaluation that limits interviewer bias that may arise based on responses to behavioral questions.

In conjunction with the introduction of the MMIs, serial adjustments to the interview day were made in response to feedback to ensure that candidates had adequate time with preceptors and current residents while maintaining an appropriate overall length. These adjustments resulted in a smaller, core group conducting one-on-one interviews, and therefore less variability in which interviewers asked specific questions. This change may have subsequently decreased subjectivity in rubric completion, but this is not something we evaluated formally at the time.

With the core group conducting one-on-one interviews being primarily residency leadership team members, candidates may not have felt they were meeting enough other preceptors, or that they had sufficient time to ask questions of preceptors, and this was observed in the candidates’ narrative responses to survey questions. To address this concern, we added roundtable format sessions. This also allowed more preceptors to stay engaged in the process with lower minimum time commitment for participation than required in our previous model.

This study has several limitations. First, the MMIs were developed each year by our PGY1 leadership team and were built to evaluate characteristics that we felt would help candidates succeed in our program. However, the characteristics that lead to successful residents may be different for each program, which may limit the external validity of this study. Further, each individual MMI activity, while thoughtfully crafted, was not externally validated to show that it was a consistent way to measure the specific skill or skill set we were hoping to evaluate. A previous analysis showed that clinical MMI scores correlated with clinical performance better than traditional interview scores; however, future work could include evaluating residents in the other domains examined in the MMI during their first quarter of the residency year, to determine if MMI scoring is reflective of future performance in our program.²⁰ In terms of the survey itself, applicants who did not respond may have held differing views about the MMI experience and overall interview day compared to responders. Finally, we did not continue surveying candidates in subsequent years; however, MMIs continued to be utilized in our interview structure and were part of the virtual interviews as well.

Last, these MMIs all occurred during in-person interviews where candidates were physically present at the practice site. The relative ranking of MMIs compared to other items in the interview day, and even the structure of the interview day, for example the presence of a hospital tour, may be shifted in the setting of increased virtual residency program interviews conducted by programs across the country secondary to the COVID-19 pandemic. Further research would be needed to determine candidate perceptions of MMIs in a virtual setting.

Conclusions

This study demonstrates that candidates rank MMIs among one of the top experiences in the pharmacy residency interview process. Candidates feel the clinical and patient education MMIs are equally able to showcase their skills and abilities with these activities as compared to traditional interviews. The incorporation of MMIs into an interview day can be done without negatively impacting recruitment from a candidate’s perspective.

Conflicts of Interest: The authors have no relevant conflicts of interest to disclose.

The opinions expressed in this paper are those of the authors.