Perceptions and Cost-Analysis of a Multiple Mini-Interview in a Pharmacy School Admissions Process

Abstract

Objective. To improve the quality of admissions interviews for a doctor of pharmacy program, using a multiple mini-interview (MMI) in place of the standard interview.

Methods. Stakeholders completed an anonymous web-based survey. This study characterized perceptions of the MMI format across 3 major stakeholders (candidates, interviewers, admissions committee members) and included comparative cost estimates.Costs were estimated using human and facility resources from the 2012 cycle (standard format) and the 2013 cycle (MMI format).

Results. Most candidates (65%), interviewers (86%), and admissions committee members (79%) perceived the MMI format as effective for evaluating applicants, and most (59% of candidates, 84% of interviewers, 77% of committee members) agreed that the MMI format should be continued. Cost per candidate interviewed was $136.34 (standard interview) vs $75.30 (MMI).

Conclusion. Perceptions of the MMI process were favorable across stakeholder groups, and this format was less costly per candidate interviewed.

INTRODUCTION

Since the Affordable Care Act was passed in March 2010, the health care system has experimented with innovative approaches to improve access to care for the anticipated 25 million individuals who would soon be eligible for government funded or subsidized health care insurance.¹ One emerging solution to alleviate the shortage of primary care providers in the United States is to expand the role of pharmacists. Through active participation with integrated care teams and by virtue of their expertise in medication management and monitoring, pharmacists can improve medication use and safety, leading to improved patient outcomes and reduced health care costs.²

As the pharmacy profession adopts this expanded patient-centered role, strengths in “soft skills,” such as empathy, ethical reasoning, conflict resolution, adaptability, teamwork, and communication skills are essential.^3-5 Because development of these nonacademic skills is more difficult to incorporate into impacted professional curricula, it is logical that pharmacy programs would select applicants for admission who already possess strong “people skills.” Indeed, requirements for soft skills, which complement the technical skills necessary for pharmacy practice, are heavily incorporated in revised standards and guidelines issued by accrediting and academic organizations in pharmacy.^6,7

However, many pharmacy schools more heavily weigh quantitative measures, such as grade point averages (GPAs), courses completed, and standardized test scores (eg, Pharmacy College Admissions Test) during the admissions process, with less emphasis on nonacademic attributes, which are generally more difficult to measure.^3,4

At the University of California, San Francisco (UCSF), approximately 122 students are admitted to the doctor of pharmacy (PharmD) program each year. The San Francisco campus is unique within the University of California system in that it is the only school dedicated solely to professional and graduate education in health and biomedical sciences. The UCSF PharmD curriculum is a 4-year course of study, and approximately 98% of the entering students possess a bachelor’s degree or higher. Over the past 5 years, the number of fully completed applications received through the Pharmacy College Application Service (PharmCAS) exceeded 1000 annually. Of these, approximately 300 were selected for further evaluation through a 5.5-hour onsite interview day.

The onsite interview process includes a welcome session (60 minutes), a written essay (60 minutes), a formal interview (60 minutes), an informal “chat” session with current students (60 minutes), a financial aid session (30 minutes), and a campus tour (60 minutes). Prior to 2013, the school used a standard interview format during which each prospective candidate met with a faculty member and a current PharmD student. Faculty-student teams had access to selected materials within the candidates’ application file (PharmCAS statement, supplemental application essays, extracurricular activities, and work experience), which enabled the interviewers to ask candidate-specific questions.

Each standard interview lasted 30 to 40 minutes, after which each member of the faculty-student team independently completed a written candidate rating form. Although faculty members and students received annual training in the conduct of admissions interviews, frequent complaints from members of the admissions committee, specifically related to inadequate or inconsistent documentation on the candidate rating form, suggested the interview process was not standardized and should be a target for quality improvement.

In 2013, UCSF replaced the standard interview with the multiple mini-interview (MMI) approach to improve the quality and consistency of the information obtained from the formal interview. The MMI, which is modeled after objective structured clinical examination (OSCE) assessment methods, uses short independent assessments in a timed circuit to obtain an aggregate score of a candidate’s interpersonal communication skills and other measurable attributes.⁸

In 2004, Eva and colleagues reported that the MMI was a reliable tool, and both candidates and examiners were positive about the experience.⁸ Since then, a number of institutions across several disciplines have adopted the MMI format, deeming the approach to be effective and reliable.^5,9-18 In 2008, the University of Arkansas College of Pharmacy incorporated the MMI into their admissions process,¹³ and this institution provided guidance and mentorship to facilitate implementation at UCSF.

After completion of the first MMI admissions cycle, this study was conducted to characterize perceptions of the MMI format across 3 major stakeholders (candidates to the PharmD program, interviewers, and admissions committee members), and estimate the comparative human and facility resource costs required to implement the MMI vs the standard interview format.

METHODS

Organization of the Multiple Mini-Interview

The MMI was conducted as a 4-station (6-minute encounter per station) circuit in the UCSF Simulation and Clinical Skills Center (SCSC) using standard procedures for OSCEs. Twelve interview rooms were equipped with an overhead audio system and a dual camera, digital recording system managed with clinical skills software (B-Line Medical, LCC; Washington, DC). Immediately prior to the first mini-interview, candidates received a 15-minute orientation from a faculty member, which included an overview of the MMI approach and logistics associated with the exercise. After the orientation, the faculty member or a member of the SCSC staff positioned each candidate in front of a computer monitor outside of their first assigned station. An overhead announcement prompting candidates to read the station instructions and knock and enter when ready preceded the appearance of the mini-interview scenario instructions on the monitor. Candidates then entered the room for the up-to-6-minute encounter.

To assist the candidates with time management, a one-minute warning was given via the overhead audio system. At the conclusion of each mini-interview, candidates rotated to the next station, and the cycle was repeated. A faculty member and SCSC staff member served as hall proctors to assist the flow of traffic through the circuits.

Although the literature supports using a greater number of stations (eg, 8 to 12) to enhance reliability and validity,^8,11,16,17 the feasible number for UCSF interview day was 4, which is consistent with previous experience in pharmacy admissions.¹³ Two of the scenarios were developed by UCSF faculty members on the MMI workgroup, and 2 were adapted with permission from the University of Arkansas College of Pharmacy.¹³

In brief, the MMI circuit included the following: (1) a professional goals station at which standardized questions were asked related to candidates’ professional goals and suitability for the school; (2) a rapport and empathy station that assessed candidates’ ability to develop rapport and empathize with a standardized professional in a simulated free clinic setting; (3) an ethics station that presented candidates with an ethical dilemma requiring a course of action, assessed by a standardized professional; and (4) a professionalism station staffed by pharmacy students who asked candidates to discuss pros and cons of a proposed process to identify unprofessional online behavior by students.

Interviewers and Evaluation Process

Interviewers included the school’s faculty members and students and standardized professionals (paid actors) with experience in the conduct of OSCEs at the university. Of note, participation as a standardized professional did not require specific knowledge about pharmacy or health care. Student interviewers consisted of volunteers from the third-year and fourth-year PharmD classes. Faculty interviewers consisted of volunteers from the clinical and basic science departments.

Faculty interviewers participated a mandatory 2-hour training session conducted by faculty members on the admissions MMI workgroup. Students and standardized professionals participated in a similar 3-hour training session to allow for a more detailed discussion of the overall admissions process and extra time for scenario based role-playing practice and feedback from faculty members on the admissions MMI workgroup. All interviewers participated in a group training session specific to their assigned case that included detailed instructions for completing the electronic evaluation form.

Evaluators (faculty, students, and standardized professionals) were given 2 minutes to score each candidate immediately after the encounter in the following areas:^8,13 communication skills, strength of discussion/argument, suitability for the program, and overall performance. Response options ranged from unacceptable (coded 1) to exceptional (coded 10). To assist members of the admissions committee during the final review phase, a 2-page summary MMI report was created for each candidate.

The summary report included an overall mean performance score for the 4 domains as well as performance scores for the individual stations. For comparison purposes, aggregated performance data for all candidates on a given interview day were also computed and included in the MMI summary report. Digital videos of all MMI encounters were made available to the admissions committee members through a secure web-based portal. As part of the final review process, committee members examined the PharmCAS application materials (ie, personal statement, academic transcripts, letters of recommendation, extracurricular activities, work experience), the UCSF supplemental application materials, the handwritten onsite essay, and the MMI summary report and videos (when needed).

Surveys

The data presented here derive from a series of 3 voluntary and anonymous web-based surveys, administered electronically using Qualtrics (Qualtrics, LLC, Provo, UT) to 3 groups: candidates, interviewers, and admissions committee members. This research was deemed exempt by the UCSF Committee on Human Research.

Prior to launch, the instruments were reviewed by faculty members (n=5) and admissions staff (n=2), including one faculty member with expertise in survey design and measurement. Using an e-mail recruitment approach, candidates received surveys approximately 1 week after their onsite interviews, but prior to receiving notification of their final admission status. Interviewers received surveys approximately 4 weeks after completing candidate interviews, and admissions committee members received surveys after the final admissions committee meeting. All stakeholders received a reminder e-mail approximately one week after the survey launch.

The surveys contained 7 to 10 items assessing stakeholder-specific demographic data and perceptions of the MMI interview process. Response options and corresponding coding included: “strongly agree” (1), “agree” (2), “neither agree nor disagree” (3), “disagree (4), and “strongly disagree” (5). Additionally, all respondents were asked to rate the extent to which they perceived MMIs to be an effective approach for evaluating an applicant’s nonacademic qualities for the pharmacy profession, and whether they believed that the school should continue to use the MMI format instead of the standard interview.

Candidates indicated their sex, previous experience with pharmacy school interviews, and exposure to the MMI process in the preceding 6 months. Candidate-specific perceptions of the MMI process were assessed using the following survey items: (a) “The 15 minute instruction session prior to the MMI adequately prepared me for the experience;” (b) “Compared to a standard interview, the MMI is less stressful;” (c) “I had sufficient time to communicate my thoughts and ideas in the MMI stations;” and (d) “The MMI format process allowed me to present an accurate portrayal of my abilities.”

Interviewers indicated their academic status (faculty member, student, or standardized professional) and previous interview experience (first-time participant, participated in 2-5 admissions cycles, 6-10 admission cycles, or >10 admissions cycles). Interviewer-specific perceptions of the MMI process were assessed using the following survey items: (a) “The training session prior to the MMI adequately prepared me for the experience;” (b) “The instructions given to the candidates before my station appeared to be clear;” (c) “The duration of each MMI station was adequate;” and (d) “I was able to formulate an accurate assessment of the candidate’s abilities.”

Admissions committee members indicated their academic status (faculty member or student), previous committee experience (first-time participant, participated in 2-5 admissions cycles, 6-10 admissions cycles, or >10 admissions cycles), and perceptions of the MMI format. Committee member-specific perceptions of the MMI process were assessed using the following survey items: (a) “I was able to formulate an assessment of the candidate’s communication skills using the MMI score;” (b) “Review of the MMI videos was helpful in determining the candidate’s final rating score;” (c) “In general, my assessments of candidates’ performance (after reviewing the MMI videos) were consistent with the interviewer’s numerical scoring;” and (d) “Information on the interview evaluation form (used in previous cycles) was more useful than that obtained via the MMI (scoring sheet and MMI videos) for rating the candidate’s file.”

Costs and Statistical Analysis

Costs were estimated using human and facility resource hours (training and staffing schedules) from the 2012 cycle (standard interview format) and the 2013 cycle (MMI format). For the standard interview format, faculty hours required for training (2 hours/person), application file review (45 minutes/file), and actual interview times were included when calculating total cost. For the MMI format, faculty hours required for training (2 hours/person), actual interview time (6-7 hours/person), MMI coordination time (6-7 hours/person), standardized professional training (3 hours/person), interview time (6-7 hours), and facility-staffing fees (7 hours for weekend interviews) were included in calculations. Costs were estimated as follows: faculty members, $82.80 per hour (average salary and benefits for full-time paid faculty members); standardized professionals, $22.00 per hour; and SCSC staff, $135.00 per hour (weekend rates). Faculty time required for the development of the MMI stations and student time for participation in the interview process were not included in the cost analysis. Statistical analyses involved computation of summary statistics to characterize the survey responses. Analyses were conducted using SPSS, v21 (IBM, Armonk, NY).

RESULTS

Of 285 candidates who received a survey, 244 responded (86% response rate); of these, 65% were female, and 84% had participated in a pharmacy school admissions interview (3% in an MMI) in the previous 6 months. Table 1 presents candidates’ perceptions of the MMI approach. The majority of candidates agreed (86%) that the 15-minute instruction session prior to the MMI was adequate preparation for the experience. When asked if the MMI was less stressful compared to a standard interview, 34% agreed and 43% disagreed with this statement. In terms of having adequate time to convey their thoughts and ideas in the MMI stations, similar proportions of candidates agreed (39%) and disagreed (38%). Furthermore, 37% of candidates agreed that the MMI process allowed them to present an accurate portrayal of their abilities (vs 32% who disagreed). Sixty-five percent perceived the MMI format to be effective for evaluating an applicant’s nonacademic qualities for the pharmacy profession, and 59% agreed that the school should continue to use the MMI format instead of the standard interview (Figure 1).

Table 1.

Perceptions of The MMI Approach: Candidates, Interviewers, and Admissions Committee Members.

Figure 1.

Stakeholder Perceptions Regarding Continued Use of the MMI.

Of 53 interviewers who received a survey, 44 responded (83% response rate: 23 students, 14 faculty, and 7 standardized professionals); of these, 48% had prior experience serving as an interviewer for the School of Pharmacy Admissions Committee, with 21% serving for 6 or more admissions cycles. Interviewers’ perceptions of the MMI approach are found in Table 1. Interviewers agreed that training prior to the MMI experience was adequate (86%), the instructions provided to the applicants before their station were clear (80%), and the duration of each MMI station was adequate (77%). Nearly three quarters (73%) of interviewers agreed that they were able to formulate an accurate assessment of the applicants’ abilities. Most (86%) perceived the MMI format to be effective for evaluating an applicant’s nonacademic qualities for the pharmacy profession, and 84% agreed that the school should continue to use the MMI format instead of the standard interview (Figure 1).

Of 39 admissions committee members who received a survey, 34 responded (87% response rate; 4 students, and 30 faculty members). Most (82%) had served as an admissions committee member in previous years, with 53% serving for 6 or more admissions cycles. Committee members’ perceptions of the MMI approach are presented in Table 1. When evaluating applicants, 88% reported that they had watched MMI videos, and 97% agreed that this information was helpful in determining final rating scores. More than half (62%) agreed that they were able to formulate an assessment of the candidate’s communication skills using the MMI scores. When asked whether the information from the standard interview evaluation form (used in previous cycles) was more useful than that obtained via the MMI for rating the candidate’s file, 35% agreed and 48% disagreed. Most (79%) perceived the MMI format as effective for evaluating an applicant’s nonacademic qualities, and 77% agreed that the school should continue to use the MMI format instead of the standard interview (Figure 1).

For the standard interview process, 180 hours was required to train the interviewers (63 hours) and for interviewers to review the candidate application files in preparation for the interviews (117 hours). In contrast, the MMI process did not require interviewers to review the candidate files but required a total of 64 hours for the training of faculty interviewers (40 hours) and standardized professionals (24 hours). During the candidate interview phase, 278 faculty hours were required in the standard interview process vs 349 hours, which included faculty members (148 hours) and standardized professionals (201 hours) for the MMI process.

Additional fees were required with the MMI process for use of the SCSC on the interview day to cover over-time pay for technical staff ($945.00). The total estimated cost for standard interviews was $37 903 vs $21 461 for MMIs. Adjusting for the total number of applicants each year (standard interviews, n=278; MMIs, n=285), the total cost per applicant was $136.34 for the standard interviews and $75.30 for MMIs. The cost difference between the standard and MMI format was $61.04, approximately a 45% reduction.

DISCUSSION

This study characterized perceptions of the MMI approach across 3 stakeholder groups involved in the pharmacy school admissions process and included a comparative cost analysis. Similar to previous studies from other disciplines,^10,19-21 feedback from stakeholder groups was largely favorable regarding the MMI process, although among the groups, the candidates were the least positive. However, the majority of respondents in all 3 groups perceived the MMI format as an effective approach for evaluating nonacademic qualities for the pharmacy profession, and nearly 3 quarters of all respondents supported the continued use of the MMI format at UCSF instead of standard interviews.

Most candidates had no previous experience with the MMI process because, at the time, UCSF was the only school in California that had adopted this interview format. Unfamiliarity with the MMI process might explain why 43% of candidates did not agree that the MMI format was less stressful than the standard format. However, when the MMI approach was incorporated into the medical school admissions process at McMaster University, Eva and colleagues reported that multiple stations with different interviewers might enable candidates to recover from poor performances.⁸

Although many candidates perceived the MMI stations to be too short, interviewers believed that the time allotted was appropriate. This is consistent with the experience of Dodson and colleagues, who reported that reducing the length of the MMI stations from 8 to 5 minutes had minimal effects on the final candidate rating.²² Likewise, a pilot study at Leslie Dan Faculty of Pharmacy in Toronto found that the optimal MMI station length was 7 minutes.⁵ Considering this was the first year that the MMI approach was used at UCSF, refinement over time will likely enhance the quality of the process. As the MMI develops a stronger presence in the admissions process at other institutions, it is anticipated that candidates will perceive lower levels of stress.

Although the primary goal of implementing MMIs at UCSF was not fiscally driven (ie, the goal was to standardize and improve quality of information obtained during onsite interviews), the comparative cost analysis revealed meaningful differences in the cost per applicant, illustrating cost efficiency with the MMI approach. The MMI format required lower levels of faculty time spent in training, reviewing files, and interviewing candidates. Moreover, by using standardized professionals, fewer faculty hours were needed for the interviews, which released faculty time for other activities, such as leading innovative teaching and patient care efforts and enhancing grantsmanship and scholarship outcomes.

The MMI format requires fewer person-hours of effort, is at least as cost efficient as other in person interview formats, and does not require more interviewers.^19,23 There are, however, initial investments associated with the MMI format, specifically with respect to case development time and implementation. Nonetheless, once implemented, similar cases can be used for subsequent admissions cycles, thereby reducing future development time. The university has an affordable onsite simulation center that provides an ideal setting for the MMI interviews. Costs would vary among pharmacy schools because infrastructures and access to resources differ.

From the admissions committee’s perspective, data from this tri-population evaluation were sufficiently positive to warrant continued use of the MMI approach at UCSF, and the MMI format afforded a cost-efficient alternative to the standard interview format. Strengths of the study include strong survey response rates and assessment of perceptions from multiple key stakeholder groups. Limitations include the possibility of candidates responding positively to the MMI format, believing negative feedback might influence their final admission outcome, as well as limited generalizability because data were derived from one cohort, at one institution. Furthermore, our pilot phase was limited to 4 MMI stations, and our study design did not afford an opportunity to determine whether the standard interview approach would have yielded different admissions outcomes for individual candidates than did the MMI approach.

This study advances current knowledge related to the use of MMIs within the pharmacy profession. Additionally, it provides insight into the relative costs associated with MMI implementation vs a standard interview approach. Future studies should focus on longitudinal assessments of student performance over time. The MMI format might be viewed as a viable selection tool for pharmacy school admissions, but whether it is a predictive tool for overall performance in pharmacy school and beyond is debatable. In medical schools it is predictive of performance—students accepted via an MMI process to medical school scored higher on a national licensing examination when compared to students who were not accepted via the same MMI process, but were ultimately accepted at other medical schools.¹¹

Additionally, MMI scores predict experiential performance, with higher MMI scores correlating with higher clinical clerkship performance ratings for medical students.¹⁶ Because the MMI format is used to a limited extent in pharmacy education and training,^5,9,13 additional studies are needed to validate the individual MMI stations and overall approach and to characterize predictive relationships with key outcome variables. Because our data were collected anonymously from candidates, prospective analyses are not possible. Future studies, if able to link MMI ratings with individual performance characteristics, might aid in identifying strengths and weaknesses of the MMI approach.

CONCLUSION

Most respondents perceived the MMI format as an effective approach in evaluating nonacademic qualities for the pharmacy profession. Furthermore, the MMI process was less costly per candidate interviewed when compared with traditional interview methods. These results support the continued use of the MMI format as part of the admissions process at our institution.