Application and Interview Features Used to Assess Applicant Qualifications for Residency Training

Abstract

Purpose:

To determine what factors residency program directors (RPDs) consider and what methods they use to assess applicants.

Methods:

Respondents ranked the importance of 27 applicant features within domains: academics/credentials, application features/program fit, involvement, professional experience, research/ teaching experience, and postgraduate year 1 (PGY-1) residency experience. Rank was assigned in an ordinal fashion (1 = most important feature). The domains were characterized by their importance (mean % ± SD) in selecting candidates for interviews. Participants characterized their screening process according to 8 application and 6 interview features and the corresponding applicant dimensions evaluated. RPDs rated the importance of 14 methods applicants used to communicate with the program and 3 methods by which references were obtained. A Likert scale was used for rating (4 = crucial features). The approaches the program used to evaluate 12 application features or interpersonal interactions were reported.

Results:

The most important application domain was application features/program fit (26.28 ± 19.11). The highest ranked application feature was program fit (2.04 ± 1.17). The applicant’s cover letter, recommendation letters, curriculum vitae, and interview meal were commonly used to assess communication and interpersonal skills, knowledge base, and experience. The most important communication venue was the on-site interview (3.95 ± 0.23). Recommendations solicited by RPDs (3.42 ± 0.69) were most important. Programs formally evaluated the interview (89%) and recommendation letters (84%).

Conclusion:

Understanding the importance that RPDs place on application and interview features, as well as the process used to assess communication skills and interpersonal interactions, should allow residency candidates to become more competitive residency prospects.

The role of pharmacists in clinical health care practice is growing and evolving. As the number of available entry-level community pharmacy positions dwindles and opportunities for clinical specialists increase, more pharmacy students are choosing to pursue residency training. Furthermore, the American Association of Colleges of Pharmacy (AACP) and the American Society of Health-System Pharmacists (ASHP) support a vision in which, by the year 2020, all pharmacists involved in direct patient care will have completed a residency.^1,2

The pharmacy residency application environment is becoming increasingly competitive, thus students must prepare themselves for the recruitment process. An understanding of which application features are weighted most heavily by programs when they are selecting candidates for an on-site interview may greatly enhance an applicant’s successful pursuit of residency training. In addition to strengthening their application with leadership experience, a strong academic record, and challenging advanced pharmacy practice experiences (APPEs), applicants must also be aware of how their interpersonal communication and professional interactions with residency program directors (RPDs) influence their likelihood of being successful candidates for residency training. Although there are several objective measures (eg, grade point average [GPA], years of work experience, poster presentations) that allow RPDs to screen and stratify applicants, a separate realm of subjective measures aid RPDs in selecting candidates who are more compatible with their program and may be more likely to be successful in their program. Affective domain and emotional intelligence (EI) measures are becoming more prominent and valuable across many fields. Affective domain refers to traits such as self-efficacy, self-assessment, reflection, entrepreneurship, leadership, and advocacy; EI integrates emotional expression, self-awareness, and empathy as predictors of various facets of success.^3,4 Today’s health care practitioners must acquire additional competencies in the areas of empathy, coaching, problem solving, adaptability, critical thinking, and interpersonal communication to effectively contribute to an environment of interprofessional collaboration.⁵ How applicants can demonstrate their EI and how programs assess those domains are not fully understood. However, it is reasonable to believe that candidate review committees look beyond a paper application in order to adequately assess soft skills.

The 2013 residency match cycle saw a 6% increase in postgraduate year 1 (PGY-1) candidates and a 10% increase in postgraduate year 2 (PGY-2) candidates. The trend continued into the 2014 match cycle with a 5% increase in the number of PGY-1 candidates and a 20% increase in PGY-2 candidates. On March 21, 2014, 2,640 PGY-1 positions were filled by the residency match program (RMP) with 1,502 candidates left to scramble for 222 unfilled positions. A total of 446 PGY-2 positions were filled by the RMP, leaving 211 candidates to scramble for 86 unfilled positions.⁶ This capacity imbalance results in an increasing number of applications each year.

The 2013 residency match cycle was the first to utilize the Pharmacy Online Residency Centralized Application Service (PhORCAS). It is now more convenient for applicants to apply to an increasing number of programs, albeit at an increased financial burden. Programs are able to sort applicants based on a variety of characteristics to streamline their interview invitation and ranking process. The impact of the new application process must not be overlooked when considering the application and match figures. While the true effects of PhORCAS on the 2013 and 2014 RMP results have yet to be delineated, it is reasonable to conclude that the streamlined application process will continue to improve both the submission and review process.

The purpose of this 2-phase study was to elucidate how pharmacy RPDs in one state evaluate residency applicants’ qualifications and which features they consider to be the most important. In phase I of the study, we sought to determine which factors RPDs, both PGY-1 and PGY-2, take into consideration when inviting residency applicants for on-site interviews. In the second phase of the study, we sought to determine the venues RPDs use to judge a candidate’s communication skills, interpersonal interactions, and other soft skills and how they score such traits.

Methods

RPDs from Kentucky received an e-mail invitation to participate in a series of institutional review board (IRB)–exempt, pretested electronic surveys; the first was distributed after the March 2011 national residency match and the second survey was distributed immediately prior to the March 2012 match. The surveys targeted RPDs who were members of the Kentucky Pharmacy Residency Network (KPRN), a collaborative of ASHP-accredited residency training programs from across the Commonwealth of Kentucky. Members are representatives from academia, community pharmacy, community (not-for-profit) health systems, for-profit health systems, government health systems, and university health systems.⁷

Phase I

In response to an 18-point questionnaire, participants ranked the relative importance of 27 application features within 6 major domains: academics/credentials (eg, GPA, state licensure eligibility), application features/program fit (eg, letters of recommendation, professional goals), professional and civic involvement (eg, formal leadership positions), professional experience (eg, pharmacy employment, APPE rotations), research/teaching experience (eg, evidence of scholarly contributions), and previous residency experience for PGY-2 programs only (eg, reputation of PGY-1 program).Table 1 displays the domains and application features evaluated by the RPDs. Rank was assigned in an ordinal fashion (eg, in a domain with 5 individual features; 1 = most important feature and 5 = least important feature). Additionally, respondents assigned a percentage to each domain to indicate its relative importance in their selection of an applicant for an on-site interview (eg, academics/credentials composing 40% of an applicant’s suitability for an interview and professional experience composing 20% of an applicant’s suitability) as displayed in Table 2. After all information was gathered, descriptive statistics (mean ± SD) were used to evaluate the findings.

Table 1. Residency application features considered for on-site interview selection.

Academics/credentials (ranked in ordinal fashion where 1 indicates the most important feature and 7 the least important feature)

All degrees held

Certifications

College of pharmacy attended

Honors/awards received

Pharmacy grade point average

Pharmacy licensure eligibility

Other

Application features/program fit (ranked in ordinal fashion where 1 indicates the most important feature and 5 the least important feature)

Letters of recommendation

Perceived program fit

Professional goals

Verbal/written communication skills

Other

Involvement (ranked in ordinal fashion where 1 indicates the most important feature and 5 the least important feature)

Community service

Leadership roles

Nonpharmacy organizational involvement

Pharmacy organizational involvement

Other

Professional experience (ranked in ordinal fashion where 1 indicates the most important feature and 4 the least important feature)

Nonpharmacy work experience

Pharmacy student rotations

Pharmacy work experience

Other

Research and teaching experience (ranked in ordinal fashion where 1 indicates the most important feature and 7 the least important feature)

Grantsmanship

Poster presentations

Professional presentations

Publications

Research project

Teaching experience

Other

PGY-2 applicant’s PGY-1 residency training (ranked in ordinal fashion where 1 indicates the most important feature and 5 the least important feature)

Extent of experience PGY-1 gained in specialty area

“Reputation” of PGY-1 program director

“Reputation” of PGY-1 program

Setting of PGY1 program

Other

Note: PGY-1 = postgraduate year 1; PGY-2 = postgraduate year 2.

Table 2. Relative importance of application domains.

Application domain	Assigned weight
Academics/credentials
Application features/program fit
Involvement
Professional experience
Research and teaching experience
PGY-2 applicant’s PGY-1 residency training
Total	100%

Note: PGY-1 = postgraduate year 1; PGY-2 = postgraduate year 2.

Phase II

As prompted by the phase II questionnaire, RPDs of the same residency network were invited to characterize their candidate screening process according to 8 application features and 6 interview features and the corresponding dimensions of an applicant (communication skills, interpersonal skills, knowledge base, and experience) that each feature evaluated. Interpersonal skills were defined as traits such as confidence, poise, assertiveness, nonverbal cues, self-awareness, and mannerisms. Communication skills referred to both oral and written communication. Next, the RPDs ranked application and interview features in an ordinal fashion by response rate within each of the 4 dimensions. RPDs rated the importance (crucial, important, somewhat important, and not considered) of 14 methods (eg, interaction at a residency showcase or interview, correspondence following a showcase or interview, and interactions through APPE rotations) by which applicants communicated with the program and 3 means (unsolicited verbal recommendations, solicited verbal recommendations, and applicant-solicited formal recommendations) by which candidate recommendations were obtained. RPDs then reported the approaches (formally evaluated, informally evaluated, evaluated based on performance extremes, other, or not evaluated) their program used to assess 12 application features or interpersonal interactions. Formal evaluation referred to features assessed using a specific scoring or rating system. Informal evaluation was defined as general consideration; there was no defined unit of measure to score the applicant. Respondents could also report characteristics that were evaluated based on performance extremes or exceptionally positive or negative performance. To quantify the relative importance of communication methods and recommendation solicitation methods, a Likert scale (4 = crucial, 3 = important, 2 = somewhat important, and 1 = not considered) was applied, allowing the investigators to analyze the data via descriptive statistics (mean ± SD). The application features and interactions were ranked in an ordinal fashion by response rate for each mode of assessment.

Respondents to both questionnaires were able to note additional comments in open-ended text fields following each question. Demographic data were collected with each phase to allow for stratification of results.

Results

Demographic data are represented in Table 3. Thirty-one RPDs received an e-mail invitation to participate in the 2011 survey; 14 PGY-1 and 11 PGY-2 responded (81% response rate). Forty-eight RPDs were invited to participate in phase II of the survey; 14 PGY-1 and 5 PGY-2 program directors consented to participate (40% response rate). The decline in response rate from phase I to phase II may be related to the timing of the solicitation relative to the residency cycle (the 2011 survey was sent following the results of the PRM while the 2012 survey was sent immediately prior to match day).

Table 3. Demographics of survey respondents.

Characteristic	Phase 1			Phase 2
	Total (n = 25)	PGY-1 (n = 14)	PGY-2 (n = 11)	Total (n = 19)	PGY-1 (n = 14)	PGY-2 (n = 5)
Years program in existence, mean (range)	9.24 (0-42)	10.79 (1-42)	7.27 (0-25)	10.26 (1-40)	10.93 (2-40)	8.40 (1-30)
No. of applicants, mean (range)	14.96 (0-100)	22.71 (4-100)	6.22 (0-6) (n = 9)	22.06 (0-104)	25.13 (0-104)	7.20 (1-25)
Applicants interviewed, mean (range)	7.64 (0-24)	11.50 (3-24)	3.33 (0-6) (n = 9)	10.63 (0-36)	13.36 (0-36)	3.00 (0-8)
Resident positions available, mean (range)	1.80 (0-6)	2.42 (1-6)	1.10 (0-2) (n = 10)	2.89 (0-9)	3.50 (2-9)	1.20 (0-2)
Resident positions filled by early commitment, mean (range)	N/A	N/A	0.60 (0-2) (n = 10)	N/A	N/A	0.40 (0-2)
Program accreditation status, n (%)
Accredited	16 (64%)	10 (71%)	6 (55%)	15 (79%)	13 (93%)	2 (40%)
Preliminary accreditation	2 (8%)	0	2 (18%)	2 (11%)	1 (7%)	1 (20%)
Candidate	5 (20%)	3 (21%)	2 (18%)	2 (11%)	0	2 (40%)
Pre-candidate	2 (8%)	1 (7%)	1 (9%)	0	0	0
Practice setting, n (%)
Academia/college or university	3 (12%)	0	3 (27%)	2 (11%)	1 (7%)	1 (20%)
Community pharmacy	1 (4%)	1 (7%)	0	1 (5%)	1 (7%)	0
Community (not-for-profit) hospital/health system	7 (28%)	2 (14%)	5 (45%)	8 (42%)	7 (50%)	1 (20%)
For-profit hospital/health system	9 (56%)	8 (57%)	1 (9%)	2 (11%)	1 (7%)	1 (20%)
Government hospital/health system	4 (16%)	2 (14%)	2 (18%)	2 (11%)	2 (14%)	0
University hospital/health system	1 (4%)	1 (7%)	0	4 (21%)	2 (14%)	2 (40%)

Note: PGY-1 = postgraduate year 1; PGY-2 = postgraduate year 2.

Phase I

According to the PGY-1 RPDs who were surveyed, they weighted the broad domain of application features/program fit most heavily when selecting residency candidates for an on-site interview (27.29 ± 12.95). Involvement was considered to be the second most highly valued domain amongst first year programs (20.36 ± 4.99). Conversely, PGY-1 residency training features was the domain considered to be most valuable by PGY-2 RPDs (26 ± 8.76), followed by application features/program fit (25 ± 25.59).

Within the domain of academics/credentials, pharmacy GPA was the most important feature overall in both PGY-1 and PGY-2 RPD responses (2.44 ± 1.36). This was also true of PGY-1 program directors as a subset (2.21 ± 1.13), however PGY-2 program directors as a subset placed a slightly greater emphasis on honors/awards received by the candidate (2.54 ± 1.21). Perceived program fit was the most important component within the application features/program fit domain amongst the combined group of respondents (2.04 ± 1.17). Again, this result was consistent with the responses of the PGY-1 group (1.50 ± 0.85) but not the PGY-2 group. Within this domain, PGY-2 RPDs cited verbal and written communication skills to be the most important (2.00 ± 1.00). Of the features posed within the professional experience domain, the combined group of respondents chose pharmacy student rotations (1.56 ± 0.77) as the most valuable, as did the subset of PGY-2 RPDs (1.27 ± 0.47). Pharmacy work experience, on the other hand, was more heavily considered amongst PGY-1 respondents (1.50 ± 0.52). Within the research and teaching experience domain, professional presentations were highly regarded (2.68 ± 1.14). Results from PGY-1 RPDs were similar (2.57 ± 1.16), however PGY-2 RPDs placed a higher emphasis on the applicant’s research project (2.91 ± 0.36). Leadership positions were considered to be the most important feature within the involvement domain overall (1.56 ± 1.08) and amongst PGY-1 RPDs (1.71 ± 1.20) and PGY-2 RPDs (1.36 ± 0.92). When asked to identify which feature within the PGY-1 residency training domain they considered to be most important, PGY-2 program directors selected the reputation of the PGY-1 program itself (1.54 ± 0.69).

Phase II

Of the application features tested, the applicant’s cover letter (100%), recommendation from a faculty member (74%), and recommendation from a clinical preceptor (63%) were the components most commonly used overall to assess communication skills. Although the combined findings were consistent with the responses of PGY-1 RPDs, PGY-2 RPDs as a subgroup placed a stronger emphasis on the candidate’s application essay (60%) and curriculum vitae (60%) in addition to the cover letter (100%) when assessing this dimension. Letters of recommendation from a clinical preceptor (90%), faculty member (84%), and employer (74%) were the application features most frequently used to assess interpersonal skills. To evaluate a candidate’s knowledge base, respondents reported letters of recommendation from a clinical preceptor (90%) and faculty member (84%) and the curriculum vitae (74%) to be the most useful application components. As a gauge of pharmacy experience, RPDs most commonly used the curriculum vitae (100%), letter of recommendation from a preceptor (79%), and letter of recommendation from an employer (74%). PGY-1 and PGY-2 RPDs responded similarly in their assessment of the 3 preceding domains.

A meal between the candidate and preceptors/ residents was the venue most commonly used to evaluate communication skills (95%), interpersonal skills (90%), and experience (32%). PGY-2 RPDs also placed a heavy emphasis on a formal presentation given during an interview in evaluating a candidate’s communication skills (80%) and knowledge base (80%). An in-person screening interview (eg, Personnel Placement Service) prior to an on-site interview was often used by PGY-2 programs in assessing communication skills (60%), interpersonal skills (60%), and pharmacy experience (60%). PGY-1 programs, conversely, rarely used the additional interview features included in the survey (eg, journal club, case presentation, phone screening) for the purpose of elucidating information regarding their candidates’ communication skills, interpersonal skills, knowledge base, or pharmacy experience.

The venue considered to be most important overall for communication between an applicant and RPD was the on-site interview (3.95 ± 0.23), followed by student rotations at the residency site (3.68 ± 0.48) and employment at the residency site (3.47 ± 0.90). Directors of PGY-2 programs also highly rated interactions with candidates through professional organizations (3.40 ± 0.55) and meetings (3.40 ± 0.89). Recommendations solicited by the RPDs (3.42 ± 0.69) were more useful information sources than those that were solicited by the applicants (3.32 ± 0.58) or were unsolicited (3.21 ± 0.85).

The interview (89%), letters of recommendation (84%), cover letter (79%), curriculum vitae (79%), and application essay (48%) were reported to be formally evaluated by both PGY-1 and PGY-2 programs. PGY-2 programs were more likely to formally evaluate an applicant’s prior performance at the training site than PGY-1 programs (80% vs 29%, respectively). Resident performance through correspondence following the on-site interview (74%) and residency showcase (74%) was commonly evaluated using informal methods by both PGY-1 and PGY-2 RPDs. PGY-1 programs reported informal evaluation of interactions at a residency showcase as common practice (86%), whereas such interactions were less commonly evaluated by PGY-2 RPDs (40%). It was uncommon for programs to utilize performance extremes as a means of evaluating candidates.

Discussion

Our surveys were designed to determine the value RPDs place on various application and interview features when assessing applicants. RPDs can use these results to compare their assessment methods with those of their colleagues and perhaps improve their candidate screening methods. Pharmacy students and PGY-1 residents can utilize the results to better position themselves in the increasingly competitive recruitment process. Finally, colleges of pharmacy may benefit from the results presented in order to improve their method of preparing students for the residency application process and interviews. For example, a previous study demonstrated that students who complete a course or seminar specifically targeted toward residency preparation recognized improvements in their curriculum vitae, letter of intent, and confidence in navigating the residency application and interview process.⁸

Our study identified pharmacy GPA, perceived program fit, leadership involvement, student rotations, and professional presentations as highly weighted factors in interview offer decisions. Results of a previous study showed that in addition to a candidate’s academic record, their letters of recommendation, curriculum vitae, and letter of intent were the most commonly used application components in determining which candidates to invite to an on-site interview (these documents can be used to collect data on the key features identified in our study).⁹ It is reasonable to conclude that each of these application features are of great value and should be a focus of residency candidates as they proceed through the application process.

Also critical to the success of a residency applicant is the demonstration of interpersonal and communication skills. Jellinek-Cohen and colleagues identified the importance of the applicant’s ability to learn, listen, and work with a team.¹⁰ The current study sought to determine how such skills are evaluated by RPDs. We found that an applicant’s letter of intent, letters of recommendation, and a meal between candidates and program representatives are the most impactful means of assessing such qualities and thus should be a focus of selection committees.

Candidates should be prepared to engage in a meal with preceptors and current residents prior to, during, or after their on-site interview. This venue can be very helpful in assessing soft skills. While the meal portion of the interview is not likely to be formally evaluated, interactions and etiquette displayed can leave a lasting impression and significantly impact the candidate’s standing. Regardless of the level of formality of the meal, applicants must be aware that they are still on an interview and should conduct themselves as such. Case presentations, journal club presentations, or other formal presentations are infrequently employed by programs but can also be useful forums in which to evaluate communication skills and the candidate’s knowledge base.

Applicants can expect that their letter of intent, curriculum vitae, response to written application questions, and letters of recommendation will have a large impact on their candidacy, particularly as a means of demonstrating their communication skills, interpersonal skills, knowledge base, and experience. Applicants should use these forums to express their motivation, leadership qualities, and interest in the program in addition to conveying how the program fits with their career goals. However, letters of intent and letters of recommendation can be difficult to measure objectively and thus may not be as helpful in differentiating or scoring candidates. An applicant’s GPA, pursuit of graduate education (eg, MBA dual degree enrollment), and previous degree(s) awarded can also impact the overall impression he or she makes on the program representatives.

Interactions during an on-site interview and submitted application components are formally evaluated by programs as part of the applicant assessment process, but many other interactions are often informally evaluated. These interactions include communication prior to or following an interview or residency showcase and prior performance at the residency site through employment or student rotations. According to respondents of this study, handwritten thank-you notes following the on-site interview were considered slightly more important than e-mailed notes or typed notes. The same trend was seen when considering notes received following a residency showcase.

Our findings warrant distribution of the survey to RPDs nationwide to determine whether the results differ by demographic subsets (eg, program size, practice setting, accreditation status, and years program in existence), as one of the limitations of our study was the small sample size. There are a variety of residency program types, settings, and sizes that are represented in these findings, but we cannot assert that they are fully representative of the national spread of residency programs. Additionally, candidate assessment approaches may change as the national residency application numbers and program capacity figures evolve.

The impact of the PhORCAS on candidate evaluation is unknown. One hundred percent of eligible residency programs (any program that has applied for accreditation with ASHP and planned to participate in the RMP) took part in PhORCAS during its first year of existence, and the number of participating programs and applicants is expected to grow each year. RPDs are now able to search candidates by numerous criteria, such as nonpharmacy degree, pharmacy college attended, pharmacy GPA, PGY-1 site, extracurricular activities, reference recommendations, or a combination of any 3 criteria.¹¹ References are asked to evaluate 12 characteristics of the applicant, including oral and written communication skills, professionalism, emotional stability and maturity, effective patient interactions, and assertiveness when completing the standardized reference form.¹² With the wealth of information provided in a PhORCAS application, residency programs may need to streamline their application review process. Ensor and colleagues identified a list of 7 application variables highly associated with extension of an interview offer at their institution. The list included pharmacy work experience, professional association involvement, rotation experiences, publications, presentations, skills and certifications, and research experience. Letters of recommendation and letters of intent were notably eliminated in the expedited review approach, thus removing information sources identified as critical by the RPDs surveyed in our study.¹³ Further research is needed to determine how PhORCAS affects the manner in which RPDs assess candidates, the sources of information they utilize, and the applicant characteristic they extrapolate from the information available.

Another potential change to residency applicant selection is the introduction of new interview styles, such as the multiple mini-interview (MMI) approach, that are designed to better assess various soft skills. The MMI interview consists of a series of short interviews with a number of different interviewers, each intended to assess a distinct noncognitive attribute (eg, advocacy, empathy, self-assessment, and cultural sensitivity). In each interview, the applicant is presented with a scenario that he or she will discuss with the interviewer(s) over 8 to 10 minutes. Studies of MMI in the medical school admissions process have provided evidence for the validity and reliability of the interview method.¹⁴ This interview approach was first piloted in the pharmacy residency interview process at an academic medical center during the 2013-2014 residency cycle.¹⁵ Another novel approach for screening applicants is the use of Web-based teleconferencing to conduct a mini-interview prior to extending an on-site interview offer. For the applicant, this technology provides an additional opportunity to interact with program leadership and identify whether the program is a good fit for them. The program receives the same benefit, in addition to allowing preceptors to better balance their time between assessing candidates and maintaining their patient care and administrative duties. This approach also gives programs the opportunity to better assess soft skills prior to bringing the candidate on-site for an interview.¹⁶ As competition for available positions increases and technology evolves, pharmacy residency applicants will need to be prepared for and able to adapt to alternative assessment approaches.

Conclusion

A variety of application and interview features are used by residency programs, each intended to evaluate various dimensions of an applicant. PGY-1 and PYG-2 program directors place different emphasis on many aspects of the residency application. Applicants should be aware that both informal interactions with RPDs and formal application and interview components can influence their standing. Understanding the process RPDs use to assess communication skills and interpersonal interactions should help residency candidates to become more competitive residency prospects.