Characteristics That Define a Successful Pharmacy Resident as Perceived by Residency Programs

Abstract

Purpose:

To identify the subjective and objective characteristics that pharmacy residency programs use to define a successful resident and to determine what percentage of their 2009-2010 residency class they felt were successful.

Methods:

An electronic survey was sent via e-mail to all residency program directors (RPDs) of postgraduate year 1 (PGY1) and postgraduate year 2 (PGY2) pharmacy residency programs in the United States. A 3-part survey instrument was developed following validation of questions for clarity and reliability using a pilot survey. Respondents were asked to rank the importance of 20 subjective characteristics for a resident to possess in order to be considered successful and the importance of different objective measurements of accomplishment in the definition of a “successful” resident using a Likert scale where 1 = not at all important, 2 = some importance, 3 = very important, and 4 = critical.

Results:

Of the 1,081 surveys sent to RPDs, 473 respondents answered at least one question, yielding a response rate of 43.8%. The most critically important subjective characteristics in defining a successful resident as ranked among PGY1 residency programs are dependability, professionalism, self-motivation/initiative, and work ethic. PGY2 programs ranked clinical knowledge and skills, critical thinking, and dependability as the most important. The most critically important objective characteristic in defining a successful resident as ranked among both PGY1 and PGY2 programs is obtaining a clinical position. The majority of PGY1 and PGY2 respondents felt that 76% to 100% of their 2009-2010 residency class was successful based on the characteristics they rated most important.

Conclusion:

Identification of the characteristics that pharmacy residency programs use to define success will allow them to identify predictors of success and optimal methods of selecting residents who possess these characteristics.

Pharmacy has seen an increase in applicants seeking postgraduate residency training. This increase can be expected to continue, as professional organizations advocate that all pharmacists providing direct patient care be required to complete a residency by the year 2020.¹ However, unlike the medical profession, current pharmacy literature has not addressed methods for selecting and ranking candidates who will be successful in achieving the outcomes and goals of the residency program.

Residency programs have attempted to identify strategies that would ensure greater success in matching with the best applicants through the National Resident Matching Program. Several studies have attempted to determine which characteristics are predictive of success during residency as well as the optimal method of selecting residents who possess these characteristics.^2–5 Leonard and Harris⁶ suggest that the latter goal is elusive, particularly in the light of the problems associated with defining the ultimate criterion, “the good physician.” Medical literature has criticized academic criteria, letters of recommendation, and the applicant interview as being unreliable predictors of an applicant’s performance. ⁷ In one study, Borowitz et al⁸ concluded that the idea of past performance as the best predictor of future performance is only partially correct. Professional success during and following residency may be more accurately reflected by various personal and professional values, described in literature as the affective domain,⁷ that are difficult to quantify in the interview process. Poirier and Pruitt⁹ suggest that professional success is especially difficult to quantify because success is a very subjective achievement and has yet to be defined.

Due to the investment made in recruitment and the loss in productivity when a candidate is a poor performer, it is paramount for pharmacy residency programs like their medical counterparts to establish criteria that help them predict successful performers to ensure success of the match. We hypothesize that identification of the characteristics that pharmacy residency programs use to define the success of a resident will then allow them to identify these predictors and the optimal methods of selecting residents who possess these characteristics. Thus, the objectives of this study are to identify the subjective and objective characteristics pharmacy residency program directors (RPDs) use to define a successful resident and to determine what percentage of their 2009–2010 residency class these programs consider to be successful.

Methods

This study was approved by Maimonides Medical Center’s institutional review board. The subjective characteristics used in this study were determined by a typed-in answer, included in a previous survey,¹⁰ in which residency programs were asked to list the terms they believe define “successful.” The top 20 responses were included in this survey. An initial pilot survey was developed and administered to assess the validity of the survey instrument. The pilot survey was sent to 20 RPDs, coordinators, or preceptors of residency programs in New York City. Respondents were instructed to identify any confusing terms, misleading questions, or any other unclear aspects of the survey. Based on the feedback received, the survey instrument was revised to enhance ease of use and the quality of the data collected. The results of the pilot survey are not included in this article.

The final survey was divided into 3 parts. Part I included 6 demographic questions that all survey responders were required to answer if they consented to participate in the survey via a presurvey waiver. The 6 demographic questions focused on getting a better understanding of the experience level of the responder, the type and size of the facility in which the residency program was based, as well as the size (ie, how many residents) and location by state of the residency program. A question was then posed to survey responders to determine whether they were the program director of a postgraduate year 1 (PGY1) residency, postgraduate year 2 (PGY2) residency, or both. If responders chose PGY1, Part II of the survey was prompted. Part II consisted of 9 questions. If responders chose PGY2, Part III of the survey was prompted. Part III consisted of 10 questions. If responders indicated they were the director of both a PGY1 and PGY2 program, they were asked to complete both Part II and Part III (a total of 19 questions).

In Part II and Part III, 5 of the questions were demographic questions to ascertain American Society of Health-System Pharmacists (ASHP) accreditation status and participation in National Matching Services (NMS) or “Match.” PGY2 RPDs were asked an addition demographic question to determine which specialty their program offered. The remaining questions asked recipients to rate on a Likert scale (1 = not at all important, 2 = some importance, 3 = very important, 4 = critical):

• the importance of 20 subjective characteristics for a resident to possess in order to be considered successful, and

• the importance of different objective measurements of accomplishment in the definition of a “successful” resident.

They were also asked to assess the percent of the 2009–2010 residency class that they considered successful based on the characteristics that were rated most important. RPDs were encouraged to think retrospectively about what characteristics past residents who they felt had achieved success possessed and rate the questions accordingly.

The instrument was uploaded to Survey Monkey (www.surveymonkey.com), and an invitation including a brief explanation of the research project and a link to the survey was sent via e-mail to 1,081 PGY1 and PGY2 program directors of pharmacy residencies in the United States. We obtained the contact information from the online ASHP residency directory. 11 Clinical coordinators, clinical preceptors, and individuals listed as “other” were permitted to complete the survey instrument in lieu of the RPD, provided that they disclosed their position as it related to the residency program. Duplicate survey submissions were prevented by unique e-mail link identification via Survey Monkey. The survey was linked to the respondent’s e-mail address; once completed, the survey could not be completed again, even if the RPD had forwarded it to another person within the program to complete, because the link would be disabled. Additionally, prompts were embedded throughout the survey reminding recipients that the survey was to be completed only once by each PGY1 and PGY2 program. Completion of the presurvey waiver, which stated that the survey responses would be kept confidential and that data would be reported only in aggregate, served as consent to participate in the study. The survey instrument was sent via e-mail on March 3, 2011, and results were collected through June 30, 2011. Reminder e-mails were sent biweekly to individuals who did not complete the survey.

The estimated time to complete the online survey was stated as 15 minutes. No form of compensation was offered in exchange for survey completion. Survey results were downloaded from Survey Monkey and exported into IBM SPSS 15.0 (IBM, Armonk, NY) for analysis. The chi-square test was used to compare differences in demographic data for categorical variables among PGY1 and PGY2 programs. Independent one-sample t tests were used to compare differences in demographic data for continuous variables among PGY1 and PGY2 programs. Likert responses were categorized as either critical or not and analyzed using the chi-square test or the Fisher’s exact test. Levels of significance were tested at P < .05.

Results

Demographics

The survey was e-mailed to 1,081 RPDs; 478 respondents consented to participate in the study and 473 respondents answered at least one question, yielding a response rate of 43.8%. Of the 603 individuals who did not respond, 90 of the e-mails (14.9%) were undeliverable. The 473 respondents represented 313 PGY1 programs and 196 PGY2 programs. Forty respondents represented both PGY1 and PGY2 progams, resulting in the number of programs exceeding the number of respondents.

As seen in Table 1, most respondents were RPDs (96.5% PGY1 vs 95.9% PGY2) and were from the South (37.4% PGY1 vs 53.6% PGY2). Most reported practicing in teaching hospitals (45% PGY1 vs 27.9% PGY2), and the majority represented residency programs that had been accredited for 10 or more years (45.5% PGY1 vs 29.1% PGY2). The majority of PGY2 respondents specialized in critical care (29.1%).

Table 1. Demographics of survey respondents and residency program characteristics.

	PGY1 programs (n = 313)		PGY2 programs (n = 196)
Demographics	n	%	n	%	P
Title of the person completing the survey
Residency Program Director	302	96.5	188	95.9	.743
Clinical Coordinator	6	1.9	2	1	---
Clinical Preceptor	2	0.6	3	1.5	---
Other (eg, Director of Pharmacy)	3	1	3	1.5	.816
Years in this position
<5 years	163	52.1	100	51
≥5 years	150	47.9	96	49
Type of facility
Teaching hospital pharmacy	211	45	131	27.9	.001
Acute care (primarily)	188	40.1	131	27.9	.001
Private	69	14.7	90	19.2	.622
Non-teaching hospital pharmacy	47	10	23	4.9	.061
Chronic care (primarily)	42	9	8	1.7	.002
Community pharmacy	26	5.5	19	4.1	.707
Managed care pharmacy	15	3.2	7	1.5	.128
Location		4	0.9	.249
South	117	37.4	43	53.6	.001
Midwest	93	29.7	38	23.5	.124
West	57	18.2	44	14.3	.0248
Northeast	46	14.7	17	8.7	.045
Residents accepted per year (median)a	469	3	469	1	.001
Accreditation status
Accredited	266	91.1	135	80.4	.001
Candidate	14	4.8	25	14.9	.001
Pre-candidate	10	3.4	6	3.6	.934
Not seeking accreditation	2	0.7	2	1.2	.625
Years accredited
<1 year	25	8.6	31	19.6	.002
1–3 years	35	12	36	22.8	.019
4–6 years	56	19.2	30	19	.848
7–9 years	43	14.7	15	9.5	.345
≥10 years	133	45.5	46	29.1	.003
PGY2 specialty			196
Critical Care			57	29.1
Oncology			53	27
Pediatrics			29	14.8
Ambulatory Care			28	14.3
Infectious Diseases			25	12.8
Health System Pharmacy Practice Administration/MS			14	7.1
Internal Medicine			14	7.1
Solid Organ Transplantation			14	7.1
Cardiology			13	6.6
Pharmacotherapy			12	6.1
Psychiatry			12	6.1
Health-System Pharmacy Practice Administration			11	5.6
Drug Information			8	4.1
Geriatrics			6	3.1
Pharmacy Informatics			6	3.1
Managed Care			3	1.5
Medication Use Safety			3	1.5
Palliative Care/Pain Management			3	1.5
Critical Care/Nutrition Support			22	1
Emergency Medicine			2	1
HIV			2	1
Nuclear			1	0.5

Note: PGY = postgraduate year. Chi-square test was used to compare differences for categorical variables; independent one-sample t test was used to compare differences for continuous variables. Levels of significance were tested at P < .05.

^an = number of respondents who answered this question.

The following results are presented in Table 2 and include the percentage of responses ranked as critical for each characteristic and a statistical comparison of the characteristics between PGY1 and PGY2 programs.

Table 2. Survey respondents ratings of characteristics that are critically important for applicants to possess in order to become successful residents.

	PGY1 programs (n = 313)		PGY2 programs (n = 196)
Characteristics	n	%	n	%	P
Subjective characteristics
Professionalism	212	73.6	111	72.1	.729
Dependability	210	72.9	121	78.6	.191
Self-motivation/initiative	201	69.8	110	71.4	.720
Work ethic	201	69.8	116	75.3	.218
Clinical knowledge and skills	196	68.1	120	77.9	.029
Reliability	193	67	115	74.7	.095
Positive attitude	191	66.3	98	63.6	.572
Dedication	185	64.2	107	69.5	.267
Critical thinking	168	58.3	123	79.9	.001
Interpersonal skills	162	56.3	84	54.5	.731
Time management	162	56.3	88	57.1	.857
Team player	155	53.8	98	63.6	.047
Organization skills	136	47.2	71	46.1	.822
Independence	105	36.5	74	48.1	.018
Compassion	64	55.7	51	44.3	.013
Confidence	52	18.1	48	31.2	.002
Leadership	40	13.9	30	19.5	.125
Assertiveness	35	12.2	27	17.59	.121
Innovative	27	9.4	15	7	1.00
Communication skills	3	1	0	0	.555
Objective characteristics
Obtaining a clinical position	99	34.4	66	42.9	.079
Achieving board certification	24	8.3	38	24.7	.001
Obtaining a PGY2 position	22	7.7	N/A	N/A	N/A
Publishing a manuscript	15	5.2	20	13	.005
Obtaining a teaching position	11	3.8	14	9.2	.029
Holding a leadership role on a National committee	6	2.1	3	2	1.00
Obtaining a fellowship	4	1.4	0	0	.303
Receiving a grant	2	0.7	0	0	.545
Obtaining a research position	1	0.3	1	0.7	1.00

Note: N/A = not applicable; PGY = postgraduate year. Likert responses were categorized as critical or not and analyzed using the chi-square test or Fisher’s exact test. Levels of significance were tested at P < .05.

Subjective Characteristics

The most critically important subjective characteristics in defining a successful resident as selected among PGY1 residency programs are “dependability,” “professionalism,” “self-motivation/initiative,” and “work ethic.” PGY2 programs selected “clinical knowledge and skills,” “critical thinking,” and “dependability” as the most important. Comparing PGY1 to PGY2 program responses, PGY1 programs believe “compassion” is more important than PGY2 programs when defining a successful resident. PGY2 programs believe that “clinical knowledge and skills,” “confidence,” “critical thinking,” “independence,” and being a “team player” were more important than PGY1 programs in defining a successful resident.

Objective Characteristics

The most critically important objective characteristic in defining a successful resident as ranked among both PGY1 and PGY2 progams is “obtaining a clinical position.” Comparing PGY1 to PGY2 program responses, PGY2 programs believe “obtaining a teaching position,” “publishing a manuscript,” and “achieving board certification” were more important than PGY1 programs in defining a successful resident.

Percent Successful

The majority of PGY1 and PGY2 respondents felt that 76% to 100% of their 2009–2010 residency class was successful based on the characteristics they rated most important (219 [76.3%] PGY1 programs vs 132 [85.7%] PGY2 programs; P = .019).

Discussion

Based on the data from this survey, PGY1 residents who display dependability, professionalism, self-motivation/initiation, and good work ethic will be characterized as successful residents. PGY2 residents who are dependable, demonstrate clinical knowledge and skills, and are able to think critically will be characterized as successful. It is essential for applicants to demonstrate these behaviors during the interview process and for recruiters to assess residents in these areas.

The number of applicants that programs have to interview can be daunting. By identifying the characteristics that predict success in their program, recruiters can modify their application process to elicit information from applicants and references that will provide detailed examples of how the applicants have demonstrated these characteristics during past jobs or rotations. For example, in addition to requiring that applicants submit their curriculum vitae, a recruiter can request that they complete an application form that includes specific behavior-based questions. These would differ from traditional questions that explore potential behaviors; applicants often provide memorized answers that they think the recruiters want to hear. Questions about past behaviors could serve as a screening tool that centers on real-life experiences. The value of asking behavior-based questions is based on the concept that past behavior is a better predictor of success than potential behavior. Behavior-based questions are considered more pointed, probing, and specific than traditional questions.

Alternatively, the letter of intent can be used as a tool to demonstrate the applicant’s ability to display the characteristics of a successful resident. Recruiters can request that the letter of intent include examples of how the applicant has demonstrated particular characteristics. The same request can be made to persons writing letters of recommendation or completing applicant evaluations using Pharmacy Online Residency Centralized Application (PhORCAS). Rather than having a subjective rating of the applicant’s ability to exceed expectations on a particular characteristic, the recruiter would have a detailed description of how the applicant has demonstrated this behavior. This information might be compared to the information provided by the applicant and assessed for similarities and differences in perception of events. Behavioral or situational questions can be used to evaluate an applicant’s critical thinking skills, however, these rely on the recruiter’s instincts. Another approach is to use standardized critical thinking tests such as the Watson-Glaser II Critical Thinking Appraisal.

Residents who acquire a clinical position upon completion of the residency will be deemed a successful resident by PGY1 and PGY2 programs. PGY2 programs also prefer that their residents obtain teaching positions, publish manuscripts, and achieve board certification. Residency programs frequently ask applicants during the interview process to describe their future goals; a road map of what is deemed successful by programs is given to applicants during the interview.

Characteristics of effective performance during a pharmacy residency can be systematically and empirically determined through a process of a job analysis called the critical incident technique.3 The objective of this technique is to elicit from job experts (eg, faculty physicians) concrete and specific descriptions of behaviors that designate outstanding performance or a definition of what was required for successful resident performance. This information can then be used to develop a job analysis with mutually exclusive categories of performance that are defined by actual incidents of behavior. Medical residency programs have used this technique to design forms used during the residency interview and a selection process that evaluate these characteristics.12 Tarico et al13 reported that scores on interviews using techniques based upon critical behavioral descriptors for candidates correlated well with residency performance.

Altmaier et al³ used the critical incident technique to determine the attitudes and behaviors deemed critical for successful performance of residents in a pediatric training program. They determined that 70% of the faculty rated noncognitive skills as being critical for success. These results are similar to pharmacy residency programs that ranked many noncognitive characteristics as critical when defining a successful resident.

Daly et al¹⁴ conducted a study to determine predictors for otolaryngology resident success using data available at the time the candidates were interviewed and data that emerged during residency. They concluded that success was significantly related to information available at the time of residency interview. Establishing a definition of a successful resident allows pharmacy residency programs to conduct similar studies; they can examine data available at the time candidates are interviewed and data that emerge during the residency as predictors of success.

Stohl et al¹⁵ conducted a study to determine whether objective information supplied in medical students’ applications can help predict resident success. They report that faculty ranking of residents is the most commonly used method for assessing “success.” Other indicators, such as fellowship matching, continuation in academic medicine, and passing specialty-specific licensing board examinations, have also been used. Typically, these modalities are used in combination with some form of faculty assessment. Residents who are clinically proficient and have achieved competence in each of the 6 Accreditation Council for Graduate Medical Education competencies (patient care, medical knowledge, professionalism, systembased practice, practice-based learning and improvement, and interpersonal and communication skills) may be highly “successful,” yet they may not elect to pursue subspecialty training or academic positions. Stohl and colleagues suggest that using markers such as fellowship placement or remaining within academic medicine as the only measures of success may skew the data to favor academic expertise rather than clinical abilities and professional accomplishments.

Our survey has several limitations. The response rate for this survey was 43.8%; this is slightly higher than the average response rate expected from an electronic survey. Cook et al¹⁶ conducted a meta-analysis of response rates in Web- or Internet-based surveys. They found that the mean response rate for 68 surveys reported in 49 studies was 39.6% (SD 19.6%). Due to the impracticality of designing this study to impact the difference between the observed and nonrespondent answers, we focused our attention on reducing the nonresponse rate in order to reduce bias. This included sending multiple e-mail reminders, using a short survey length, and including clear and concise wording. Intrinsic variability may have existed among respondents’ interpretations of the survey questions. It is not known how this variability affected the data. We conducted a pilot survey and found that questions were interpreted correctly. The likelihood of response bias is low, as questions did not contain leading opinions. Our survey reflected the opinions of the respondents completing the survey regarding the subjective and objective characteristics that define a successful resident. Other faculty may have different opinions. The majority of respondents completing the survey on behalf of PGY2 programs specialize in critical care. Although this may represent a potential bias, these data are consistent with Summary Statistic Data from the ASHP Resident Matching Program from 2010, which state that the total number of critical care residencies available (74/358; 20.7%) exceeded that of any other specialty offered.¹⁷

Future work may focus on the development, implementation, and assessment of a behavior-based questionnaire that would be used as part of the application process when recruiting pharmacy residents. Outcomes such as quality and usefulness of the information obtained should be considered. It should also be determined whether the additional information increased the likelihood that an applicant would be granted an onsite interview and ultimately ranked.

Conclusion

Identification of the characteristics that pharmacy residency programs use to define success will allow them to identify predictors of success and optimal methods of selecting residents who possess these characteristics.