Engaging Postgraduate Year-2 Pharmacy Residents in Formal Co-evaluation of Platform Presentations at a Regional Residency Conference

Abstract

Objective. To implement and assess the perceived impact of a program designed to engage postgraduate year-2 (PGY2) pharmacy residents in formal co-evaluation of platform presentations at a regional residency conference.

Design. A PGY2 formal co-evaluator program was designed and conducted over two years. Postgraduate year-2 residents were paired with a preceptor for modeling, coaching, and facilitating. To assess the perceived usefulness of this program, a 2-question presurvey and an 11-question postsurvey were distributed to participating residents.

Assessment. Eighty-two residents completed the program and pre/postsurveys (response rate=94.3%). The percentage of residents who rated themselves as skilled in critically evaluating a platform presentation increased from 56.1% to 100%, while the percentage of residents who rated themselves as skilled in providing constructive feedback increased from 67.1% to 98.8%.

Conclusion. This novel program, which was designed to engage PGY2 pharmacy residents in formal platform presentation co-evaluation, was well received and improved resident self-reported skills.

INTRODUCTION

The Eastern States Conference for Pharmacy Residents and Preceptors (ESCPRP) is one of several pharmacy resident-focused conferences across the United States.¹ This annual conference is the regional residency conference for pharmacy residents and preceptors residing in Connecticut, Delaware, the District of Columbia, Maine, Maryland, Massachusetts, Pennsylvania, New Hampshire, New Jersey, New York, Rhode Island, Vermont, Virginia, and West Virginia. It is designed to provide postgraduate year-1 (PGY1) and postgraduate year-2 (PGY2) pharmacy residents with the opportunity to formally present their research projects to preceptors and to resident colleagues, while also providing opportunities to network with other resident colleagues, meet with leaders in pharmacy practice, and augment their knowledge base as they embark on their careers.²

Historically, PGY1 residents attending the ESCPRP presented their projects as platform presentations. Postgraduate year-2 residents, although given the option to present platforms, are encouraged to present their project as a poster. Residency preceptors having expertise in specific practice areas are assigned to formally evaluate platforms and posters, and in doing so, provide constructive feedback to the presenting resident on their residency project and on their presentation skills.³ Although PGY2 residents attending the ESCPRP traditionally attend platform presentations of interest, they have not been actively engaged in the formal evaluation process. Because much can be learned through active engagement in this activity, including how to critique a study and how to provide feedback in a constructive manner, excluding PGY2 residents was viewed as a missed opportunity.

In 2014, the ESCPRP advisory board discussed ways to better involve PGY2 residents in the conference to increase the value of the conference in this subset of attendees and better prepare them for their postresidency career. In doing so, the advisory board designed and implemented a formal evaluator program for PGY2 residents to actively engage them in the evaluation process for platform presentations under the direct mentorship of a residency preceptor and to facilitate resident/preceptor networking. This program was intended to compliment material taught within teaching and learning curriculum programs, focusing on what was deemed most valuable by residency graduates: providing effective feedback to trainees.⁴

A research project was paired with this initiative. The purpose of this study was to implement and assess the perceived impact of a program designed to engage PGY2 pharmacy residents in formal co-evaluation of platform presentations. To our knowledge, this is the first program designed to teach PGY2 pharmacy residents how to formally evaluate platform presentations using the concept of modeling, coaching, and facilitating, an educational approach recommended by the American Society of Health-System Pharmacists.^5,6

DESIGN

All PGY2 residents from pharmacy residency programs residing within the geographic region encompassed by the ESCPRP who registered for the 2014 and 2015 conferences were eligible to participate in this program. Participation in the formal evaluator program was expected; however, completion of the pre/postsurveys were strictly voluntary.

Prior to the 2014 and 2015 ESCPRP, all PGY2 residents registered for the conference were scheduled to serve as a formal co-evaluator for a platform presentation session. The PGY2 resident was paired with a preceptor from another residency program scheduled to serve as formal evaluator. Preceptors were assigned to one PGY2 resident as part of their role as formal evaluator. The following was considered, in order of importance, when scheduling PGY2 residents: the resident’s platform presentation schedule (eg, residents were not scheduled to co-evaluate the session immediately prior to or during the session in which they were scheduled to present their residency project, if scheduled to present a platform presentation), and alignment of the session topic and preceptor’s area of practice with the PGY2 resident’s specialty area of practice (eg, pediatric PGY2 residents were scheduled to co-evaluate a pediatric session with a pediatric-focused preceptor whenever possible). Approximately two weeks prior to the start of the conference, PGY2 residents and preceptors involved in this program were sent a link to a brief, 6-minute premeeting online training module. This module was created and reviewed by residency preceptors experienced with the formal evaluation of residency research projects at local/regional residency conferences and ad hoc reviewing for medical journals. The module consisted of prerecorded voice-over slides and was designed to provide participants with an overview of the co-evaluator program, the roles and responsibilities of the formal evaluator, and tips for effectively evaluating a research project platform presentation and for providing constructive feedback.³

Specifically, tips were provided to equip the PGY2 resident for co-evaluation of: presentation content (eg, inclusion of and appropriate flow of study objectives, background, methods, results, and discussion/conclusion); research design and analysis (eg, appropriateness of study design, methods, results, discussion with limitations, and conclusions); audio/visual (eg, slide design, appropriate use of tables and figures); and presentation skills (eg, speaker preparation and demonstrated expertise on the topic, verbal/nonverbal communication skills, ability to answer questions from the audience).

Preceptors involved in the formal co-evaluator program were directed to meet with their assigned PGY2 resident 15-20 minutes prior to their scheduled session to acquaint them with their resident and to introduce them to the process of formal evaluation and provision of constructive feedback. Postgraduate year-2 residents and preceptors were expected to familiarize themselves with the abstracts that were to be presented before the session. Presentations were scheduled in 20-minute increments, which included 12 minutes for the oral presentation and three minutes for questions and answers. The remaining five minutes allowed the presenting resident an opportunity to meet with the preceptor and PGY2 resident co-evaluator to receive constructive feedback. An evaluation form was used to enhance standardization of feedback to presenting residents.

During the first two platform presentations of the session, the preceptor was asked to demonstrate how to critically assess a platform presentation and how to provide constructive feedback. During the subsequent presentations, the preceptor was asked to coach and facilitate the PGY2 resident on how to effectively serve in the role of formal evaluator. The specific modeling/coaching/facilitating approach was not standardized, but instead left at the discretion of the preceptor. Preceptors were directed to provide feedback to the PGY2 resident following each presentation and again at the conclusion of session.

The development and implementation of this educational program required approximately 16 hours of one ESCPRP advisory board member’s time prior to the conference, primarily for resident scheduling, and dedicated time from 1-2 board members during the conference for program coordination. The monetary costs of the program were negligible.

To assess the perceived usefulness of this program, a 2-question presession survey and an 11-question postsession survey were developed in cooperation with and approval from the ESCPRP advisory board. Respondents were allowed to provide comments in an open-ended format. The survey instruments were distributed prior to the aforementioned platform presentation sessions at the time of meeting check-in by an advisory board member. To minimize recall bias, an ESCPRP advisory board member confirmed completion of the presurvey immediately preceding the session to which the PGY2 resident was assigned, and provided directions to complete the postsurvey immediately upon completion of the session. Postgraduate year-2 residents were asked to leave the survey instruments in the session’s room upon exiting. Information about the resident’s perceived skills was assessed on both surveys. Participants were asked to rate their ability, using a 7-point Likert scale (1=highly unskilled, 2=unskilled, 3=somewhat unskilled, 4=neither skilled nor unskilled, 5=somewhat skilled, 6=skilled, and 7=highly skilled), to critically evaluate a platform presentation and to provide constructive feedback to a pharmacy resident. The perceived value of the PGY2 evaluator program and the PGY2 resident’s impression of the mentorship they received from their assigned preceptor was requested in the postsurvey using a 7-point Likert scale (1=strongly disagree, 2=disagree, 3=somewhat disagree, 4=neither agree nor disagree, 5=somewhat agree, 6=agree, and 7=strongly agree). The time to complete the surveys was estimated to be one minute and five minutes, respectively.

Surveys and responses were numerically coded and entered into the computer database. Incomplete postsession surveys were included; however, completion of presession and postsession survey questions that pertained to the resident’s perceived skills was a requirement for inclusion. Data was analyzed using Excel and MYSTAT ^v.12 (SYSTAT Software Inc., San Jose, CA). Frequency and descriptive statistics were used to characterize responses. Wilcoxon signed-rank test was used to compare paired, ordinal data collected on both surveys and to compare data collected between the 2014 and 2015 groups. Fisher exact test was used to compare agreement/disagreement between the 2014 and 2015 groups. The a priori level of significance was 0.05. This project was deemed exempt from review by the University at Buffalo Social and Behavioral Sciences Institutional Review Board.

EVALUATION AND ASSESSMENT

Ninety PGY2 residents registered for the 2014 or 2015 ESCPRP, each of whom was paired with a residency preceptor and scheduled to serve as a formal co-evaluator for a platform presentation session. Eighty-eight of these residents attended the conference and opted into the program. One resident was excluded for not completing the session because of a scheduling conflict. Eighty-two of 87 residents who completed the program returned the surveys (survey response rate=94.3%). Of the 87 preceptors, six participated both years, equating to 17% (6/35) of the 2015 group.

Prior to the session, 56.1% and 67.1% of residents rated themselves as skilled (ie, “somewhat skilled,” “skilled,” or “highly skilled”) in critically evaluating a platform presentation and providing constructive feedback to a pharmacy resident, respectively. After participating in this program, more than 98% of residents indicated being skilled in both activities (Figure 1). Overall, the PGY2 evaluator program was well received (Table 1). Participating residents agreed that the preceptor effectively modeled, coached, and facilitated the role of formal evaluator and that the program afforded them the opportunity to network with pharmacists in their field. Respondents expressed strong agreement that the PGY2 formal evaluator program should be continued at the ESCPRP in future years (59%). Although seven of nine (78%) items had less “disagree” responses during 2015, the difference was not significant for any item (p>0.05). Open-ended responses were overwhelmingly positive: 23 positive comments, 2 neutral comments, and 3 negative comments (Appendix 1).

Figure 1.

Postgraduate Year-2 Pharmacy Residents’ Perceived Skill in Formal Evaluation of Platform Presentations at a Regional Residency Conference.

Table 1.

Perceived Value of a Program Designed to Engage PGY2 Pharmacy Residents in Formal Co-evaluation of Platform Presentations at a Regional Residency Conference^a

DISCUSSION

Following graduation, nearly three-quarters of residency graduates will precept pharmacy students and, as ASHP standards permit residency-trained pharmacists to precept PGY1 residents immediately upon graduation and PGY2 residents after obtaining one additional year of experience, two-thirds will precept pharmacy residents.^4-6 Furthermore, within three years of graduation from residency training, approximately 15% will enter academia.⁴ Thus, equipping residents for these teaching roles is imperative. In Ratka et al’s survey of pharmacy residency graduates, 64% indicated having received training on teaching skills development during their PGY1 residency, and 32% indicated having received training on teaching skills development during their PGY2 residency.⁷

It is vital PGY2 residents be provided with opportunities to teach as part of their training. This training should include, but not be limited to, providing constructive feedback to students/residents. Integrating opportunities for PGY2 residents to provide constructive feedback to students/residents as part of their training may be accomplished through a variety of means, including teaching and learning curriculum programs (previously referred to as teaching certificate programs) and academic-focused practice experiences that offer residents the opportunity to work directly with students and/or other pharmacy residents.^8-10 In their survey-based study of ASHP-accredited residency programs, Stegall-Zanation et al reported that only 27% of programs offered a teaching and learning curriculum program.¹¹ Although the commonness of such programs may have increased since publication of this study, wide variability in content and delivery of teaching skills has been observed within these types of development programs.⁷

Seventy-three percent and 62% of teaching and learning curriculum programs teach residents how to provide constructive feedback and how to assess student performance, respectively, while 77% integrate precepting of pharmacy students on practice experiences to facilitate practice of skills.⁷ Of the topics taught in teaching and learning curriculum programs, providing effective feedback is identified by residency graduates as the most valuable topic.⁴ The program described herein facilitates application of didactic content, focusing on a high-value item, and can, therefore, serve to complement teaching and learning curriculum programs, while also enhancing active participation of PGY2 residents in the regional residency conference that they attend.

In Stegall-Zanation et al’s survey, respondents reported that residents commonly served as a co-preceptor (and to a lesser extent, as the primary preceptor) for pharmacy students. In approximately one-quarter of programs, residents also served as a copreceptor to pharmacy residents.¹¹ Two practice experiences, both of which offered residents an opportunity to oversee the training of pharmacy students, either in the classroom or during experiential learning, are described to date.^8,9 Slazak et al developed a 4-week elective practice-based learning experience for PGY1 and PGY2 residents,⁸ and Ostroff et al developed a longitudinal academic rotation for PGY2 residents.⁹ Each of the experiences, which were designed to expose residents to the responsibilities of clinical faculty members, including but not limited to evaluating/grading assignments and providing constructive feedback under the guidance of an experienced preceptor/residency program director (RPD), were seen as valuable by participating residents. However, no published reports exist that provide details of PGY2 residents evaluating the performance of PGY1 residents.

The overarching goal of the PGY2 formal evaluator program was to provide residents with an opportunity to expand their understanding of how to critically evaluate a residency project and how to provide constructive feedback to the presenting pharmacy resident. Similar to the learning experiences published by Slazak et al and Ostroff et al, the residents worked under the guidance of an experienced preceptor/RPD. Following participation in this program, PGY2 residents reported being more skilled in critically evaluating a platform presentation and providing constructive feedback to a pharmacy resident, indicating the goal of this program was realized. This is not dissimilar from published reports that detail the perceived impact of teaching certificate programs on resident teaching ability, which are more longitudinal in nature.^4,7,10

In a small survey-based study that included a blend of 10 PGY1 residents, PGY2 residents, and fellows enrolled in the teaching and learning curriculum program, Castellani et al reported a significant improvement in participants’ self-reported confidence after providing feedback to students, both immediately after cessation of the program and after completion of residency/fellowship training.¹⁰ In a larger survey-based study that included 95 graduates from a resident teaching certificate program, Wahl et al reported that 71% of former residents were confident in their ability to evaluate and provide effective feedback to students.⁴ This is similar with what was reported by Ratka et al, in which 74% of residency graduates rated themselves as highly or very highly skilled in providing constructive feedback.⁷ However, in both the Wahl et al and Ratka et al surveys, many of the responding individuals possessed postresidency teaching/preceptor experience. The stand-alone program described here can serve to compliment longitudinal teaching and learning curriculum programs, while affording PGY2 residents the opportunity to engage in the teaching of PGY1 residents, which is an uncommon opportunity across residency programs that do and do not offer such a curriculum program.¹¹

As PGY2 residents graduate and transition into the role of preceptor, they may serve as a formal evaluator for platform presentations at a local and/or regional residency conference. Providing high-quality feedback to presenting residents can help residents enhance their presentation skills, while also facilitating improvement of residents’ projects prior to publication. Integrating training in the critical evaluation and provision of constructive feedback to pharmacy residents during their PGY2 residency may promote higher quality evaluation and feedback as these residents assume the role of formal evaluator in the future. Overall, the PGY2 formal evaluator program was deemed useful for training PGY2 residents to serve in the role of formal evaluator, and residents indicated that their confidence in serving in this role increased following participation. More than 95% of PGY2 residents agreed that the preceptor to which they were assigned effectively modeled how to formally evaluate platform presentations and how to provide constructive feedback to pharmacy residents, while more than 93% of PGY2 residents agreed that the preceptor coached and facilitated while they formally evaluated platform presentations and provided constructive feedback to presenting residents. The premeeting educational training module appeared to be less useful: only 77% of residents indicated that it adequately prepared them to serve in the role of formal evaluator, while approximately 17% of respondents were neutral on the topic. It is unknown how many participating PGY2 residents actually viewed the training module, but the data suggest some may not have done so. Because the presurvey was administered after the online module, it appears that the difference seen between the presurvey and postsurvey was a result of the hands-on experience of co-evaluating the presentations rather than viewing the training module. Nevertheless, the premeeting online educational training module and in-meeting, hands-on experience may have complimented each other for those PGY2 residents that completed both.

The ESCPRP has pursued ways to increase the interaction between residents and pharmacy leaders in the past; however, previous attempts (eg, resident/preceptor mixers with “People Bingo” to serve as an ice-breaker and “Game-Nights” to encourage interaction in a relaxed atmosphere), have been only modestly effective. One of the underlying goals of this program was to facilitate resident/preceptor networking. To accomplish this, PGY2 residents were strategically paired with a preceptor having similar areas of practice, whenever possible. Approximately 87% of participating PGY2 residents agreed that the formal evaluator program afforded them the opportunity to network with pharmacists in their field, indicating a secondary goal of this program was achieved.

There are several limitations to this study. First, because the data were derived from a single regional residency conference, the project may not be well suited for regional residency conferences with different structure/formats. Second, given the quality improvement nature of the survey, we did not collect demographic information from participating residents and were, therefore, unable to comment on participants’ platform presentation experience or on their prior didactic and clinical teaching experience. Third, we did not survey participating preceptors to assess their perceptions of the program or their perceived value of mentoring PGY2 residents in this manner; however, verbal feedback from preceptors indicated they were positive about the program. Finally, the data indicate that this program positively impacted perceived abilities of PGY2 residents; however, we did not objectively evaluate whether or not there was an improvement in the ability of the resident to critically evaluate a platform presentation or to provide constructive feedback to a pharmacy resident.

The ESCPRP plans to continue this program in future years. Future research should evaluate preceptors’ perspectives of this program, extension of the program to poster presentations, and impact of a similarly designed program engaging PGY1 pharmacy residents in formal co-evaluation of platform presentations.

SUMMARY

The need to develop residents into preceptors is well-recognized. This study describes a feasible and novel method for cultivating resident evaluation/feedback skills and increasing networking opportunities for residents. The program was designed to actively engage PGY2 pharmacy residents in formal platform presentation evaluation using the concept of modeling, coaching, and facilitating and was well received and improved resident self-reported skills in formal co-evaluation of platform presentations. Other regional residency conferences across the United States may want to consider implementing a similarly designed program for mentoring PGY2 pharmacy residents, and if resources are available, PGY1 pharmacy residents.

Appendix 1. Eastern States Conference for Residents and Preceptors (ESCRP) PGY2 Formal Evaluator Program: Select Open-Ended Comments.

Positive comment(s)

 “Great program. Very helpful for residents.”

 “I think this experience was really helpful and I am really glad to have done this since I had little to no experience thus far.”

 “(This program) provided insight on methods to evaluate content of presentations and taking it to the next level beyond style.”

 “This was a great program! I feel more confident in my ability to formally evaluate platform presentations and provide constructive feedback. Thank you for this opportunity!”

 “The program was very constructive. 100% agree to continue this program.”

 “I really enjoyed this experience. I would recommend doing this again next year!”

 “Great experience. Yes, (this program) should this be continued in future years! Consider opening it up to PGY1s since a lot of them do not come back as PGY2s”.

 “Good experience. Preceptor was a great role model.”

 “Enjoyed the experience. The preceptor I worked with was fantastic.”

 “I enjoyed doing this, especially having a chance to hear the evaluator's perspective and methods they looked for. Thanks!”

 “Excellent, appropriate examples, great feedback from preceptor.”

 “Perhaps include 2 sessions with different evaluator coaches.”

 “Helpful experience to transition to new role as preceptor.”

Neutral comment(s)

 “Allow for more time (15 minutes) to meet preceptor, discuss strategy.”

Negative comment(s)

 “By volunteer only. This caused me to miss my resident's presentation.”

 “I was placed in a topic that wasn't my primary specialty. I did learn a lot about different study design but was not able to make recommendations on the clinical significance.”