A Pharmacotherapy Scholars Program to Provide Intensive Training to Enhance Pharmacy Students’ Postgraduate Readiness

Abstract

Objective. To design, integrate the curriculum for, and evaluate an innovative program to facilitate placement of students into postgraduate pharmacy residency training programs involving direct patient care.

Methods. The Pharmacotherapy Scholars Program (PSP) was designed to prepare fourth-professional year students to become highly proficient in a direct patient care role and to successfully match with postgraduate residency training programs. The following elements were included in the year-long curriculum: integrated synchronous advanced pharmacy practice experiences with personal advising, team-based mentoring, peer-to-peer learning, longitudinal research, and professional development. Program goals were modeled after the accreditation standards for postgraduate year one (PGY1) pharmacy residency programs. Program faculty members ensured that the PSP had a broad scope, included rigorous student assessments, had a strong research focus, and provided scholarship opportunities.

Results. Sixty-eight students completed the program from fall 2013 through spring 2019. The overall residency match rate was 93%. Students’ performance on both knowledge and clinical skills assessments significantly improved after completing the program. There was an approximately 15% increase in knowledge and a 30% improvement in clinical skills based on comprehensive readiness assessments and an intermittent clinical examination that used patient simulation, respectively.

Conclusion. The Pharmacotherapy Scholars Program is an innovative training program designed to enhance PharmD students’ preparation for advanced clinical training. Students who completed the PSP achieved a high PGY1 residency placement rate while demonstrating significant improvements in pharmacotherapy knowledge and clinical skills in direct patient care activities.

INTRODUCTION

Over the past decade, pharmacy leadership organizations have proposed a vision for the expansion of residency training in order to meet the demands of complex medication use in evolving health care environments.^1-3 The American College of Clinical Pharmacy’s (ACCP) position statement endorses residency training prior to entry into direct patient care roles.² Similarly, the American Society of Health-System Pharmacists (ASHP) had established a vision that 90% of new pharmacists entering hospital and health-system practice would have completed accredited residency training by 2015.¹ The American Association of Colleges of Pharmacy (AACP) has advocated for schools of pharmacy to take proactive leadership roles in developing and enhancing residency programs.³ This collective vision for advancing pharmacists as providers of direct patient care recognizes the value of residency training to the individual, the organization, the profession, and most importantly, the patient.

According to 2018 data from the National Matching Services, over 5,500 applicants participated in the ASHP Match for postgraduate year one (PGY1) training programs. Among this cohort, the applicant match rate for both phase I and phase II was 64%.⁴ The number of available positions in the match for PGY1 programs has increased by approximately 30% since 2013. However, the match rate has only increased by about 5% during that time despite an approximate 40% increase in applicant participation in the match.⁴ These data underscore the ongoing demand and competitive nature of postgraduate pharmacy residency training.

Match rates at the University of Pittsburgh School of Pharmacy have historically been above the national average; however, opportunities still existed to increase the rigor and preparation for Doctor of Pharmacy (PharmD) students committed to pursuing postgraduate residency training. This endeavor closely aligns with Accreditation Council for Pharmacy Education (ACPE) accreditation standards, which maintain that graduates must be practice ready and team ready.⁵ A 2010 survey to assess the readiness of colleges of pharmacy to provide residency training found that, while many schools have activities that promote residency training, very few have designated residency preparation programs.⁶ This manuscript describes the design, curricular integration, and evaluation of an innovative approach to both successful placement of students into postgraduate pharmacy residency training programs and advanced preparation of students for direct patient care activities.

METHODS

A Pharmacotherapy Scholars Program (PSP) was created and designed to be an intensive training experience integrated into the PharmD curriculum as an area of concentration (ARCO) beginning in the spring semester of the third professional year (P3). An ARCO is an elective educational opportunity that PharmD students may pursue in an area of personal interest during their degree program. As of 2018, there were eight ARCOs at the University of Pittsburgh School of Pharmacy.

The PSP ARCO was designed to prepare fourth professional year (P4) students to become highly proficient in direct patient care roles and to successfully place trainees into postgraduate residency programs. The PSP program integrate clinically sequenced advanced pharmacy practice experiences (APPEs) with personal career advising, team-based mentoring, peer-to-peer learning, longitudinal research, and professional development. The initial design of the program in 2013 was modeled, in part, after the ASHP Accreditation Standard for PGY1 Pharmacy Residency Programs (of which the competency areas, goals, and objectives were a part) (Table 1).⁷ To ensure the quality of the program and that our short and long-term goals for it were maintained, we established a PSP leadership team. The team consisted of two PSP program directors, two pharmacy operations leaders from our academic medical center, the Director of Experiential Learning and Continuing Professional Development, the Department Chair of Pharmacy and Therapeutics, the Senior Associate Dean of the School of Pharmacy, and an education support specialist, met regularly to review ongoing learner performance, address programmatic issues, and adapt and plan for future initiatives.

Table 1.

Domains and Learning Objectives of the Pharmacotherapy Scholars Program⁷

Students were instructed on how to apply for entry into the program in the fall semester of their P3 year beginning in 2013. The decision was made to offer the PSP in the P3 year rather than in earlier years to allow them more time to identify their preferred career path as the program required students to complete prerequisite courses, structured APPEs, and a research project that aligned with the P4 curriculum. We established that eligible applicants had to have attained a GPA ≥ 3.0 in the professional pharmacy curriculum (first, second, and third years of the four-year program) and demonstrated a commitment to pursuing postgraduate pharmacy residency training. The GPA threshold represented a minimum expectation for academic performance and was adapted from screening criteria for pharmacy residency candidates at our institution. The application process was designed to reflect components of the residency selection process, including submission of a letter of intent, two letters of recommendation, an academic transcript, and current curriculum vitae. Applicants were then invited for interviews before a panel that included program directors, preceptors, and current PSP students. The interview included the following components: program introduction, case review and completion, panel interview, and case de-briefing with a preceptor. Applicants were scored by the PSP leadership team using a rubric weighted equally in three areas (application, interview, and case performance) to create a rank-order list of the candidates.

Upon acceptance into the PSP, students were required to complete two prerequisite courses in the spring term of their P3 year. One course, Acute Care Simulation, was designed to improve students’ knowledge and critical-thinking skills in the management of acutely ill patients using online independent learning coupled with human patient simulation cases.⁸ Evaluation of this course, published previously, demonstrated enhancement in students’ knowledge and acute care critical-thinking skills, while also improving their learning satisfaction.⁸ The intent of this course as a prerequisite to participating in the PSP was to improve students’ patient care knowledge and skill preparedness prior to entry into advanced clinical practice on APPEs. This course was open to any student in the P3 year; however, the course had limited capacity, so students accepted into the PSP were given first preference. The second prerequisite course, Discovering Scientific Inquiry, was a mentored experience in outcomes research design and methodology. This course prepared students to collaborate on the design and execution of an outcomes-based clinical research project. Students were expected to learn fundamentals of hypothesis generation and research aims, study design and methods, proficiency in basic biostatistics and use of statistical software, development of a study proposal, and fundamentals of the institutional review board (IRB) process. The intent of this course as a prerequisite to the PSP was to improve student preparedness for developing and completing an outcomes-based clinical research project during their P4 year. Thus, this course was open only to those students in the PSP.

In fall 2013, the first group of four students began the PSP. While the general pharmacy curriculum prepares all pharmacy students to assume the roles and responsibilities of the profession, students in the PSP were encouraged to engage the profession as “stakeholders” in the medication use process and were responsible for patient outcomes under the direction of pharmacist mentors and preceptors. During the first week of the PSP, students completed an orientation to the PSP during which clinical skills development and baseline clinical assessments occurred. The goal of the orientation was to facilitate student progression to more complex patient cases and, eventually, to real patients. APPEs for PSP students emphasized responsibility for and autonomy in the patient care focus and were personalized to the needs and interests of the student. The APPEs were selected by PSP leadership to ensure that students received comprehensive exposure to advanced pharmacy practice in both the acute care and ambulatory care environments, and to reflect the variety of direct patient care experiences that a PGY1 resident would complete during their training. The APPEs were concentrated but not exclusive to UPMC hospitals. All P4 students at the school of pharmacy were required to complete eight 5-week APPEs. For the PSP students, the core PSP APPEs included: institutional (hospital/health-system); acute care (internal medicine), acute care (critical care/cardiology), ambulatory care, community practice, and a subspecialty acute care experiences (ie, transplant, oncology, emergency medicine, etc). As much as possible, APPEs were sequentially designed so that foundational experiences (ie, institutional, internal medicine) were scheduled prior to more complex and specialized patient care experiences (ie, transplant, critical care/cardiology). In addition to the core APPEs, PSP students completed two electives of their choice within any of the above categories, typically choosing additional direct patient care experiences, or they could choose a “pure” elective (eg, research). PSP students were advised to schedule their “off” block to coincide with their anticipated residency interviews.

As noted above, PSP students were required to complete eight APPEs. Students were required to complete blocks one through four, which spanned from May through late September, locally (ie, in the greater Pittsburgh area) so that all students were on site for active participation in the longitudinal research project. Students progressed through each APPE block in groups of up to four, and this cohort also served as their research workgroup. This structure promoted collaboration, peer-to-peer learning, and cooperative problem-solving. Communication among all students and preceptors in the PSP is further solidified through an email listserv and website.

In addition to patient care experiences, PSP students also had significant research and scholarship opportunities. Students were expected to engage in clinical research by participating in the collaboration, design, and execution of a longitudinal research project. The project was initiated in the scientific inquiry class students completed in their P3 year, and continued throughout their P4 year. This goal supported the vision of the PSP to develop clinical research skills and prepare students for entrance into postgraduate residency training. The clinical research project was designed to be structurally similar to the conduct of a PGY1 research project, except that students collaborated directly with their peers. Research projects were mentored by a team that included the PSP co-directors, a clinical data scientist, a research fellow, and other mentors based on clinical and research expertise. Research faculty members were also invited on an ad hoc basis to present brief seminars on clinical research related to project topics. Our partnership with UPMC enabled our students to conduct clinically relevant research projects that addressed important questions that aligned with initiatives to improve patient care. Project topics were selected and assigned by the PSP leadership. A clinical data scientist served as a neutral third party acting on behalf of the research team for the projects, providing de-identified data once IRB approval was obtained. Patients were identified through an electronic medical record data repository that contained full-text medical records and integrated information from central transcription, pharmacy, laboratory, finance, administrative, and other departmental databases.⁹ Students worked in groups, typically on larger-scope projects that were then further divided into smaller projects with specific aims. For example, a single project would be undertaken by eight students, half of whom would address a safety question while the other half addressed an effectiveness question. Progress meetings for all students occurred weekly throughout the P4 year to further their research and to develop problem-solving, data collection skills, and proficiency in project management. Students also learned fundamentals of SPSS Statistics 25.0 (IBM, Inc, New York, NY) for data analytics and statistical testing. The primary deliverable of the research project was the development of an abstract and poster suitable for presentation at the ASHP Midyear Clinical Meeting. Thereafter, the expectation was that a manuscript would be drafted and eventually submitted for peer review in an appropriate biomedical journal based on the clinical content.

A third component of the PSP was the inclusion of a formal professional development seminar series tailored to address necessary clinical, research, academic, and postgraduate preparations. The PSP Professional Development Series complements program goals to prepare students to become highly competitive applicants for postgraduate residency training. These seminars occur once or twice monthly throughout the P4 year and include the following topics: selecting and preparing for pharmacy residency training, career planning, clinical practice and research opportunities, curriculum vitae and letter of intent preparation and review, interview skills and mock interview sessions, ASHP Midyear Clinical Meeting planning and preparation, and manuscript writing and refereeing. Additionally, the PSP hosted an external speaker each year whose presentation focused on pharmacy residency training. The professional development series helped to advance one of the primary goals of the PSP, which was to help students match with a PGY1 program.

A critical and distinguishing feature of the PSP was the scope and rigor of student assessments. During PSP orientation week, each student completed a baseline comprehensive knowledge-based examination (readiness assessment) which was developed by clinical content experts under the direction of the leadership team. This 250-question, multiple-choice format examination was administered electronically and covered the following therapeutic and practice domains: ambulatory care, critical care, cardiology, internal medicine, and either oncology or transplant. The examination evaluated students’ foundational knowledge in each of those practice areas. Each student received their score on the examination, including a breakdown of their performance on each domain. However, they did not receive the answers to individual questions. The feedback provided an opportunity for students to focus their efforts on reviewing therapeutic areas that required improvement before starting clinical APPEs. At the conclusion of the PSP, students completed the same examination in order to evaluate change in knowledge (overall and by practice domain) over the course of the year. The overall pre- and post-program examination scores were compared using a paired t test.

An intermittent clinical examination (ICE) was also integrated throughout the PSP to further evaluate students’ direct patient care skills and prepare for case-based components to PGY1 interviews. The ICE was an objective, structured clinical examination that required the student to evaluate a patient assigned by the preceptor. The evaluation included data collection and synthesis, problem identification, and presentation of treatment recommendations. This was developed by clinical preceptors under the direction of the PSP leadership. Students were given 60 minutes to complete a patient review that was new to them, and then deliver a 15- to 20-minute verbal presentation, allowing time for questions and feedback. A standardized rubric was used to evaluate the presentation and interpretation of subjective and objective data, development of the pharmacotherapy plan with associated monitoring, and effectiveness of the student’s communication to the preceptor. A Likert scale was used to evaluate each element of the rubric on the following scale: needs improvement, satisfactory progress, and achieved. Table 2 provides a summary of the key elements regarding pharmacotherapy plan development of the ICE. A baseline ICE was completed during the first week of the initial APPE using a blended case with human patient simulation. Feedback was provided directly to each student about their performance. An ICE was also administered during the final week of the following APPEs: ambulatory care, critical care/cardiology, internal medicine, and sub-specialty. Review of ICE performance with the student occurred immediately after their presentation. Results were then shared with the PSP co-directors and then aggregated by student. Feedback could then be compiled and sent to subsequent preceptors to improve program development. A follow-up ICE using the same baseline case at the human patient simulator was completed at the conclusion of the PSP to capture overall change in clinical skills performance. A pre- and post-ICE evaluation of clinical performance was conducted. Scores were compared using the paired t test. An evaluation of the effectiveness of the PSP, including the aforementioned assessments, was approved as exempt research by the University of Pittsburgh Institutional Review Board.

Table 2.

Intermittent Clinical Examination Conducted as Part of the Pharmacotherapy Scholars Program Curriculum

Each of these assessments are unique to students in the PSP, even though all students complete rigorous assessments to meet professional and accreditation expectations.

Performance during APPEs was evaluated using the same rubric and platform that is used for all students. Performance was categorized for different knowledge and clinical domains, linked to our curricular outcomes, based on the following mastery scale: awareness, beginning competence, intermediate competence, and proficient. Evaluations were completed at the midpoint and conclusion of the APPE. For students that had multiple preceptors on a given APPE, feedback on student performance could be shared.

RESULTS

Since the inception of the program in fall 2013 through spring 2018, a total of 68 students completed the PSP. The number of enrolled students by year and growth of the program over time are shown in Figure 1. The 2018 class size for the PSP was 16 students even though the number of applicants had nearly doubled over the previous two years. In 2016-2017 and 2017-2018 we had 21 and 24 applications for an acceptance rate of 76% and 67%, respectively. The overall match rate with a PGY1 residency program for the 2018-2019 year was 100% (range, 75%-100%). All PSP students matched with PGY1 acute care programs (see Figure 2 for match rate by program year). Student placement to a PGY1 residency based on their preferred (highest-ranked) program also steadily increased from 25% in 2014 to 70% by 2019. As with all student pharmacists, matching with the highest-ranked program is based on the student’s personal predilections and many factors including program quality, reputation, and geographical preference.

Figure 1.

Pharmacotherapy Scholars Program – Student Enrollment and Expansion

Figure 2.

Pharmacotherapy Scholars Program – PGY-1 Match Rate

Results of the evaluations of student preparation for direct patient care activities administered before and after completion of the PSP show significant increases in both knowledge and clinical skills assessments. There was an approximate 15% increase overall in scores on the knowledge-based examination (161 [20] vs. 184 [16], mean [SD]; p <.001). Note that data were analyzed using a paired t test for 20 students (2016-2017 and 2017-2018) who completed the pre-and post-readiness assessments. Students’ clinical skills scores, which were based on the encounter at the patient simulator, increased by approximately 30% (12.4 [3.7] vs. 16.2 [3.2]; p=.002). Note that data were analyzed using a paired t test for 15 students (2017-2018) who completed the pre- and post-intermittent clinical examination.

Research studies completed by PSP students from 2013-2019 were retrospective cohort studies of electronic health records involving large data sets (typically >5,000 patients) designed to evaluate comparative effectiveness and safety outcomes. Research projects included cardiovascular critical care, nephrology, and infectious diseases-related topics. All students presented posters at national meetings, including 14 presentations at the ASHP Midyear Clinical Meeting and three at the ACCP Virtual Poster Symposium. Of note, each poster presentation represented the work of three to four students who worked as a team. Two manuscripts were published in peer-reviewed journals and four others were either under review or in the manuscript development phase as of fall 2019.^{10, 11}

DISCUSSION

The PSP is a rigorous training program for PharmD students who are committed to postgraduate residency training. The program supports the collective professional vision for the advancement of pharmacists as “practice-ready” direct patient care providers through completion of a residency program.^1-3 Specifically, the PSP focuses on student preparation for securing a PGY1 residency program in a highly competitive environment. The success of the program in meeting this goal is evident through the high PGY1 match rates relative to national averages (64% in 2019).⁴

Our program aligns with efforts to better integrate students into direct patient care and develop responsibility for medication therapy outcomes.¹² Key assessments evaluating patient care knowledge and performance showed progressive learning. Significant improvements were observed in both pharmacotherapy care plan development and monitoring, as well as comprehensive knowledge over the course of the program.

The generalizability of the PSP to postgraduate residency training experiences was reinforced through mirroring the design and sequence of clinical experiences, structure and timing of assessments, and integration of research and professional development. The PSP is structured using a sequential APPE design that incorporates both traditional individualized and team-based precepting. This design leverages capacity for multiple learners and minimizes inefficiencies to the student and preceptor that would otherwise occur through retraining. Similar sequential APPE designs, or “complete-block scheduling,” have been described at other institutions and have proven to be successful.^13,14 Advantages associated with this design include: increased learning satisfaction, enhancement of problem-solving skills, and increased ownership of patient outcomes. Students that participate in this type of APPE design also are capable of extending pharmacy services and providing more clinical interventions.^13,14 Other models have focused on collaborative programming, including workshops and mock interviews, to enhance student preparation for the residency application and selection process, which was also well-received.^15,16 There are other examples of clinical track approaches which combine requirements for coursework, focused clinical APPEs, participation in a clinical skills competition, and completion of a skills checklist.^17,18 A survey of student perception was the most commonly reported tool to demonstrate overall impact among residency preparation programs.¹⁶ However, few programs reported residency match rates or other formal outcomes.^15,16,19 The PSP integrates elements of each of the aforementioned examples with a sequential APPE design coupled with structured programs and coursework to improve student preparation for successful placement and transition into postgraduate residency training programs. Our program also reports program impact through both residency placement and student learning.

The PSP training environment emulates certain aspects of the residency experience, including use of comprehensive and rigorous assessments to guide focused and timely feedback for students. This provides further evidence of the fidelity of the PSP training environment relative to a PGY1 residency experience. Data gathered from both knowledge-based and clinical skills-based assessments offer greater insight into the student’s areas of strength and opportunities for improvement. The clinical skills assessments rely on a blended approach through use of both direct patient care and human patient simulation scenarios. Information about each student’s performance can also be shared with the precepting team to foster more personalized learning and mentorship. The baseline clinical skills and knowledge assessments also present an opportunity to address student readiness, which may be seen as a potential barrier to student involvement in direct patient care.¹²

Additionally, students in the PSP have the opportunity to work closely with pharmacy residents, thereby gaining valuable first-hand exposure to the residency experience. Residents, in fact, serve as primary preceptors for the internal medicine experience for several of the PSP students and also share co-precepting responsibilities on several other core APPEs. Residents in the primary preceptor role provide formative and summative assessments of the students. The preceptor mentorship conferred by the consistent and organized interactions with residents is a clear advantage of the PSP.

The longitudinal research project requirement is a unique experience provided by the PSP relative to other published clinical track approaches, and another example of a core part of a PGY1 residency program. The goals of the research requirement in the PSP is to teach students fundamental concepts in the design and conduct of clinical research, and to develop skills needed for effective analysis and interpretation of data. The outcomes research projects leverage large data sets to conduct comparative effectiveness and/or safety analyses. Students gain valuable experience in data management, analysis, and interpretation. They also learn to navigate the peer-review process as their project progresses toward a completed manuscript. The exposure to the publication process has been shown to influence the likelihood of future scholarship, thereby providing another advantage for students who are committed to career advancement in clinical pharmacy and research.²⁰ Project mentors have also realized more scholarship and academic productivity through the investment of time and effort with these larger scale research projects.

The PSP also integrates into the curriculum a robust professional development series that provides student exposure to a diverse network of clinical, research, academic, and administrative experts. Invited speakers are current University of Pittsburgh School of Pharmacy faculty members as well as external individuals with national expertise and reputation in their areas of practice or research. The majority of the sessions are dedicated to preparation for residency training. We have also been able to leverage a highly engaged group of alumni and PSP graduates. These individuals have become PSP mentors and preceptors and continue to expand our PSP network and reputation.

The success of the PSP, as evidenced through the high residency placement rate (Figure 2), has led to significant program growth over the past five years. Figure 1 illustrates the expansion in the number of students that have been accepted into the PSP. Additionally, we have expanded the number of preceptors and sites, both at UPMC and outside of UPMC, in order to provide students with exposure to new and diverse clinical settings and practice models as the program expands.

While the PSP has grown considerably, so has competitiveness for entry into the program. Despite this increase in demand, we plan to maintain our current class size of 16 students at this time. Current limitations to further expansion include our ability to ensure sufficient high-quality APPEs, provision of rigorous assessments, personalized mentorship, and execution of multiple longitudinal research projects for a larger number of students. The PSP is also relatively resource intensive, as it requires dedicated time from program codirectors, school and institutional leadership, clinical preceptors, and research mentors. To meet these challenges while addressing the growing demand, an evaluation of program expansion is ongoing. As we continue to evaluate the current state and future design of the PSP, we also work to ensure that all students are afforded opportunities to personalize their education by pursuing clinical, professional, and research mentorship. For example, students have access to professional development series and the same APPEs that those participating in the PSP would typically complete. Finally, all students have opportunities to work with faculty members to engage in research and quality improvement activities. The main distinction of the PSP is the structured design and degree of active engagement in clinical, professional, and research-related activities.

CONCLUSION

The PSP was designed to enhance PharmD student preparation for advanced clinical training in a post-graduate pharmacy residency program. The program is built and sustained on a collaboration between the University of Pittsburgh School of Pharmacy, UPMC leadership, and the network of preceptors across our region. We have achieved a high placement rate among our graduates into postgraduate residency programs. Furthermore, our experience demonstrated significant improvements in pharmacotherapy knowledge and clinical skills in direct patient care activities. Our program provides a model which could be used by other schools to advance PharmD education and training, and associated placement of students into residency programs and other patient care career paths.