Practice Experiences at a Single Institutional Practice Site to Improve Advanced Pharmacy Practice Examination Performance

Abstract

Objective. To determine whether sequential assignment of students to the same facility for institutional practice experiences improves their advanced pharmacy practice experience (APPE) examination scores.

Design. Student volunteers were assigned to the same healthcare facility for all institutional introductory pharmacy practice experiences (IPPEs) and advanced pharmacy practice experiences (APPEs). Other students completed institutional IPPEs and APPEs at separate healthcare facilities, ranging from 2 to 4 different facilities per student. APPE examination scores of students assigned to the same facility for all institutional learning experiences were compared with those of students assigned to more than 1 institutional practice site.

Assessment. Holding grade point average constant, students assigned to the same facility for institutional IPPEs and APPEs scored 3 percentage points higher on the APPE institutional examination compared with students assigned to separate facilities for these experiences.

Conclusion. Assigning students to the same facility for both institutional IPPEs and APPEs positively influenced knowledge-based APPE examination performance.

INTRODUCTION

The American Council on Pharmaceutical Education (ACPE) Accreditation Standards requires all pharmacy students to complete both IPPEs and APPEs in institutional settings.¹ Changes in these standards have further defined that a minimum of 75 IPPE hours is required in institutional settings. Challenges to providing a sufficient number and quality of experiences in institutional practice have been previously noted,² and suggested entry-level competencies have also been published.³ Institutional pharmacy practice environments are heterogeneous and complex, and fewer pharmacy students have existing work experience in such environments. Because of structural and procedural differences in the delivery of pharmacy services, the institutional environment is more challenging for students, requiring preceptors to invest more time in orientation before students are able to develop and contribute in meaningful ways. Accomplishing learning objectives in these complex institutional environments within the limited time available makes an intentional design even more important to maximize the return on efforts invested by students and preceptors.

Pharmacy student contributions in institutional environments have been documented in medication education,⁴ medication reconciliation,^5-10 cost avoidance,¹¹ clinical interventions,^12-14 clinical practice guideline development,¹⁵ and support of a venous thromboembolism prophylaxis program.¹⁶ In 1 published report, a small group model was designed to integrate a 2-course series with IPPE learning. In this model, first- and second-year doctor of pharmacy (P1 and P2) students shared responsibilities of completing instructional objectives such as patient interviews and patient presentations in hospitals and clinics.¹⁷ Student and preceptor responses on a survey instrument about the model were positive, but continuity of learning at the same site was not specified and no comparison group of students was available to enable measurement of subsequent differences in APPE performance. Another study involved implementation of dedicated, midsemester scheduling blocks for P1 and P2 students to accomplish IPPE learning objectives outside of classes as well as to meet the intent of using geographically diverse practice sites and sequencing to minimize overlap of P1 vs P2 learners.¹⁸ This experiential design included hospital observations for all P1 students and hospital experiences for some P2 students, but neither reported sequential placement of students at the same sites nor evaluated impact on APPE performance. These existing reports show efforts to integrate IPPE learning across a course series and/or maximize efficiency of site/preceptor resources, but have neither studied continuity of learning within the same facility over time nor evaluated the impact on student learning relative to other experiential designs. Thus, it is unclear whether fostering continuity of a learning environment through sequential assignment of pharmacy students to the same institution affects learning outcomes.

To evaluate the feasibility of assigning the same students to the same institution for all required IPPE and APPE institutional placements, the University of Oklahoma Health Science Center, College of Pharmacy partnered with the pharmacy administration of Mercy Health Center (MHC), a local institution. With a goal of maximizing efficiency by increasing student familiarity with the institution over time, student performance was expected to improve as well. This study hypothesis was that promoting continuity of learning through assignment of students to the same facility for institutional IPPEs and APPEs would improve performance on a required APPE institutional examination. Basic structure of the IPPE program and the institutional IPPE component, as well as the process by which students were assigned for learning experiences are described; details related to the required APPE examination and the assessment methods used to compare student groups are provided; and findings and additional information from relevant publications that have examined longitudinal experiential designs in medical education are discussed.

DESIGN

Students at the study institution log 100 IPPE hours each in years P1 through P3; the following description pertains to graduating classes for the years 2011 through 2013. In the P1 and P2 years, student IPPE hours were accomplished during the fall and spring semesters; these hours had traditionally been accumulated longitudinally at assigned community pharmacies and through single-day assignments of 2 to 4 hours at specialized pharmacy-practice environments, including institutional practice sites. The P3 year was characterized by a full-time 80-hour assignment in the summer, in which the majority of students focused on community pharmacy and a smaller number received an institutional assignment based on student preference. The final 20 IPPE hours during the P3 year were typically accomplished on a faculty-based assignment during the fall or spring semester that focused on ambulatory care, acute care, or a specialty area. Prior to implementation of the most recent ACPE accreditation standards,¹ the length of institutional IPPEs varied among students, with the majority being assigned a 2- to 4-hour block for 1 to several visits during their P1 and/or P2 years and a minority receiving an 80-hour institutional IPPE for their P3 summer. Assignments were made based on student preferences, with no deliberate planning to influence the continuity of institutional facility assigned to students. Learning objectives for the institutional IPPE were provided to students and preceptors to implement within the assigned time. Students received formative assessment and were required to complete reflective journaling exercises regarding their experiences. A final summative assessment of students by supervising preceptors was required at the end of the P3 IPPE.

The college wanted to increase the number of institutional IPPE hours per student, but general discussions with local institutional partners did not result in all facilities being willing to commit to expanding the number of hours for various reasons, including the burden of repeatedly orienting and training students in the P1 through P3 years. With the belief that an improved model to support institutional IPPE training was feasible, the current intervention was conceived in conjunction with MHC. Beginning with the 2009-2010 academic cycle, student volunteers were sought for a sequential assignment process of institutional IPPE and APPE rotations at MHC. Students were identified and assigned based on replies to an e-mail describing the intended rotation sequence at MHC. The sequential design included a P2 institutional IPPE of 20 hours during 1 calendar month of the fall or spring semester, a 2-week 80-hour P3 IPPE during the summer between the P2 and P3 years, and 3 consecutive month-long P4 APPEs focused on institutional practice, acute care adult medicine, and acute care selective (eg, pain management and antibiotic streamlining). Students received orientation and progressive learning activities at MHC for institutional IPPEs and APPEs in the areas of medication procurement, order entry review and verification, compounding and dispensing sterile products, medication therapy assessment, medication administration, medication reconciliation, drug information, patient education, medication safety, pharmacy law, and formulary management.

Eight students from each P2 class were selected for placement at MHC based on their desire to commit to the entire sequential assignment process as determined from student responses to an e-mail soliciting volunteers. Other pharmacy students were assigned to their institutional IPPEs and APPEs according to preferences submitted through a Web-based scheduling process. These assignments were not necessarily to the same facility for IPPE and APPE placements, as previously described. The majority of students not assigned to MHC received between 1 and several single-day IPPE assignments in institutional practice, with a minority of students also receiving an 80-hour P3 institutional IPPE. Regardless of institutional IPPE assignments, all students were assigned a 1-calendar-month institutional APPE with a minimum of 160 hours. Students not assigned to MHC typically completed institutional IPPE learning in 1 to 3 different facilities, with 1 additional facility assigned for APPE learning (Figure 1).

Figure 1.

Comparison of Institutional Introductory and Advanced Practice Experience Sequences Prior to the APPE Examination.

In addition to receiving formative and summative performance assessments from preceptors, all students in the PharmD program were required to take an objective written examination upon completion of their required institutional APPE. This examination was based on prespecified learning objectives covering fundamental drug knowledge of approximately 100 medications (12 objectives), pharmacokinetic principles (9 objectives), community-acquired pneumonia (6 objectives), pain management (4 objectives), parenteral nutrition including calculations (9 objectives), sterile compounding/USP 797 requirements (9 objectives), pharmacy law (3 objectives), medication safety and formulary management (6 objectives), and Joint Commission goals (3 objectives). Institutional preceptors were provided with the examination learning objectives, but it was the students’ responsibility to prepare for the examination, which was administered each month by the college to all students completing the required institutional APPE. Students were given 60 minutes to complete the 100-point examination, which consisted of approximately 70% multiple-choice items and 30% short-answer questions or pharmacy math calculations. The multiple-choice examination questions were scored using Remark Classic OMR, version 2.6 (Remark Products, Malvern, PA), whereas the short-answer questions were scored by experiential office faculty members according to a pre-established grading key. A closed examination process was maintained, ie, the examination scores were posted but the examination content was not distributed back to students. In order to increase the integrity of the institutional APPE examination instrument, each year, 3 separate versions were drafted based on the learning objectives and administered. With each subsequent year of institutional APPE examination delivery, approximately 25% of the questions per version were modified. This process was employed to maintain the integrity of the examination content over time.

Univariate analyses were performed to describe individual variables, and a 2-sample t test was used to compare examination scores as 1 basis of assessing the influence of practice experience assignment method on student performance. A multiple linear regression analysis was performed to examine how facility (coded as MHC/non-MHC) and grade-point average (GPA) were related to examination score. An additional multiple linear regression analysis was performed as described above but only for students who had completed 80 or more institutional IPPE hours. Model fit statistics, residual tests, and diagnostic plots were assessed. All tests were performed using SAS, version 9.3 (SAS Institute, Cary, NC) with an alpha level of less than 0.05 used to denote significance. This study was approved by the institutional review board for the University of Oklahoma Health Sciences Center.

EVALUATION AND ASSESSMENT

In June 2010, when the first group of students had completed IPPEs and were completing APPEs at MHC, the college began comparing their scores on the required institutional APPE examinations with those of students from the Oklahoma City campus who were not assigned to the same institutional facility for their experiences. Analysis of cumulative P4 institutional APPE examination scores occurred at the end of each academic year starting with 2010-2011. The average examination score for students assigned to MHC (n=8) was higher than that of other students at the end of 2010-2011, but the difference did not achieve significance (p=0.25). Extending the analysis from 2010 through academic year 2012 (n=16 at MHC) showed the same trend, but the difference was still not significant (p=0.05). After adding institutional APPE examination scores from the 2012-2013 year through December 2012, the difference in scores achieved significance (Table 1). For the study period, 22 students were assigned to MHC for their institutional APPE, while 162 students were assigned among 17 other facilities. The results from the regression analysis show that assignment to MHC and GPA were both significantly associated with examination score, regardless of total number of IPPE hours completed, with no significant interaction existing between the predictor variables (Table 2). Holding GPA constant, students assigned sequentially to MHC for institutional IPPE and APPE learning experiences were expected to have an average examination score that was 3 percentage points higher than that of other students (p=0.03). An additional regression analysis was performed on the dataset containing only students who had completed 80 or more institutional IPPE hours, with examination score regressed on GPA, facility, and number of IPPE hours completed (Table 2). This analysis again showed that facility and GPA are both significantly associated with examination score, regardless of total number of IPPE hours completed, with no significant interaction existing between the predictor variables. The results suggest that students assigned sequentially to MHC for institutional learning experiences will have an average examination score that is approximately 4 percentage points higher than that of students assigned to other facilities, holding GPA constant (p=0.03).

Table 1.

Descriptive Statistics and Distribution of Student Assignments by Facilities (Mercy Health Center vs Other) for Institutional Advanced Pharmacy Practice Experience Assignment Years June 2010 through December 2012

Table 2.

Multiple Linear Regression Analyses of Institutional Advanced Pharmacy Practice Experience Examination Scores Based on Grade Point Average and Facility (Mercy Health Center vs Other)^a

DISCUSSION

The sequential design of institutional IPPEs and APPEs at MHC was intended to accomplish several purposes. Continuity of assignment was meant to facilitate student acclimation to the environment, minimizing the time required to adapt to structural and procedural aspects of providing pharmacy services. The benefits of continuity are experienced by both the student and the institution, as students should theoretically be able to apply knowledge and skills at an earlier stage during each subsequent experience. Thus, students have the potential to actually support and contribute to delivery of pharmacy services with less need for orientation and/or time in observation by preceptors. A logical extension of students actively contributing to delivery of pharmacy services is that they may develop skills and apply knowledge at greater depth, thus they may have an enhanced ability to perform in the environment. We sought to determine whether learning as measured through an objectively administered examination would be improved for students completing a sequential design of IPPE and APPE assignments at the same facility.

A comparative assessment of student achievement based on the summative preceptor-administered performance evaluation was also considered for this study, but it was deemed to have a lower level of discernment and reliability than a comparative written examination. More research is needed to enhance the uniformity of preceptor-administered performance evaluations and thus ensure equitable student assessment in different facilities conducting the same practice experience type. With improved evaluation methods and preceptor commitment, the impact of sequential assignments on student performance can be further evaluated.

Although we found a relationship between the assignment of students to the same facility for IPPE and APPE hours and improved examination performance, there are other factors that may have been associated with the observed difference in average score. Rather than being randomly assigned to MHC, students volunteered for placement, so it is possible that unknown factors may have influenced their examination performance regardless of the assigned facility; controlling for GPA was important, as this variable independently influenced examination scores. Although random assignment of students to MHC and other facilities would have controlled for unknown factors, it has practical limitations. Geography strongly influences student location preferences in our experience, with students seeking assignments in relative proximity to their residence. Several students lived more than 30 miles from MHC and assignment to this facility commits 33% of experiential hours to this location; thus, volunteers were sought for this assignment rather than randomly matching students to a location that might be undesirable based on travel distance.

The quantity of institutional IPPE hours achieved prior to the institutional APPE experience could intuitively impact the APPE examination performance. Most students in the aggregate analysis who were not assigned a P3 80-hour institutional practice experience logged fewer hours compared with those at MHC (range of 2 to 32 hours, Figure 1). However, the 22 students who completed a P3 80-hour institutional practice experience at other facilities logged a similar number of IPPE hours prior to their institutional APPE yet still performed significantly lower than students assigned to MHC (Table 1 and Table 2). The institutional APPE examination objectives were not distributed until the start of each APPE practice experience year, so all students had the same advance notice to study for the examination. However, all students are assigned to MHC in pairs, while only approximately 25% of the aggregate comparison group was assigned to the same facility in pairs. Though student study habits cannot be determined, being assigned with another student provides the opportunity to study jointly and discuss institutional APPE examination objectives. Students at a facility accepting only 1 practice-experience student per month would not have this opportunity for study and preparation as readily available unless they sought practice-experience classmates at a different facility for these purposes. It is also unclear whether and to what degree the sequential nature of assignment to the same facility made a difference in student examination scores compared with factors within the facility itself. Preceptors at MHC took their roles seriously and continuously looked for methods to improve student learning. However, because it is unknown whether the same level of attention would exist at a different facility committing to sequential IPPE and APPE assignments, we were unable to assess this relationship.

Students assigned to MHC had the benefit of preceptor continuity for a greater period of time than did students assigned to other institutions, and it is unclear whether this factor influenced objective measures of pharmacy student performance. It is important to consider a variety of factors that may be associated with enhancing pharmacy student learning in institutional environments, other health professions have been examining such characteristics and methods of delivery.^19-21 For example, medical schools have been examining factors associated with better performance on the Step 1 and Step 2 United States Medical Licensing Examinations (USMLEs). Medical students must complete these examinations satisfactorily in order to progress past the second and fourth years of medical school, respectively. In medical education, specifically in internal medicine clerkships, pedagogical approaches, such as more small-group conference hours per week and the use of community-based preceptors, were independently associated with higher USMLE scores on both Step 1 and Step 2.²⁰ This study also showed that a greater number of clerkship weeks improved performance for students with lower Step 1 examination scores, whereas continuity of the attending physician for 4 weeks vs 2 weeks significantly improved the change in examination scores from Step 1 to Step 2 regardless of a high or low baseline Step 1 score.²⁰ A study that compared the performance of 27 third-year medical students in a longitudinal integrated clerkship design with 45 students in a traditional block design reported statistically better performance on an objective structured clinical examination and 3 of 6 standardized knowledge assessments.²¹ In the same study, which also measured medical student perceptions related to continuity of supervision, students in the integrated design reported significantly more feedback on clinical performance and receiving 1-on-1 mentoring. Medical students in longitudinal learning experiences have described evolving relationships with preceptor investment in the learning process, feeling capable and responsible through preceptor confidence and expectations, and a collaborative approach to applying knowledge.²² A longitudinal design supporting student and preceptor continuity may modify the level of engagement and expectations of both parties, promote commitment, enhance knowledge application, and facilitate meaningful contributions to patient care. Longitudinal learning designs appear to support an environment conducive to implementing active learning in experiential education²³ and are learner-centered with respect to fostering student development.²⁴ These areas are rich for further exploration as pharmacy education continues to evolve the best approaches to learning in experiential education.

Because the sequential model of assignment meets the most recent accreditation standards for IPPE and APPE hours in institutional pharmacy practice,¹ our college implemented this method as the standard for student assignment starting with the 2011-2012 academic year, provided that the facilities were agreeable. The majority of institutions have now adopted this model, though not all have been able to support the P2 20-hour IPPE experience during the fall or spring semester. The P2 IPPE component is less manageable when a facility is geographically distant from our campuses given that students accomplish hours while attending lecture classes during the fall or spring semesters; however, the P3 80-hour summer IPPE experience itself still meets accreditation standards and provides continuity for the institutional APPE when conducted at the same facility. Though some additional surveillance and coordination efforts are required to ensure sequential assignment of students to institutional IPPEs and APPEs at the same facility, the benefits appear to justify additional steps to promote this continuity.

SUMMARY

Sequential assignment of students to the same institution for required IPPEs and APPEs in this setting increased student performance on an objectively administered examination. Continuity of learning setting benefits students and preceptors by minimizing time and energy directed at orienting students to the facility, potentially allowing more time for application and meaningful learning. Implementing a design in which students receive sequential assignment to the same facility can be accomplished through discussion and coordination with students and institutional partners. Further efforts are needed to determine how to best design and sequence IPPEs and APPEs to optimize student learning, performance, and patient care contributions in the institutional setting.