A Comparison of Student Outcomes Between Longitudinally and Traditionally Placed Advanced Pharmacy Practice Experiences

Abstract

Objective. To compare performance outcomes of students completing a longitudinal pharmacy practice experience at a single site with those of students completing advanced pharmacy practice experiences (APPEs) at a variety of practice sites.

Methods. Performance data and other meaningful outcomes from a cohort of students completing a six-month experience within a single health system (longitudinal group) were compared to data from a group of students completing six months of similar APPEs at multiple health systems (control group).

Results. Motivators for participating in the longitudinal program varied, with students in the program identifying practice interest alignment, schedule simplification, and desire to do a residency; and students choosing to not participate in identifying schedule flexibility and experience at a variety of practice sites. The 16 longitudinal students differed from the 20 control students at baseline in grade point average and the number of scheduled community pharmacy APPEs. Performance measures for longitudinal students and control students did not differ significantly, but the difference between students’ self-perceived effort and preceptor perception of student effort was significantly different for control students compared to longitudinal students. Residency match rates and number of students receiving recognition for high academic performance were similar in the two groups.

Conclusion. Students participating in the longitudinal program had different motivators but comparable outcomes to students who completed similar APPEs at a variety of practice sites.

INTRODUCTION

A longitudinal model for completion of more than one advanced pharmacy practice experience (APPE) within a single health-system or community practice setting was first described in 2011.¹ Subsequent publications have described implementation of longitudinal models through partnerships between health-systems pharmacies and one or more schools of pharmacy.^2-4 Longitudinal models, also called “block” placements, are typically sequential APPEs, including core acute care, health-system, and ambulatory care APPEs, completed over four to six months at a single health care system. The University of Florida schedules a majority of its students in longitudinal learning experiences.⁵

There are several potential benefits for sites and students using a longitudinal model. Students can provide value to sites by performing medication reconciliation interviews, discharge teaching, antimicrobial stewardship, and review and monitoring of medication therapy for a subset of admitted patients.⁶ However, before students can perform these kinds of activities, they require a significant amount of orientation to the site and training by preceptors during the first two weeks of an APPE.⁷ For this reason, they are unlikely to provide value to a site during this time. After the initial orientation, students completing longitudinal experiences need minimal orientation to new sites within the same health care system. Thus, they may be better able to extend pharmacy services sooner than students with a single four- to six-week APPE at the same health-system site. The rotation of longitudinal students through different services within a single health system and interaction with pharmacist mentors are components of pharmacy residencies. Therefore, pharmacy students in longitudinal programs might be better prepared for residency interviews, potentially increasing their chances of matching to a residency site. One study found that student participation in a longitudinal experiential training model resulted in high residency match rates, although no control group was used for comparison.³

Placement concerns about longitudinal models include the favoring of academically high-performing students for placement if the application process is competitive² and preferential assignment of longitudinal students at the most highly desired sites in a health system, leaving students not in longitudinal programs at a disadvantage. Another concern is decreased flexibility in tailoring elective APPEs to students’ interests compared to a traditional placement process. A performance assessment concern is that preceptors may have different expectations for students in longitudinal programs and as a result may score longitudinal students’ performance higher or lower than traditionally-placed students. A final concern is that residency-bound students completing traditional APPEs might be disadvantaged by not receiving the same degree of applicant mentoring, resulting in lower match rates. A comparison of outcomes between similar students in longitudinal and traditional placement models is needed to support or refute these concerns.

Authors with published descriptions of longitudinal APPE training models propose that students completing a longitudinal model have acquired high-quality learning experiences and are highly satisfied with those experiences, but most provide no data to support these assertions.^2-4,8 One study used a survey to measure satisfaction of students completing longitudinal experiences, but there was no comparator group and no learning outcomes were assessed.⁹ A literature search found no published reports comparing the outcomes of students who had completed a longitudinal learning experience with outcomes of students who had completed APPEs at a variety of practice sites.

In academic year 2015-2016, a cohort of pharmacy students at the University of Washington School of Pharmacy in Seattle completed a six-month longitudinal APPE experience. A study of the cohort was conducted to identify any differences between students who did and did not choose to participate in the longitudinal program and to confirm or alleviate student placement and performance concerns. The first objective was to identify motivators for students who chose to participate in the longitudinal program and for students who chose not to participate. The second objective was to determine if students who did not participate in the longitudinal program had decreased access to desired APPEs compared to students who participated in the program. The third objective was to determine whether performance scores for students in the longitudinal program were different from those of students who had completed similar APPEs but were not in the longitudinal program. The fourth and final objective was to examine whether residency match rates and other indicators of high student performance differed between students in the longitudinal program and students with similar APPEs who were not in the program.

METHODS

In early fall 2014, pharmacy administrators from two urban academic medical centers closely aligned with our school approached us about initiating a longitudinal APPE option at their sites for our students. This option would involve a 6-month block of experiences completed at the academic medical centers, hereafter referred to as the study centers. Accordingly, in fall 2014, third professional year (PY3) students were provided information about the longitudinal program at three different forums. Student placement into the longitudinal program was noncompetitive and only required submission of a short application and curriculum vitae, and completion of an informational interview. The interview, which was with study site representatives, was designed so that students could ask questions about the program. At the end of fall quarter of 2014, the students who had applied were asked to confirm that they wanted to participate in the program. If more students applied than the cap of 20 students stipulated by representatives of the study centers, the plan was to randomly select 20 participants from the applicant pool.

To determine why students did and did not choose to participate in the program, an anonymous questionnaire was distributed to PY3 students in January 2015, after students were accepted to the longitudinal program but before the APPE site selection process officially began for the remainder of the class. Students were asked whether they had chosen to participate in the longitudinal program (yes or no) and why or why not. Answers to the question of why or why not were first thematically analyzed independently by two individuals who then met iteratively to resolve discrepancies and write coding instructions.¹⁰ A third individual acted as verifier, applying the coding instructions to each of the student comments.¹¹ Percent agreement between coders and verifier was calculated and tested for congruence using Cohen kappa, with a value of 0.6 or higher constituting satisfactory agreement.¹² Proportions of students answering no and yes for each of the themes were compared using Fisher’s exact test.

In the longitudinal program, which began in August 2015, students completed six 4-week APPEs: four acute care APPEs (general medicine, cardiology, and two electives), one clinic experience, and a hospital pharmacy administration experience. Students’ start in the program was staggered such that roughly equal numbers of longitudinal students began in August, September, October, and November.

To compare outcomes between longitudinally- and traditionally-placed students, a cohort of control (non-longitudinal) students were identified. Included control students had at least six scheduled inpatient or clinic APPEs, with the first of the six starting in one of the same months as a longitudinal student, and the subsequent five APPEs finishing about the same time as the longitudinal students finished their program. Identification of the control students occurred in fall 2015, but data were not accessed and analyzed until all students finished their APPE year in June 2016.

Learning outcomes compared between the longitudinal and control students were composite scores from preceptor assessments of student performance during the APPE, including seven specific performance areas: knowledge; use of the literature; altruism and excellence; interpersonal skills; verbal, nonverbal, and listening skills; written skills; and patient care skills. Also compared were global performance ratings, a separate metric using a different rating scale than the composite performance score.

Three additional performance outcomes were examined. One was the preceptor’s perception of the amount of effort required to precept the student and the effort put into the APPE by the student relative to other students. Another was the student’s impression of the extent of learning and effort put into each experience relative to other APPEs. Finally, students and preceptors were asked about their level of integration into the health care team (ie, fully integrated, actively participating, passive listening) and accountability for patient outcomes (ie, full, partial, none) to the health care team.

To measure whether access to study sites decreased for traditionally-placed students upon implementation of the longitudinal program, the average number of APPEs per student completed at the study sites in academic year 2014-2015 was compared to the average number completed by students in the control group. Because there are normally small fluctuations in annual site availability due to personnel and administrative factors (eg, a preceptor takes parental leave, adjusted pharmacist service patterns closes an existing site to students), a change of 10% or greater was considered different from the previous year.

Two final outcomes that were measured were residency match rates and high overall performance on APPEs. Residency match data included the number of students in the longitudinal and control cohorts who did and did not participate in the residency matching program, and the number of students in each cohort participating in the matching program who did and did not match to a residency. The compared measure for high performance in the longitudinal and control cohorts was the number of students in each cohort who qualified to receive a letter of commendation from the dean. Letters of commendation from the dean were sent to all students who averaged a composite performance score of 3.5 or higher on a 4.0 scale. A score of 3.5 or higher indicated that a student had received the same or a higher number of scores of “exceptional” performance than scores of “competent” performance. A test of agreement was used to determine if receipt of a dean’s letter in either cohort corresponded with matching to a residency or with student GPA prior to starting APPEs.

De-identified data about baseline and outcome characteristics was provided to the investigators by a data broker who was not part of the data analysis team. A University of Washington Human Subjects Division subcommittee reviewed the protocol and determined that it did not qualify as human subjects research because of the de-identification of study subject data.

Cohen’s kappa was used to test agreement. Categorical data in the baseline characteristics were compared using Fisher’s exact test, and continuous data were compared using a Welch’s t test for unequal variances. Likert-type ordinal data from preceptor assessment of student performance and student and preceptor perceptions of learning and effort were compared using a Wilcoxon rank sum test. The three categories for level of integration into and accountability for patient outcomes to the healthcare team were first tested with a three-sample chi-square test and further explored through pairwise comparisons using Fisher’s exact test. A Pearson correlation coefficient was used to examine relationships between preceptor perception of effort required to precept students and global and composite performance scores. All statistical tests were done using R, version 3.3.1 (The R Project, Vienna, Austria).¹³

RESULTS

Seventeen of 97 pharmacy students elected to participate in the longitudinal program during its first year. Eighty-six of the 97 students (89%) responded to the questionnaire asking why they did or did not choose to participate in the longitudinal program. Five themes were identified during data analysis of the responses (Table 1). Practice interests was the most frequently expressed theme, with longitudinal students stating that the experiences offered through the program fit their practice interests, and students not choosing the longitudinal program wanting to obtain experience at a variety of sites outside of the study sites. In the schedule theme, longitudinal students named schedule simplification as a consideration, while students not choosing the longitudinal program more frequently named lack of schedule flexibility as a consideration. In the career theme, students choosing the longitudinal program were more likely to be interested in a residency, while students not choosing the longitudinal program worried that participation would limit their career options. The theme of program structure was expressed by many longitudinal students, who expected to build strong relationships with preceptors, residents, and medical team members; transition between experiences more easily; and gain project management skills. Both longitudinal and non- longitudinal students expressed concern over the six-month time commitment required to complete the longitudinal program.

Table 1.

Reasons for Pharmacy Students Electing to Participate in Longitudinal Rather Than in Traditionally-Placed Advanced Pharmacy Practice Experiences

Twenty non-longitudinal students who completed the six APPEs similar to those completed by the longitudinal students formed the control group for this study. Baseline characteristics of the 17 students originally included in the longitudinal cohort and those of the control group are shown in Table 2. Baseline characteristics were similar between the groups except in two areas: grade point average (GPA), which was significantly higher among the longitudinal students; and number of scheduled community pharmacy APPEs, which was significantly higher among students in the control group. One student in the longitudinal program dropped out after three months. The remaining 16 students finished the longitudinal experience and their data were included in the outcome analysis.

Table 2.

Baseline Characteristics of Pharmacy Students Completing a Longitudinal Experience Compared to Those Completing Traditionally-Placed Advanced Pharmacy Practice Experiences

Average composite scores and scores on individual performance elements assessed by preceptors in the 96 APPEs completed by the longitudinal students and 120 APPEs completed by the control students were not significantly different (Table 3). Control students’ average global performance scores as rated by their preceptors were lower than those of the longitudinal students, although the difference did not reach statistical significance.

Table 3.

Preceptor Assessment of Performance of Pharmacy Students Completing a Longitudinal Experience Compared to Those Completing Traditionally-Placed Advanced Pharmacy Practice Experiences

Student and preceptor perceptions of the student’s role in and accountability to the interprofessional team (IPT) are shown in Table 4. Although student and preceptor perceptions of the student’s IPT role were not different in the 3-sample comparisons, subsequent pairwise comparisons showed that longitudinal students were more likely to be rated as “integrated” into the team rather than “participating” on the team by both the students themselves (p=.01) and their preceptors (p=.03). The longitudinal and control students’ ratings of themselves in terms of accountability to the IPT for patient outcomes were significantly different, but preceptors’ ratings of students on this variable were not significantly different. The longitudinal students were more likely to consider themselves fully accountable, while control students were more likely to consider themselves partially accountable for patient outcomes.

Table 4.

Preceptor and Student Measures of Interprofessional Team Roles and Accountability in Longitudinal and Traditional Advanced Pharmacy Practice Experiences

Student and preceptor responses to questions about learning and effort are summarized in Table 5. Average student self-perceived learning scores were higher in control students than in longitudinal students, although this difference was not significant. Average student and preceptor perceptions of relative effort were nearly identical among the longitudinal students, but not among the control students, where student and preceptor perceptions of relative effort differed significantly. Control students reported higher levels of relative effort, while preceptors perceived that control students put in less relative effort. Preceptors of longitudinal students felt they took less effort to precept compared to preceptors of control students, but this difference was not significant. Analysis of the 12 APPEs in which preceptors perceived students required more or much more effort to precept showed that this metric was highly predictive of a low global score (p<.001) and tended to occur in specialty acute care experiences completed early in the student’s APPE year. Preceptor perception of effort required to precept students correlated inversely to global (r=.58; p<.001) and composite (r=.41; p<.001) assessment of student performance.

Table 5.

Student and Preceptor Perceived Measures of Learning and Effort in Longitudinal and Traditional Advanced Pharmacy Practice Experiences

In academic year 2014-2015, the placement rate for all students at the two study centers was 2.57 APPEs per student. In academic year 2015-2016, the control student placement rate for the two study centers was 2.45 APPEs per student. This 0.12 APPE per student difference represented only about a 5% decrease in availability between 2014-2015 and 2015-2016 and therefore was not considered different from the placement rate in the previous year. We also did not notice any increase in student reports of dissatisfaction with their placements.

One of the 16 longitudinal students (6%) and seven of the 20 control students (35%) did not participate in the residency matching process (p=.05). Of those who chose to participate in the residency matching process, five of 15 (33%) longitudinal students and four of 13 (31%) control students did not successfully match to a residency, while ten of the 15 (67%) longitudinal students and nine of the 13 (69%) control students successfully matched (p=1.0).

Eight (50%) longitudinal students and eight (40%) control students received a letter of commendation from the dean (p=.73). There was good agreement between students who matched to a residency and those who received a dean’s letter for both longitudinal students (87% agreement; kappa=.73) and control students (83% agreement; kappa=.63). Student GPA prior to starting APPEs did not predict who would qualify for a dean’s letter for either longitudinal students (56% agreement; kappa=.13) or control students (63% agreement; kappa=.23).

DISCUSSION

Students who elect to complete a longitudinal learning experience may be fundamentally different than students who choose not to complete such an experience. In our study, students in the longitudinal program had a higher initial GPA, were more career-oriented and more highly valued the structure inherent in the longitudinal program. Students in the longitudinal program were also more likely to participate in the residency matching program than were the control students. The noncompetitive nature of student selection for the longitudinal program eliminated the possibility that the health systems might select only the highest performing students, yet many of the highest performing students chose to participate in the longitudinal program. Compiling these results helped us better understand the motivators for students who did and did not choose to participate in the longitudinal program, which was the first study objective.

A concern we communicated early on to study site representatives was that the longitudinal students not be given all the most-desired APPEs, leaving few of those for students not in the longitudinal program. The representatives responded by opening more placements in highly desired APPEs and ensuring that longitudinal students took half or fewer of these placements. The single most highly desired APPE by students at the study sites, an elective APPE precepted by a school-based faculty member that was annually requested by more than 50% of the class, was not allowed placement through the longitudinal program. In order to get placed in this site, longitudinal students could only select it as their preference for one of their three non-longitudinal placements. Because this request went through the normal randomized matching program, it was unlikely that students in the longitudinal program would get placed at this highly desired site. This information was given to all students prior to the application deadline for the longitudinal program and may have contributed to the lower-than-expected number of applicants for the program. Determining that the placement rate of control-group students at study sites was not markedly different from the historical placement rate for students, which was the second objective of this study, alleviated concerns of decreased access to study sites by non-longitudinal students.

Standard 13.7 in the Accreditation Council for Pharmacy Education (ACPE) standards for the Doctor of Pharmacy degree dictate that elective APPEs be intentionally structured to explore various sectors of practice, a process that could be impeded by a rigidly narrow longitudinal program. We discussed the opportunity for variety in the elective APPEs with representatives of the study centers during the design of the longitudinal program, and they responded by letting students choose the two elective inpatient APPEs from a list of options (eg, oncology, emergency medicine), allowing students to tailor their program to their practice interests. The willingness of the pharmacy administrators at the study centers to be flexible in the design of the program was a key element in addressing and alleviating our concerns about placement fairness.

We had been initially concerned that the preceptors at these sites, all of whom were volunteer faculty members, might give higher performance scores to longitudinal students because they required less orientation effort or because they were perceived as more committed to health system practice. An examination of differences in performance scoring between the longitudinal and control students thus became our third objective. No significant difference between groups was found in individual performance elements evaluated by preceptors, suggesting that preceptors did not distinguish between the groups and that both longitudinal and control students had similar and adequate learning outcomes despite some between-group differences in scheduled APPEs.

Students in the longitudinal group were significantly more likely to be integrated into and to report full accountability for patient outcomes to the interprofessional team compared to control students. However, our previous work showed that all students at the two study centers were significantly more likely to be fully integrated into the healthcare team and feel fully accountable to the team for the outcomes of their patients as compared to other teaching or non-teaching sites in the geographic area.^7,15-17 It was the practice model in place at the two study centers rather than the longitudinal program that accounted for the higher scores by the longitudinal students.

The difference in student and preceptor perceptions of effort between longitudinal and control students requires scrutiny. This difference may have been influenced by the time required for student orientation to each site. If student perception of effort and learning is tied to the steepness of the learning curve for orienting to the site, then the reason control students self-perceived both greater effort and greater learning could be because of the orientation process alone. If preceptor perception of student effort is more strongly tied to preceptors’ perceptions of the effort students expend after the orientation process, then it may be that time needed to orient hampers preceptor perception of student learning. This difference could be an advantage for students in longitudinal programs and may warrant further study.

Among those students who chose to participate in the residency matching process, there was no difference in residency acceptance rates between the longitudinal and control groups. The match rates were similar to those reported for students in other longitudinal programs.^2,3,9 Students in the longitudinal program were not more likely to receive a letter of commendation from the dean compared to control students. Each of these results suggests that students participating in a longitudinal program will have similar rates of residency placement and high performance as students completing traditionally placed experiences, which addresses the fourth objective of this study.

This study had several limitations. It only looked at one year of data; thus, it will be important to track subsequent pharmacy classes to see if their results are similar to those from the first year of the longitudinal program. There were some baseline differences between the control and longitudinal students, so the matching criteria may have needed more rigor, although increasing matching rigor would have decreased control sample size and study power. The higher GPA of the students choosing to participate in the longitudinal program potentially confounds data interpretation for student- and preceptor-perceived effort and learning by the students. The higher rate of scheduled community pharmacy APPEs in the control group may explain why fewer of those students than students in the longitudinal program chose to apply for residencies, but motivation to apply for residency was not examined in this study. Finally, both study sites were at locations highly desirable for student placement; thus, results might differ if the longitudinal sites were located in areas requiring a long commute or student relocation.

Regardless of the limitations, this study enabled an evidence-based approach to examining the performance of a new APPE model. The control group allowed us to compare this model to an existing model traditionally used in experiential education placements, a method in line with the guidance provided by Persky and Romanelli.¹⁸ It helped to allay initial concerns about this model, and we now feel that we can provide future students with data that can facilitate their decision-making as they plan their APPE curriculum.

CONCLUSION

Students in this study who chose to participate in a longitudinal learning experience had different motivations and career interests compared to a matched sample of their peers, suggesting that longitudinal experiences are unlikely to satisfy every student. Residency-bound students who highly value structure may prefer a longitudinal experience, whereas students who highly value flexibility may prefer traditional APPE placements. Frank discussions between schools and longitudinal site representatives will be needed to address concerns about site access fairness for students who are not in the longitudinal program. Our study detected a misalignment in perceptions of expended effort between control students and their preceptors compared to longitudinal students and their preceptors, but this difference did not appear to affect student performance scores, which were similar between the longitudinal and control groups. Fewer students in the control group chose to participate in the residency matching process, but among those who did, the match rate was similar to students in the longitudinal program. Thus, students who choose to participate in a longitudinal program should not have any advantage or disadvantage in outcomes over students who do not choose to participate in such a program. The information from this study may be useful for students considering whether to pursue a longitudinal APPE similar to the experience described in this report.