Tracked Patient Encounters During Advanced Pharmacy Practice Experiences and Skill Self-assessment Using Entrustable Professional Activities

Abstract

Objective. To determine if the number of patient encounters during advanced pharmacy practice experiences (APPEs) relates to student self-assessment of patient care skills using entrustable professional activities (EPAs).

Methods. During 12-week acute care/institutional (AC/INST) APPEs, 15-week combined community pharmacy and ambulatory care (CPAC) APPEs, and three 5-week AC/INST or CPAC elective APPEs, fourth-year pharmacy students completed patient tracking surveys. Students documented the number of encounters, type of care provided, primary and secondary diagnoses, and special dosing/population considerations. Students completed self-assessment surveys for 12 EPAs. Students rated their ability to perform each EPA using a four-point scale (1=still developing this skill; 4=can do this independently) at the start and after each APPE semester.

Results. Data were collected from May 2016 through April 2017. During this time, 165 students completed APPEs. Students reported 79,717 encounters. There was no significant correlation found between total number of encounters and EPA scores. The baseline EPA mean score was 3.1 and semester 3 EPA mean score was 3.7. The mean student-reported EPA scores did increase over time, some more quickly than others.

Conclusion. Tracking student patient encounters provided insight into the quantity and variety of patients and conditions seen and level of care provided by students during APPEs. Mean scores on EPAs increased over time with increased exposure to patients. Patient tracking can be used to inform the curriculum by identifying potential gaps in both didactic and experiential education.

INTRODUCTION

The minimum number of hours that pharmacy students must spend on site for the four required advanced pharmacy practice experience (APPE) settings is defined by the Accreditation Council for Pharmacy Education (ACPE).¹ While the number of APPE hours students spend gaining experience at a particular APPE setting is defined, the number of patient encounters and opportunities to practice and develop patient care skills and abilities varies among practice settings and sites. This variability may have an impact on student opportunities to develop patient care skills and become confident in providing pharmaceutical care to patients.

There has been debate in clinical education as to the direct influence that the number of patient encounters has on a pharmacy student’s development and achievement of desired skills and abilities.² Further, assessment of the student may be biased by the preceptor’s own practice setting and number of patient encounters that he or she experiences and considers typical, thereby limiting interpretation of the preceptor’s assessment of the student’s skills and abilities across practice sites. While students bring their own unique skills, abilities, and experiences to the APPE, the experience specific to the site, including the number of patient encounters they have, is also be an important factor in students’ own perception of their ability to provide patient care in the practice setting.

Tracking of the significance of APPE pharmacy student clinical interventions³ and the positive impact of pharmacy student interventions^4-7 has been reported previously, as has the tracking of student patient encounters and clinical skills assessment associated with those encounters.⁸ Data on the correlation between clinical experiences and objective performance in healthcare education generally have been mixed. In medicine, McManus and colleagues and Wimmers and colleagues found no relationship between performance on examinations, preceptor evaluation, and clinical experiences.^9,10 In Kim and colleagues’ study, the quantity of patient encounters had a small but positive relationship with the performance of medical students on objective structured clinical examinations (OSCE).¹¹ In pharmacy education, less is known about the relationship between the extent of a student’s contact with patients and clinical performance. Hall and colleagues described tracking activities as a component to curricular assessment.⁷ Within pharmacy experiential education, we previously reported finding a positive association between the quantity of patients that students encountered and clinical skills they performed and their competency based on preceptor rating.⁸

Entrustable professional activities (EPAs) have emerged as an assessment model in experiential education, and have been developed and fully implemented in our institution.¹² Entrustable professional activities help translate competency-based assessment into levels of trust to measure student performance, where EPAs describe individual units of practice. Entrustable professional activities are organized into five levels, ranging from the learner only observing (lowest level) to the learner overseeing other learners (highest level). Our institution established expected EPA performance levels based on curricular milestones, including graduation, and those also became the expected performance level for students on APPEs. These are used by preceptors to assess student performance of clinical skills and abilities during midpoint and final APPE evaluations. We set out to determine whether the number of patient encounters by students during their APPEs related to students' self-assessment of their ability to perform EPAs, specifically those related to patient care.

METHODS

Patient tracking surveys were developed for fourth-year students to complete during their last three semesters, which included a 12-week acute care/institutional (AC/INST) APPE, a 15-week combined community pharmacy and ambulatory care (CPAC) APPE, and 15 weeks of electives (three 5-week APPEs). The APPE course directors and instructors developed survey templates (AC/INST and CPAC) with assistance from the college’s Office of Assessment. The CPAC template was used for required CPAC APPEs and elective patient care APPEs in a CPAC setting and students reported data for each patient encounter. Students recorded the type of encounter (new or follow-up) and care provided (comprehensive medication management, focused visit, education/counseling, self-care consult, medication reconciliation, transitions of care). The AC/INST template was used for required AC/INST APPEs and elective patient care APPEs in an AC/INST setting. Students reported data weekly, summarizing their patient encounters. Students recorded the number of intensive care unit (ICU) patients, non-ICU patients, patients new to the service, and patients discharged from the service. Students also tracked the number of patients for each type of care provided (eg, comprehensive medication management, education/counseling, inpatient consult, medication reconciliation, transitions of care), number of formal presentations delivered, and days spent participating in interprofessional or pharmacy only patient rounds. Both tracking surveys also collected the primary and secondary patient diagnoses by having students select from a comprehensive list of conditions and health states, and patient or dosing considerations (eg, kidney dysfunction, pregnancy).

The two templates were built into one electronic survey using Qualtrics (Qualtrics Labs, Inc., Provo, UT). Students accessed the tracking survey link and student instructions through the APPE course sites. Students selected their APPE type (CPAC, AC/INST or elective) which sent respondents to the corresponding template (CPAC or AC/INST). In order to link entered data to students, they also entered identifying information (username of email address).

Student instructions were developed that described the purpose, expectations, how and when to track for each APPE type (CPAC or AC/INST), and definitions for the types of care. The tracking templates and student instructions were reviewed by the APPE course directors and by other practice faculty members. The patient-tracking process was piloted with 20 students on APPEs (two in community practice, three in ambulatory care, and 15 in acute care) during a five-week block in February 2016. During the final week, students were sent a survey to gather feedback on the tracking process. The surveys, student instructions, and tracking process were then updated based on the results, feedback from the pilot survey, and discussions with the course directors and instructors.

Patient tracking was implemented in the 2016-2017 academic year, which was the first year EPAs were used in APPE evaluations. Students were expected to complete the tracking survey on all AC/INST, CPAC, and elective patient care APPEs in a CPAC or AC/INST setting. However, completion of the survey was not associated with an assignment nor was it monitored throughout the semesters. At the end of each APPE semester, tracking data was compiled to create reports, both semester totals for each APPE type (CPAC, AC/INST, electives) and individual student reports. The total tracked patient encounter data for the APPE was posted in the respective APPE course sites. Students’ individual reports were distributed to them through the course sites.

Additionally, students completed self-assessment surveys based on 12 EPA skill statements that reflected the curriculum.¹² Students rated their ability to perform each EPA using a four-point scale (1=still developing this skill to 4=I can do this independently) at the start of APPEs and after each of the three APPE semesters. The self-assessment survey was built into Qualtrics (Qualtrics Labs, Inc., Provo, UT) and students accessed it by a link on the course sites.

Data were extracted from the raw survey files and analyzed using R program, version 3.4.4 (R Foundation for Statistical Computing, Vienna, Austria). Correlations between number of patient encounters and self-reported EPA scores were explored using the Pearson method, with a two-sided statistical analysis. A p value of .05 was used as the cutoff for statistical significance. One student was excluded as an extreme outlier due to completing tracking surveys at a rate far outside the norm for the rest of the group. For the primary analysis cohort, students were assumed to have reported 100% of their patient encounters, with no extrapolation of data for days or weeks that they did not report. The University of Minnesota Institutional Review Board Human Subjects Committee determined that this project did not require review.

RESULTS

Data were collected from May 2016 through April 2017. During this time, 163 students completed APPEs. Students self-reported 79,717 encounters. During required AC/INST APPEs, 163 students documented 47,636 encounters. During elective patient care APPEs in an AC/INST setting, 117 students documented 19,921 encounters. During required CPAC APPEs, 163 students documented 10,042 encounters. During elective patient care APPEs in a CPAC setting, 94 students documented 2,118 encounters. The majority of students experienced encounters that involved patient or dosing considerations with special populations, and they reported participating in a variety of types of care. Most students were exposed to a variety of conditions.

Seventy-seven students completed the baseline EPA self-assessment survey and 102 students completed the EPA survey at the end of the third APPE semester. Fifty-three students completed both the baseline and final EPA surveys, creating the preliminary primary analysis cohort. All 53 students completed the AC/INST patient tracking survey at least once, and 50 students completed the CPAC patient tracking survey at least once. The 50 (31%) students who completed all four surveys composed the final primary analysis cohort.

The correlation statistics between reported patient encounters and change in self-reported EPA scores for the final primary analysis cohort are shown in Table 1. There was no significant correlation between total number of encounters and EPA scores, nor between total number of encounters and APPE type. Table 2 shows the mean EPA scores over time.

Table 1.

Correlation Between Pharmacy Students’ Number of Patient Encounters and Self-Reported Entrustable Professional Activity Scores

Table 2.

Changes Over Time in Entrustable Professional Activity Scores Reported by Fourth-Year Pharmacy Students Completing Advanced Pharmacy Practice Experiences

DISCUSSION

Tracking pharmacy students’ patient encounters during APPEs provided insight into the diversity of patients, level of care provided, and quantity of patients seen by students during practice experiences. However, none of the variables explored were correlated with students’ self-assessment of their patient care skills. None of the 12 EPA statements, individually or collectively, revealed differences between students who reported greater numbers of patient encounters compared to those who reported fewer encounters. The lack of correlation between student self-assessment and number of patients seen suggests that simply increasing the quantity of patient encounters is not sufficient to improve students’ self-assessment ratings using the EPAs. This finding may further support previous research in medicine clerkships that failed to find a correlation between educational outcomes and quantity of patient encounters.^9,10

The data did reveal growth trends in terms of students’ self-reported ability to perform EPAs. In other words, students self-reported EPA scores increased over three semesters, from the start of the first APPE to the end of the final APPE. However, there were differences among students in terms of starting points (their first self-assessed EPA scores) and rate of growth in their mastery of specific EPAs. The EPAs in which students progressed slowly started with an above average baseline score followed by a slower rate of growth. The three slowest EPA growth rates involved skills such as professionalism, communication, and teamwork. In contrast to these EPAs that required longer for students to master, those EPAs with the highest rates of change over the three semesters related to direct patient care decision-making. This is best demonstrated by the EPA, “Provide therapeutic drug monitoring for common medications,” on which students reported the highest rate of improvement but which had the lowest baseline score among all EPAs measured. This relationship between baseline EPA scores and rate of change over time may suggest that student’s self-rating on EPAs may reflect student readiness. For example, students rated themselves higher on skills that they practiced more often in the didactic curriculum, such as documentation, team communication, and professionalism. Conversely, students’ insecurity with drug monitoring and making therapeutic decisions, areas in which they had less experience prior to APPEs, quickly improved, approaching or surpassing the mean EPA score as they progressed through their APPEs and gained experience.

This raises the question as to the cause of the increase in EPA scores. Some may argue that time alone is the variable linked to improving students’ self-assessments of EPAs. On the other hand, students’ exposure to experiences within an APPE itself may be a related factor. While the number of patient encounters did not correlate with the increase in students’ self-assessed EPA scores, there may be a correlation between number of patient encounters and exposure to preceptors in APPEs that may account for the rise in EPA scores. Universal EPA statements were created and adopted at our institution prior to publication of the Entrustable Professional Activities for New Pharmacy Graduates.¹³ Although similar in scope, using these core EPAs rather than our EPAs may have yielded different results.

The data collected from the 79,717 student-patient encounters also revealed an understanding of the types of patients being seen by our students. Students were exposed to a variety of patients and conditions and in various practice settings. Practice settings varied in the following ways: the focus of care, such as specialty medicine vs general practice; outpatient vs inpatient; and geographic settings, such as urban vs rural. In our study, the APPE type did not determine the rate of EPA growth (Table 1). Further study of the potential impact of differences in practice settings may reveal a connection to students’ self-assessment of skills. A variety of factors may cause differences in the exposure of students to patient populations. Students may be excluded from the care team when patients deny students from participating in their care. When students are placed in practice sites with specific disease focuses (eg, a cardiac intensive care unit, HIV clinic), this may limit the scope of patient populations. Ultimately, this data directly informs the professional program of the educational opportunities’ students encounter in the experiential setting, as evidenced by Hall and colleagues.⁷ If discrepancies exist beyond acceptable ranges, quality improvement strategies may need to be implemented.

This study has several limitations, most notably regarding data collection. The accuracy of the data was dependent on students consistently and honestly documenting their experiences. Although most students completed the surveys, there was not 100% participation; thus, the generalizability of the results may have been affected. For the patient encounter and EPA analysis, the number of students included in the analysis (N=50) was lower than the total number of students. Thus, a correlation trend may have existed but could not be detected because of the smaller number of students. In addition, a broad range of patient encounters were reported by students. Although all students were given the same set of instructions, it is impossible to determine whether students entered the information consistently and as directed as this was not verified. Students may have misunderstood what we meant by being directly involved in patient care (eg, observing a comprehensive medication review vs leading a review) or forgot to document their encounters. Future training sessions may help mediate this inconsistency. Also, the timing of when students completed some of the surveys was not controlled. Specifically, students were instructed to fill out the self-assessment survey at the end of the semester; however, as the link was accessible, some students filled it out earlier than the end of the APPE semester.

CONCLUSION

The number of patient encounters pharmacy students had did not directly correlate to students’ perceptions of their preparedness to perform various patient care skills. However, students’ mean scores on EPAs did increase over time, along with increased exposure to patients. Tracking student-patient encounters provided valuable insight into the quantity and variety of patients and conditions that fourth-year pharmacy students see, as well as the level of care provided by students during APPEs. This information provides a wider and more comprehensive view of student experiences. Matching student experiences to either self-assessed or preceptor-assigned EPA scores may be an interesting area for future study. This information may be used as feedback into curricular design to both improve and integrate didactic and experiential education.