Pharmacy Preceptor Judgments of Student Performance and Behavior During Experiential Training

Abstract

Objective. To report the findings of how Canadian preceptors perceive and subsequently evaluate diverse levels of trainees during pharmacy clerkships.

Methods. Using modified Delphi technique, 17 Doctor of Pharmacy (PharmD) preceptors from across Canada categorized 16 student narrative descriptions pertaining to their perception of described student performance: exceeds, meets, or falls below their expectations.

Results. Twelve (75%) student narratives profiles were categorized unanimously in the final round, six of which were below expectations. Out of 117 ratings of below expectations by responding preceptors, the majority (115, 98%) of post-baccalaureate PharmD students described would fail. Conversely, if the same narrative instead profiled a resident or an entry-to-practice PharmD student, rotation failure decreased to 95 (81%) and 89 (76%), respectively.

Conclusion. Pharmacy preceptors do not uniformly judge the same described student performance and inconsistently apply failing rotation grades when they do agree that performance falls below expectations.

INTRODUCTION

Experiential training is an integral component of Canadian pharmacy education.¹ Conducted in outpatient and inpatient practice settings under the mentorship of experienced pharmacists throughout the country, students have the opportunity to reinforce and shape development of existing knowledge and skills while engaged in direct patient care.² While campus-based instruction may employ simulation strategies (eg, exercises using dispensing software and role plays with standardized patients) useful for students to apply knowledge and rehearse provision of care without risk to real patients, these cannot completely replicate the conditions and interactions in authentic practice environments. ^3-6

Supervised pharmacy practice experiences are similarly invaluable for the public. Some provincial and territorial regulatory bodies mandate minimum internship training periods following completion of the pharmacy degree and prior to licensure to further ensure candidates’ abilities to provide competent care.⁷ In the face of ongoing contributions of preventable medical error to the morbidity and mortality within health care systems, patient safety principles are being increasingly emphasized and incorporated earlier in pharmacy and other health professional curricula.⁸ However, demonstrated mastery of theoretical content does not guarantee translation into professional competencies in Canada’s diverse care settings. It is not uncommon for health professional students with high academic standing to perform poorly during clerkships.^9-11 Supervising pharmacists model professional behaviours and guide novice learners toward independent and responsible patient care.¹² Pharmacy preceptors then not only serve vital functions as student teachers and mentors, but also as guardians for patient safety.

Pharmacy programs rely heavily on practice-based partners to provide experiential training opportunities and guide students under conditions of real professional work in their care settings. Demand for student placements will assuredly rise given both the expansion of class sizes and increasing early-program and overall experiential hour requirements outlined by the Canadian Council for Accreditation of Pharmacy Programs (CCAPP).¹³ Most provinces’ conversion of the Baccalaureate (BSc, BSP) to the Doctor of Pharmacy (PharmD) as the entry-to-practice degree will be coupled with the necessary reconfiguration of community- and hospital-practice pharmacy residencies and post-baccalaureate PharmD programs. As such, challenges associated with the inherent dual expectations placed on pharmacist preceptors to serve as both a teacher and an evaluator may be further exacerbated.

The assessment of health professional students in workplace-based settings may be considered excessively variable and subjective.^14,15 While one might surmise variations in evaluation are accounted by differences in trainee performance in changing settings or contexts over their clerkship experience, research shows it may be largely attributed instead to the preceptors.¹⁶ Study of medical student and residency training find sources of rater subjectivity include a preceptor’s body of supervisory experience as increased exposure to students achieving varying performance levels over time shapes assessment practices.¹⁷ Perceptions of competence or clinical ability likely also differ among preceptors resulting in variable student clinical internship scores.¹⁸ We are familiar with voiced student experiences with a “tough” preceptor. While efforts are made to increase the objective rigor of workplace-based assessment (performance dimension and frame-of-reference training, as examples) the context of a clinical clerkship still requires the assessor to judge and subsequently evaluate the student.^19,20 It is unclear if preceptors form judgments of student performance and behavior according to their prior teaching experiences and/or personal perceptions of competence, how this might account for the assessment of diverse levels of trainees in this new landscape of Canadian pharmacy student education. We therefore sought to explore how pharmacist preceptors perceive and subsequently evaluate students during experiential training.

METHODS

In a prior study, 20 preceptors in post-baccalaureate PharmD programs in Canada and the Middle East gave personal accounts of students (nameless) they have supervised during the experiential training component of their program.²¹ Following constructivist grounded theory principles, 16 narratives were devised to describe students who fell below, met or exceeded expectations according to the 14 total categories of performance and behavior outlined in these interviews (Table 1). Each narrative encompasses a range of 8-12 of these elements in the overall story. Ten narratives described features related to capabilities to deliver safe patient care in either outpatient or inpatient settings (Appendix 1). The descriptions were further framed as students who were two-thirds through their respective training programs.

Table 1.

Major Categories Arising From Interviews With PharmD Preceptors Discussing Graduate Student Expected Performance and Behavior ²⁰

Using a modified Delphi technique, 17 pharmacists from across Canada who precept post-baccalaureate PharmD students categorized these narratives according to their perceptions of the outlined performances or behaviors: exceeds, meets, or falls below their expectations.

In Delphi methodology, data is gathered from respondents within their domain of expertise.²² While panel numbers may vary widely (from 10 to more than 1,000), it has been demonstrated that heterogeneous groups allow consideration of different perspectives and high quality consensus.^22,23 We drew from a convenience sample of pharmacists from six provinces who had been precepts from 3 to 20 years. Participants accessed the narratives electronically which were presented in random sequence to minimize responder bias. In the subsequent round of review, aggregate results were shared with participants who could then alter their own initial choices. Preceptors were additionally asked to indicate if they would fail the student described in the narratives they had rated below expectations and if this judgment would change if the described PharmD student was instead a pharmacy resident (post-baccalaureate) or an entry-to-practice PharmD (EPPD) student. In Canada, the entry-to-practice degree currently remains the Bachelor of Science in Pharmacy (BScPharm) or (BSP). Pharmacy residency programs typically enroll pharmacists with a baccalaureate degree and are not usually pursued by pharmacists who have obtained the PharmD degree. A PharmD student in Canada then has historically been a learner pursuing subsequent additional post-baccalaureate training (which may have been preceded by a pharmacy residency). The trajectory of pharmacy practice education would be baccalaureate, optionally followed by residency and/or PharmD. However, a national shift is underway to transform pharmacist training to the PharmD degree as the entry-to-practice credential, as it is in the United States. Baccalaureate programs will be discontinued and the educational path for new pharmacy students will soon become EPPD, optionally followed by the pharmacy residency.

The proportion of ratings was calculated for the three levels of expectation categories for each description. Consensus of student narrative categorization was considered achieved if the level of agreement among participants was greater than 80% (14/17=82%).²³ Overall rotation failures assigned to PharmD students, residents, and EPPD students by the participants were calculated. The study was approved by the Qatar University Institutional Review Board.

RESULTS

Pharmacist preceptors from six provinces (10, 59% female) participated. Consensus judgments were obtained for only 8 (50%) described student narratives following the first round and increased to 12 (75%) in the final consultation round to establish agreement (Table 2). Six of these were unanimous categorization of described students who fell below expectations. Out of the 117 ratings of below expectations among student narratives for which consensus judgments were achieved (profiles 2, 4, 7, 8, 13, 14, 16), the majority (115, 98%) of post-baccalaureate PharmD students would fail the rotation. When the same described student was instead considered as a post-baccalaureate pharmacy resident, this failure rate decreased (95, 81%), as it did for an EPPD student (89, 76%).

Table 2.

Preceptor Student Narrative Categorization and Rotation Failure Assignment

Preceptors assigned discrepant rotation failure judgments when considering different learner levels for four students profiles: 8, 10, 13, 16 (Table 2). In only one instance (student profile 13), did preceptors more often offer a fail for an EPPD student than the pharmacy resident (58% vs 50%). Unfavorable features described in these specific narratives included the inability to apply knowledge, to work independently, or to meaningfully contribute to team decision-making. Narratives describing performance or behavior having potentially negative impact on patient safety were characterized below expectations by all preceptors for PharmD students, residents, and EPPD students; however, 9% and 8% of preceptor ratings for residents and EPPD students, respectively, were still assigned a pass for the rotation.

DISCUSSION

This study assessed how a sample of Canadian preceptors classify and determine pass-fail decisions for students according to written narratives describing experiential behaviors. Participants held EPPD students to a lower standard versus post-baccalaureate PharmD students, as demonstrated by higher failure rates for those PharmD students classified as “below expectations.” The same cannot be said for comparisons between residents and EPPD students, where similar failure rates were observed for “below expectations.” When failure decisions were examined further it appeared that deficiencies in motivation, independence, responding to feedback, time management, and superficial patient care seemed to be consistent among narratives designated as “failures.” It is evident that preceptors have differing competency expectations across these three comparison groups, yet the reasons are unclear. These observed differences could be attributed to preceptors’ perceived higher level of practice for the post-baccalaureate trainees in that unlike entry-to-practice PharmD students, pharmacy residents and post-baccalaureate PharmD students in Canada have already met requirements for pharmacist licensure and should exhibit advanced skills gained through prior workplace-based experiences in undergraduate clerkships or employment. In contrast, preceptors may be more lenient and assign less rotation failures to EPPD students whose performance falls below expectations.

As in studies of medical student training, we found pharmacy preceptors do not share perspectives when evaluating the same student performance.^16,17,24 For example, Holmboe and colleagues have described how physician clinical faculty viewing the same videotape of residents’ performance assigned significantly varying scores on a 9-point Likert scale, ranging from unsatisfactory to superior.²⁵ Similarly, documented clinical evaluation scores of the same observed student performance by nursing instructors and nursing mentors exhibited poor or no level of agreement.²⁶ Inabilities of preceptors to form common judgments has been attributed to various factors, including the differences in frames of reference invoked when these individuals assess students. Preceptors will use other clinicians and themselves as comparative benchmarks for evaluation, and these perspectives are obviously not uniform.²⁷ Differences in workplace-based assessment may also be accounted for by different preceptor traits, including how those supervisors categorized higher in neurotic domains on a personality inventory actually contributed more and better quality of student feedback than others.²⁸ Variability in assessed performance can be frustrating for students, and not a benign phenomenon, because how preceptors distinguish between performance perceived-to-be below expectations and performance that meets expectations has implications for students’ ability to satisfactorily complete the clerkship, and to ultimately graduate.^29-32

Another important consideration is how pharmacy preceptors perceive the same performance, but for different levels of students. To our knowledge, this has not previously been investigated in health professional training, although in one study, medical student supervisors were interviewed to elicit what teaching strategies they employ to teach different levels of learners.³³ Pharmacists are recruited by faculties to precept students enrolled in all years of the pharmacy program, as well as in community and hospital residencies and other types of study traditionally considered “graduate” training. How will preceptors contextualize the performance of a student on rotation in the final year of an “entry-to-practice PharmD” program in the context of prior experience with senior baccalaureate pharmacy students or “post-baccalaureate PharmD” students? Study of assessors in medicine has shown how the preceding supervisory experience can influence evaluation of the subsequent student, in that comparative performance may be rated as unduly good or unduly poor, regardless of the clearly described criteria.³⁴ However, our findings do illustrate that preceptors consider certain student features of performance as “non-negotiable” in terms of expected performance, regardless of student status. These traits and abilities described in the student narratives included those associated with a provision of safe patient care: making appropriate recommendations, demonstrating responsibility for follow-up, recognizing limitations and proactively seeking preceptor consultation.

The public’s confidence in pharmacist abilities to safely deliver, advise, and manage drug therapy is an important factor in the trust notably and consistently afforded the profession. It should be no surprise then that trust is often expressed as an intangible element in workplace based assessment for students and employees alike. Supervisors will cite a “gut feeling” about an individual’s capabilities and offer them increasing levels of responsibilities. ²⁷ Workplace-based assessment in health professional education has begun a gradual shift to alternative means of student evaluation to harness such instincts. The concept of entrustable professional activities (EPA) has been proposed and adopted for postgraduate medical programs in the Netherlands and soon in the U.S.^35,36 An EPA is considered a “unit of professional practice” or a discrete collection of tasks, that must be assessed during training, as determined by opinion leaders in the profession.³⁷ A preceptor would determine what level of trust they would grant to a student in assuming specific care responsibility once he or she has demonstrated necessary competence. Examples of progressive assessed independence include trust to: observe only, be observed by a supervisor in the room, have a supervisor distantly available for double checks, act unsupervised, and supervise others.³⁷ The American Association of Colleges of Pharmacy (AACP) is the first organization to pursue development of EPAs for pharmacy students and has initiated stakeholder consultation on a first draft of a core set of intended activities (modeled after 6 of their 15 educational outcomes) and the level of trusted performance expected by graduation.³⁸ In addition to better grounding assessment of experiential learning according to what students actually do, evaluation is explicitly tied to the trusted level of independence permitted for the student’s provision of care which has positive implications on patient safety.^39,40 Grounding assessment strengthens the ability of decisions to be agreed upon by one or more impartial observers as opposed to conclusions that would not be supported.

Irrespective of chosen framework, experiential training assessment processes for health professional students must be credible, fair, and defensible.⁴¹Central to this premise is the collection of independent judgements from many raters on multiple occasions over time.⁴¹ Developing competencies are not static and perceived shortcomings within a given situation is not necessarily a predictor for application in another contetx.⁴³ Rater discrepencies illustrated in our study underscores the importance for programs to ensure pharmacy students have opportunities to demonstrate their attitudes and abilities in other clinical settings under the supervision of different preceptors, especially when confronted with a poor evaluation. Finally, ascribed judgements must be accompanied by formative feedback for students to sufficiently understand how he or she might address preceived limitations (and reinforce strengths) to position learning as the driver of assessment.⁴⁴

Findings of this study must be interpreted in light of several limitations. First, many participants lacked exposure to the supervision of entry-to-practice PharmD students; therefore, rater judgments made for students described in this course of study were likely speculative or based on preconceived expectations. Pharmacists chosen to supervise students may have focused on distinct clinical areas or patient population. Although the narratives addressed competencies relevant to pharmacy practice as a whole, there may have been certain components of a specialty practice not addressed that might have associated with preceptor expectations of our participants. For example, no pharmacist currently acting as a preceptor in a community pharmacy practice was included in our sample. However, we believe the general nature of the narratives were applicable to most clinical practice settings and anecdotal feedback from participants indicated how the descriptions resonated with them. Lastly, this study did not explore assessor cognition to determine why choices were made and what student factors were highly valued for pass-fail decision-making. Future studies should address this point to better understand how competency expectations are processed and eventually scored according to assessor judgments.

CONCLUSION

Pharmacy preceptors do not uniformly judge the same described student performance and inconsistently apply failing rotation grades when they do agree that performance falls below expectations. Given the changing landscape of Canadian degree and residency programs, strategies to address anticipated assessment challenges for pharmacist preceptors involved in all parts of the pharmacy student’s continuum of education must be considered.

Appendix 1. Sample Narrative Student Profile

M is a student who initially appears to have good potential. Throughout the rotation, M is enthusiastic about learning the “ins & outs” of care in the practice setting and is always seeking preceptor feedback for improvement. M is quickly accepted by the pharmacy staff and allied health members and they see how kind and conscientious M is with patients. M spends significant time listening to patient concerns and preparing educational resources and takes the initiate to locate other health information on the patient’s behalf. When making recommendations, M is an advocate for expressed patient needs, but sometimes the urgency of these medication problems are not perceived by the prescriber as important as more life-threatening issues. M does not always effectively justify the recommendations to be understood by the prescriber.

M has some difficulty making progress early in the rotation. Despite spending time with the preceptor discussing specific topics with relevant case examples, M continues to overlook these issues among drug-related problems identified for the patients. The preceptor must correct M several times before M “catches on.” M is very diligent in accomplishing assignments, but has often started in the wrong direction. M will make a detailed, well-written, well-resourced report on something that was the opposite of what the preceptor actually assigned. With direction, M will eventually finish the task, but M will have wasted time focusing elsewhere. M understands the preceptor’s perspective in all feedback interactions and makes corrective effort. The preceptor spends time working on these retention and prioritization issues and M eventually “gets up to speed” by the end of the rotation.