Evaluation of Clinical and Communication Skills of Pharmacy Students and Pharmacists with an Objective Structured Clinical Examination

Abstract

Objective. To evaluate how effectively pharmacy students and practicing pharmacists communicate and apply knowledge to simulations of commonly encountered patient scenarios using an objective structured clinical examination (OSCE).

Design. Second-, third-, and fourth-year pharmacy students completed an OSCE as part of their required courses in 2012 and 2013. All students in both years completed identical OSCE cases. Licensed pharmacists were recruited to complete the OSCE and serve as controls in 2012. A survey assessed student perception and acceptance of the OSCE as well as student confidence in performance.

Assessment. Licensed pharmacists had significantly higher clinical and communication skills scores than did pharmacy students. Student progression in communication and clinical skills improved significantly over time. Survey results indicated that students felt the OSCE was well-structured and assessed clinical skills taught in pharmacy school; 86% of students felt confident they could provide these skills.

Conclusion. Objective structured clinical examinations can evaluate clinical competence and communication skills among professional students. Implementation of OSCEs may be an effective tool for assessment of the Center for the Advancement of Pharmacy Education domains.

INTRODUCTION

Multiple-choice testing is a widely implemented method used to assess student performance in doctor of pharmacy (PharmD) programs. However, this form of testing may not be the best way to assess clinical and communication skills. Health care professionals are expected to exhibit good communication skills to provide effective patient care. In addition, pharmacists are expected to implement pharmaceutical care plans that require advanced clinical skills.

The Center for the Advancement of Pharmacy Education (CAPE) 2013 Educational Outcomes provide curricular guidance on the knowledge, skills, and attitudes necessary for entry-level graduates.¹ Domain 1 focuses on foundational knowledge used to evaluate scientific literature and solve drug therapy problems. Domain 2 focuses on essentials for patient-centered practice and care. Domain 3 encourages effective communication, education, and problem solving. Domain 4 promotes the ability to develop personally and professionally. Pharmacy school curricula should prepare students to not only pass licensure examinations, but also provide excellent clinical services to patients. Unfortunately, multiple-choice testing does not assess clinical ability and has limited capacity to ensure competency in CAPE domains 2, 3, and 4.²

Multiple-choice examinations are reliable for measuring knowledge, but interviewing, interpersonal, physical examination, and problem-solving skills are not assessed objectively.² Multiple-choice examinations require supplementation to fully assess competence.³ The Accreditation Council for Pharmacy Education’s (ACPE) Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree encourage the development of critical-thinking and problem-solving skills through active-learning strategies.⁴ Guideline 14.5 notes the use of standardized patients in introductory pharmacy practice experiences.⁴ The upcoming 2016 ACPE Standards will expect schools to use more varied and realistic-type assessments, which may include the OSCE, to better capture student learning.⁵

The OSCE has demonstrated validity and reliability for assessing medical trainees’ clinical skills in multiple diciplines.⁶ Students are presented with a clinical situation that cannot be duplicated on paper, such as an opportunity to assess interviewing and interpersonal skills.² Literature supports the use of OSCEs to evaluate clinical knowledge and competence, professional judgment, problem-solving skills, and interpersonal and communication skills.^6-8 The combination of the OSCE with standardized board examinations has the potential to become the gold standard for measuring physician competence.⁹

For years, physicians have used OSCEs as part of their licensing process. For instance, the United States Medical Licensing Examination and the Medical Council of Canada Qualifying Examination use OSCEs as part of their examination process.¹⁰ More recently, pharmacy bodies have begun to use this tool to assess competency among its applicants. The Canadian Qualifying Examination for pharmacists incorporated an OSCE component in 2001.¹¹ This OSCE consists of stations related to patient care; professional collaboration and teamwork; ethical, legal, and professional responsibilities; drug, therapeutic, and practice information; communication and education; drug distribution; and management principles. Between September 2008 and May 2010, Sturpe interviewed pharmacy faculty members and found that 32 schools used OSCEs in their curriculum.¹⁰

The objectives of this study were: (1) to evaluate how effectively second-, third-, and fourth-year (P2, P3, and P4) pharmacy students and practicing pharmacists assessed a medical condition, applied knowledge to implement a pharmaceutical care plan, and communicated using simulations of commonly encountered patient scenarios in an OSCE; (2) to compare student progression in communication, clinical skills, and overall performance among academic years; (3) to evaluate student perceptions and acceptance of the OSCE as an assessment tool; and (4) to evaluate student assessment of their confidence in performance.

DESIGN

Second-, third- and fourth-year pharmacy students enrolled at the University of the Incarnate Word Feik School of Pharmacy were required to complete an OSCE as part of their required courses in 2012 and 2013. Second-year pharmacy students completed the OSCE in a patient assessment course. Third-year students completed the OSCE in an advanced pharmacy care laboratory, and P4 students completed the OSCE as part of a capstone pharmacotherapeutics seminar after advanced pharmacy practice experiences (APPEs) were completed.

The OSCE accounted for 20% in each course grade. Although pharmacy students completed the OSCE as a component of required courses, only the performance results of students who provided consent were included in the analysis. Study participation was voluntary and resulted in no bonus or additional points for participating students. More than 90% of students in each class consented to participate. Licensed pharmacists were recruited to complete the 2012 OSCE and serve as the control group to assess students’ ability to perform tasks at which licensed, practicing pharmacists demonstrate proficiency. Licensed pharmacists were offered a $60 gift card as compensation. The costs of the OSCEs were covered through internal grant funding and by the pharmacy practice department. This study was approved by the institutional review board at the University of the Incarnate Word.

Each professional year, the OSCE was comprised of 4 stations consisting of a simulated task or problem. In each academic year, all students completed identical OSCE cases. The cases used in 2012 related to the treatment of hypertension, anemia, urinary tract infections, and dyslipidemia. Each case began with a 2-minute pre-encounter period followed by an 8-minute encounter and a 2-minute postencounter period. Students were asked to record a written recommendation for the anemia and dyslipidemia cases in the postencounter period. The cases used in 2013 related to management of dyslipidemia, influenza vaccination, hyperkalemia, and hypoglycemia. Each case began with a 5-minute pre-encounter period, followed by a 10-minute encounter. The disease states were previously taught in the curriculum for all students at the time of the assessments. The pre-encounter period in 2012 and 2013 allowed students to review the door sign and look up information using resources students brought to the site, which could have included paper or electronic resources. The tasks focused on assessment of 3 levels of Bloom’s Taxonomy: application, analysis, and evaluation.

A 4-member OSCE task force was created to develop and review the OSCE cases and checklists. Each member wrote a separate case and checklist, which included 10-15 items with specific answers related to the case to allow for a reliable assessment. Communication checklists consisted of the same 9 items for each case and evaluated professionalism, empathy, and clear and logical communication. All checklists items were weighted equally, with a maximum score of 100%. Individual cases were then reviewed by the other members of the task force. Finally, each case and checklist was reviewed by faculty members not involved in the development of either tool.

Each assessment was timed, and students rotated through each station. As noted above, each case consisted of a 2- to 5-minute preparation period, in which the participant reviewed the task assigned. The preparation period was followed by an 8-10-minute interaction period between the standardized patient and participant in which the participant obtained a focused history or performed a physical examination. The participants were signaled to rotate from case to case through computerized overhead announcements. Standardized patients were hired by the clinical skills center and trained by the pharmacy practice faculty members in a group setting to ensure consistency across standardized patients. Immediately after each encounter, these trained standardized patients graded the interaction based on whether or not the student performed each item on the clinical skill and communication checklists.

The 2013 OSCE was followed by a short, anonymous survey of all students to assess their perception and acceptance of the OSCE as an assessment tool. Students were surveyed on 14 different questions about the OSCE. These questions were adapted from a validated assessment tool.⁹ Responses were measured on a 5-point Likert scale of strongly agree, agree, neutral, disagree, and strongly disagree. Student completed the survey either electronically or on paper immediately after exiting the testing center.

Data were analyzed using JMP, v10.0 (SAS Corp., Cary, NC) and SPSS, v21 (IBM, Armonk, NY). Descriptive statistics were used to report student perceptions assessed through responses on a Likert scale. Nominal data were analyzed using the chi-square or Fisher exact test as appropriate. Continuous variables were tested for normality using the Shapiro-Wilk W goodness-of-fit test. Most continuous data were nonparametric; therefore, all data were reported as median and interquartile range. Nonparametric continuous data were analyzed by Wilcoxon rank sum test. Comparisons of continuous data with 3 samples were done using one-way analysis of variance (ANOVA) between groups. Paired comparisons of nonparametric continuous data were made with Wilcoxon signed rank test. All comparisons were considered significant if the p value was less than an a priori alpha level of 0.05.

EVALUATION/ASSESSMENT

In 2012, 290 pharmacy students completed the OSCE and 275 students consented to participate in the study. Six licensed pharmacists completed the OSCE and consented to participate in the study in 2012. Overall, licensed pharmacists had higher clinical skills scores than the pharmacy students (p=0.001; Table1). Pharmacists performed better than P2 (p=0.02) and P3 (p=0.002) pharmacy students; however, there was not a significant difference in performance between the P4 pharmacy students and pharmacists (p=0.07; Figure 1). Table 2 compares the scores for each case and overall averages by graduation year. The P4 students achieved higher clinical skills and overall scores than P2 or P3 students. Communication scores were high in all 3 classes on each of the 4 cases. Second-year students scored higher on clinical skills than both P3 and P4 students on the dyslipidemia case.

Table 1.

Comparison of Student vs Pharmacist Scores on 2012 OSCE

Figure 1.

Comparison of Communication and Clinical Skills Scores between Students and Pharmacists (2012 OSCE Data).

Table 2.

Comparison of Scores for Each Case and Overall Averages by Graduation Year

In 2013, 286 students completed the OSCE and 274 students consented to participate in the study (Table 2). Overall, P3 students performed better than P2 and P4 students (p=0.04). Communication skills varied by class per case; however, communication scores were higher than clinical skill scores in all classes. Overall, there was no significant difference in clinical skills between the classes (p=0.48), but this was different based on case. The P2 students performed better on the dyslipidemia and hypoglycemia cases than P3 and P4 students (p<0.0001 for both). However, P3 and P4 students performed significantly better on the hyperkalemia case than P2 students (p<0.0001).

Student progression in communication, clinical skills, and overall performance was compared between 2012 and 2013 (Figure 2). Data were available for 83 students from the class of 2014 and 85 students from the class of 2013. Both classes demonstrated significant improvement in their communication and clinical skills in 2013.

Figure 2.

Comparison of Students Progression in Communication, Clinical Skills, and Overall Performance between Academic Years.

In 2013, 281 students completed the survey to evaluate perceptions and acceptance of the OSCE as an assessment tool. Upon exiting the OSCE, students who did not consent to participate in the study were asked not to complete the survey. However, 7 additional responses were submitted. As shown in Table 3, more than 97% agreed or strongly agreed that the OSCE was a well-structured and practical experience.

Table 3.

Evaluation of Student Perceptions and Acceptance of the OSCE as an Assessment Tool

Students felt the OSCE provided an opportunity to learn from real life scenarios and challenged them to think critically. Nighty-six percent agreed or strongly agreed that the tasks reflected skills learned in pharmacy school. However, only 86% felt competent to provide the type of care seen in the OSCE. Ninety-four percent of the students felt the OSCE highlighted areas of weakness in skills and knowledge. Thirty-nine percent noted the OSCE was an intimidating assessment method; however, 64% felt it was less stressful than other examination types.

Eighty-six percent, 83%, and 92% answered that they felt confident in their performance related to cases for dyslipidemia, influenza, and hypoglycemia, respectively. This finding was consistent for all 3 classes completing the OSCE. In contrast, only 39% overall felt confident in their performance on the hyperkalemia case, which was the single acute care case included in the OSCE. This result was similar in all 3 classes (class of 2015: 35.6%, class of 2014: 42.7%, class of 2013: 42.1%; p=0.57).

DISCUSSION

The OSCE was first introduced in the 1970s and for decades has been an important assessment tool used in medical school programs across the nation.¹⁰ The interest and use of the OSCE is growing within pharmacy education. Of note, it is a major component of practice licensing examinations including the United States Medical Licensing Examination and the Canadian Pharmacist Qualifying Examination.^10,11 Our program uses OSCEs throughout the pharmacy curriculum to assess clinical ability and effective communication. The overall goal is to assess interviewing, interpersonal, physical examination, and problem-solving skills, which are difficult to evaluate using current standardized testing methods.

Our findings showed that overall communication scores were higher than clinical skill scores. Communication scores appeared strong throughout both years and all OSCE cases. In relation to clinical skills performance, licensed pharmacists performed better overall than pharmacy students, an unsurprising result given the academic background and practice experience of the licensed pharmacists compared to the students.

Of the 6 licensed pharmacists who participated in the study, 4 worked in the retail setting with at least 5 years of pharmacy experience, and 2 worked in an academic setting with a clinical practice focus in ambulatory care. Fourth-year pharmacy students’ scores were not significantly different than those of practicing pharmacists, which may be evidence of student learning of clinical and communication skills in our curriculum, particularly in APPEs. No previous data report a comparison of pharmacy students to practicing pharmacists related to OSCE performance with the purpose of ensuring pharmacy students’ competency in skills required for practice.

In the 2012 OSCE, P4 students performed better than P2 and P3 students. This result was expected because of their more advanced position in the curriculum and recent completion of APPEs. Third-year students did not perform as well as P2 and P4 students despite being more recently exposed to the disease state topics covered in the cases. In contrast to P2 students, P3 students were not at the time taking a laboratory-based course that provided the opportunity to role play and practice their clinical skills. The OSCE is the only simulation experience provided in the third year.

In the 2013 OSCE, P2 students performed better than P3 and P4 students on the clinical portion of the dyslipidemia and hypoglycemia cases. This may because P2 students had recent exposure to these clinical concepts in the cardiovascular didactic lectures and patient assessment course. Despite differences in performance among classes, students demonstrated significant academic progression in performance between 2012 and 2013.

The performance on the hyperkalemia case across all graduation classes was poor. This particular case consisted of a physician-pharmacist interaction in which the pharmacist gathered and assessed new information, provided an initial recommendation to the physician, and responded to physician questions. Poor performance may potentially be explained by a lack of knowledge retention on the topic. In addition, students were not exposed to this type of communication interaction prior to receiving the hyperkalemia case. Because of these findings, the SBAR communication method (Situation, Background, Assessment, and Recommendation) was added to the curriculum and is taught to P2 students in the patient assessment course.

Evaluation of student perceptions, acceptance, and confidence related to the OSCE were secondary objectives of this study. Students agreed that the OSCE was a challenging, yet fair assessment that evaluated real-life practical experiences. In comparison to results of similar surveys among pharmacy students, our results are consistent with respect to the OSCE providing a real-life practical experience.^7,8,12

In contrast to other surveys of pharmacy students, more of our students responded that they agreed that instructions were clear and that the OSCE was less stressful than other examination types.^7,12 This may because the OSCEs were tied to laboratory-based courses, in which students typically had higher grades than in didactic courses. Most students felt confident in their performance on the hypoglycemia case, which correlated to their high performance score on this case. Few students felt confident in their performance on the hyperkalemia case, which correlated to their low performance score on this case. As a result of this finding, more acute care cases have been integrated into our OSCE. In contrast, 86% of all students felt confident on the dyslipidemia case despite low performance scores.

Potential limitations to our study exist. Because of limited funding, only a few practicing pharmacists were able to perform the OSCE. This restricted our ability to evaluate further how practical experience affected OSCE performance. There was a small potential for selection bias, as licensed pharmacists were compensated for their time. While the OSCE is an effective assessment tool, it incurs higher resource usage than other assessment methods. This includes higher direct costs and faculty workload for case development and validation. In 2013, the pre-encounter OSCE preparation time was increased from 2 minutes to 5 minutes based on feedback received in class evaluations. This was provided to ease testing anxiety and could have partially contributed to the significant increase in student scores as they progressed between 2012 and 2013.

Finally, trained standardized patients graded the interaction based on a rubric that consisted of clinical skill and communication checklists. The checklists were created to facilitate grading by the standardized patients. Sturpe found that 47% of US colleges and schools of pharmacy that use OSCEs have standardized patients serve as the grader.¹⁰ Some studies have noted a difference in consistency or reliability between standardized patients and health care professionals. However, the majority of studies show that standardized patients may score performance as reliably as experts.^13,14 There are exceptions when standardized patients may find judging more difficult; therefore, skills e being assessed should play a role when selecting the grader.¹⁴

CONCLUSION

This study provides a strong comparison of the clinical and communication skills of practicing pharmacists and pharmacy students, and among pharmacy students at different stages of the curriculum. This study adds to the increasing body of research supporting OSCE as a type of clinical skills assessment for pharmacy students. It also provides significant insight regarding the use of OSCEs in a PharmD curriculum as a way to evaluate clinical competence and communication skills among professional students. The results of the OSCE can be used to evaluate curriculum and potentially modify it. A future goal is to use the OSCE throughout the program to assess student progression and competence at critical stages in the curriculum, such as at the conclusion of educational training, before and after APPEs. Implementation of OSCEs may be an effective tool for assessment of the CAPE domains and critical stages in the curriculum.