A Comparison of Students’ Self-Assessments With Faculty Evaluations of Their Communication Skills

Abstract

Objective. To compare students’ self-assessment of their communication skills with faculty members’ formal evaluation of their skills in a therapeutics course.

Methods. Over a 3-year period, faculty members evaluated second-year pharmacy students’ communication skills as part of a requirement in a therapeutics course. Immediately following an individual oral assessment and again following a group oral assessment, students self-assessed their communication skills using the same rubric the faculty members had used. Students’ self-assessments were then compared with faculty members’ evaluation of students’ communication skills.

Results. Four hundred one (97.3%) students consented to participate in this study. Faculty evaluation scores of students for both the individual and group oral assessments were significantly higher than students’ self-assessment scores. Students’ self-assessment scores of their communication skills increased from the individual to the group oral assessment.

Conclusion. Students’ self-assessments of communication skills were consistently lower than faculty members’ evaluations. Greater use of oral assessments throughout the pharmacy curriculum may help to improve students’ confidence in and self-assessment of their communication skills.

INTRODUCTION

The Center for the Advancement of Pharmaceutical Education (CAPE) and the Accreditation Council for Pharmacy Education (ACPE) advocate for patient-centered pharmaceutical care.^1-3 The CAPE Outcomes recommend that pharmacy graduates must be able to communicate and collaborate with prescribers, patients, caregivers, and other involved healthcare providers to foster a team approach to patient care.² CAPE also recommends that pharmacists communicate clearly with patients and other healthcare providers to improve clinical and economic outcomes as well as patient reported outcomes like satisfaction and health-related quality of life.³ Because of this, ACPE recommends that colleges and schools of pharmacy teach students how to communicate research and clinical findings to interprofessional and interdisciplinary audiences as part of their professional communication development and to ensure effective delivery of patient care.¹

Colleges and schools of pharmacy have incorporated communication skills development and assessments into their curriculum in order to increase students’ competency and proficiency in this skill. To date, the published literature on communication skills training and assessment in pharmacy education is focused on communication with patients.^4-8 There are a limited number of studies on the importance of pharmacy students developing appropriate communication skills with other healthcare providers.⁹ Similarly, in medical education, core behaviors and interpersonal patient-relation skills have been developed that are taught to medical students and assessed.^10,11 However, the focus is on the assessment of communication with actual or simulated patients rather than with other healthcare providers or peers.

At the College of Pharmacy and Health Sciences at Mercer University, there are many opportunities to reinforce students’ communication skills with patients including through a communications course, introductory pharmacy practice experiences, mock patient counseling, and oral examinations. Fewer opportunities exist early in the pharmacy curriculum for students to strengthen their communication skills with healthcare providers, and include only introductory pharmacy practice experiences and 4 classroom hours of instruction in and application of presenting clinical recommendations on drug therapy. Most occasions for students to practice their communication skills with healthcare providers do not occur until they begin their advanced pharmacy practice experiences in the fourth year.

Self-assessment is an important component of pharmacy students’ learning. ACPE recommends that colleges and schools of pharmacy encourage and assist students in assuming responsibility for their learning through self-assessment of their level of knowledge, skills, attitudes, values, and achievement of desired competencies.¹ ACPE also encourages both student self-assessment and faculty evaluation of professional competencies and behaviors including communications skills. To evaluate students’ communication skills with healthcare providers and students’ self-assessment of these skills, second-year pharmacy students’ self-assessments of their communication skills were compared with faculty evaluations of their communication skills-based performance during therapeutics oral examinations.

METHODS

Over a 3-year period, 2 patient case-based oral assessments were given to all second-year students enrolled in a required Cardiovascular/Renal III therapeutics course. Students were provided with a patient case prior to each oral assessment to allow adequate preparation time. Also, each patient case covered disease state and pharmacotherapy knowledge on which the students were tested in written format as well.

The first oral assessment was conducted individually, with 1 faculty member assessing 1 student, and consisted of the faculty member asking the student clinically relevant questions about the assigned patient case. To use as a reference during the oral assessment, students were provided with a copy of the pharmacotherapy note they had written and submitted for the patient case.

The group oral assessment took place approximately 1 week after the individual oral assessments and involved groups of 4 students and 2 faculty members. Each group of students presented their assessment and plan for the patient case provided, incorporating relevant subjective and objective information. Faculty members then asked clinically relevant questions regarding the patient case and the supporting primary literature. During the individual and group oral assessments, the student(s) simulated a pharmacist developing and communicating therapeutic recommendations to the faculty member(s) who simulated another/other healthcare provider(s) (peer-to-peer communication).

Faculty members used a scoring rubric developed by the course coordinators to evaluate students’ communication skills during both the individual and group assessments. The rubric was divided into 2 areas: rapport (demonstrates confidence; maintains respectful tone; attentive; absence of obvious nervousness) and presentation of therapeutic recommendations (correct professional pronunciation and phraseology; individualized approach; well-prepared; concise and articulate). Immediately following each oral assessment, students were asked to assess their own communication skills using the same rubric. The faculty members involved in evaluating the students’ communication skills were licensed pharmacists and therefore provided students with a realistic experience of communicating with a professional peer.

To increase inter-rater reliability among faculty members evaluating students’ communication skills, faculty members were oriented to the scoring rubric prior to the oral examination and shown how to apply the rubric. They were also provided with more detailed descriptions of how to rank students using the 4-point Likert scale based on the frequency with which students performed the assessed skill (ie, 1=none of the time, 2=some of the time, 3=most of the time, 4=all of the time). Additionally, prior to the individual oral assessment, the participating faculty members were shown an example of a mock oral assessment.

Several days after the oral assessment, each student received faculty feedback on their communication skills via their graded rubric. Course coordinators reviewed each completed rubric and finalized the student’s grade prior to providing a copy of the rubric to the student. Students were provided with faculty feedback on their communication skills on the individual oral assessment prior to the group oral assessment.

Students’ self-assessments of their communication skills during the individual and group oral assessments were compared with the faculty members’ evaluations. In addition, students’ self-assessments of communication performance on their individual and group oral assessments were compared. Faculty evaluations of each student’s communication performance on the individual and group oral assessments were also compared.

This study was approved by Mercer University Institutional Review Board. Students signed informed consent forms prior to participation. Descriptive statistics and Wilcoxon signed rank tests were used to compare students’ self-assessments of their communication skills on the individual and group oral assessments, and to the respective faculty member’s evaluation of communication skills. Statistical Package for the Social Sciences (SPSS) version 16.0 (SPSS Inc, Chicago, IL) was used for data analysis. Incomplete communications assessments were handled using the SPSS option “excluded cases pairwise.” Therefore, survey instruments with missing data were excluded only if they were missing the data required for the specific analysis.

RESULTS

Four hundred one (97.3%) students consented to participate and completed communication self-assessments over a 3-year period. Demographic information was not collected through this study, however, demographic information was available from admissions data on students in the 3 class years. The mean age of students was 26 years with a range of 20 to 58 years; 36.7% of students were male, 63.3% had a prior bachelor’s degree, and 38.1% were minorities (black, Asian/Pacific Islander, or Hispanic). An average of 9 faculty members from the Department of Pharmacy Practice conducted the individual oral assessments each year, and 6 faculty members participated in the group oral assessments each year.

For the individual oral assessment, students’ mean self-assessment score was 3.2 ± 0.5 and the mean faculty assessment score for the students’ performance was 3.5 ± 0.4 . For the group oral assessment, students’ mean self-assessment score was 3.4 ± 0.5 and faculty members’ mean assessment score was 3.5 ± 0.3 . Faculty members’ individual oral assessment scores were significantly higher than students’ self-assessment scores in all 8 areas of the scoring rubric (p<0.01) (Table 1). For the group oral assessment, faculty members’ scores were significantly higher than the students’ self-assessment scores in the following areas: demonstrates confidence; correct professional pronunciation and phraseology; absence of obvious nervousness; well-prepared; and concise and articulate (Table 2). Students’ self-assessment scores of their communication skills increased from the individual oral assessment to the group oral assessment for all communication skills assessed.

Table 1.

Individual Oral Assessment of Second-Year Pharmacy Students’ Communication Skills Using a Rubric (N=401)

Table 2.

Group Oral Assessment of Second-Year Pharmacy Students’ Communication Skills Using a Rubric (N=401)

DISCUSSION

Oral assessments are designed to provide students with an opportunity to develop and demonstrate their command of spoken presentation content. Some students will perform better in an oral format than a written format and vice versa, and there is not always a correlation between written and oral assessment results.^12,13 Oral assessments allow students to prepare and present findings in a context relevant to professional practice.

Oral assessments also provide students with an opportunity to use higher-order thinking skills, including analyzing and evaluating information, and to communicate that information in an appropriate manner to other healthcare professionals. These assessments further evaluate students’ level of competence in both their knowledge and communication skills. In the pharmacy curriculum, oral assessments may further help to facilitate ACPE’s goal to prepare students for communication with other healthcare professionals in the effective delivery of patient care.¹ It also fosters students’ self-assurance to perform this task, especially with repeated oral assessments, as was evidenced in this study. The Pharmacy Board of Australia incorporates an oral examination as the final assessment of a pharmacy intern’s competence to practice as a pharmacist.¹⁴ In addition to evaluating the candidate’s knowledge and skills and application to practice-based scenarios, the oral assessment also evaluates the intern’s ability to communicate with patients and healthcare providers using appropriate language.

Self-assessment is a critical aptitude for pharmacy students and practitioners.¹⁵ Within professional practice, self-assessment is the basis on which continuous professional development and self-directed learning is constructed.^15,16 However, there is an assumption that students are ready to self-assess upon entry into the profession. In this study, students’ self-assessment of communication skills were consistently lower than the evaluation scores provided by faculty members in both the individual and group oral assessments.

Students’ lower self-assessment of communication skills may have been caused by a lack of practice in the verbal communication of clinical recommendations, which is supported by higher student self-assessment scores on the second oral assessment. Familiarity with an activity has been shown to positively affect students’ ability to self-assess performance of that task.¹⁷ In addition, students’ lack of experience in self-assessment may also account for the disparity in self-assessment scores in comparison to the faculty members’ evaluation.¹⁵ Another possible explanation is that students tend to judge themselves more harshly than faculty members. In areas where students tended to be particularly self-critical, providing them with faculty evaluations and feedback could result in a more balanced assessment. Students have noted that feedback is essential in developing accuracy in self-assessment.¹⁸

Students’ self-assessments of their communication skills were higher in the group oral assessment setting than in the individual oral assessment setting. Because the group oral assessment occurred after the individual oral assessment, self-assurance in skills and performance of communicating therapeutic recommendations may have increased, and the students likely felt more self-confident with the support of their peers.

In a study designed to evaluate the accuracy of the self-assessment skills of 80 senior-level bachelor of science pharmacy students, the authors found that students overestimated their clinical knowledge and communication skills compared to assessments by external raters.¹⁹ This was particularly evident among students in the lower quartile of performance, while those in the highest quartile significantly underestimated their self-assessment scores. They noted that accurate and appropriate self-assessment skills do not come naturally as a skill or propensity. In another study evaluating first-year pharmacy students’ self-assessment of communication skills with standardized patients before and after a videotaped review of the session, students were found to have poor correlation between their self-assessment scores and those of the faculty member even after the video review.¹⁷ Poor correlations have also been found between medical students’ self-assessment of their skills and their actual performance.^20-22

Students’ tendency to overestimate or underestimate their ability suggests that students may not have the accurate self-assessment skills they are assumed to possess. In this study, feedback was provided to the students several days after the oral assessment in the form of the evaluator’s grading rubric; however, no verbal feedback or discussion opportunity was provided. In the future, providing individualized, face-to-face feedback regarding communication skills will be considered to enhance student development of self-assessment and communication skills.

The participants in this study represent 1 college of pharmacy’s second-year students, limiting the external validity of the findings. Student self-assessment data could have been skewed by those not fully engaged in the research process after the completion of the oral assessments. In addition, although steps were taken to increase inter-rater reliability in the evaluation of students’ communication skills, inter-rater reliability was not assessed in this study.

Evaluation of students’ communication skills with healthcare providers may contribute to a better comprehension of communication with other healthcare professionals in the practice of pharmacy. Greater use of this method of evaluating communication skills may improve student’s confidence and competence in the verbal communication of clinical recommendations to healthcare providers and better prepare them for practice as future pharmacists. In addition, appropriate self-assessment of performance is a key component of professional development. Incorporating self-assessment activities in various aspects of the curriculum may contribute to students’ development and improvement of this skill.^17,18

CONCLUSION

Oral assessments are an essential component of pharmacy students’ education as they provide students with an opportunity to develop both their communication and self-assessment skills. Colleges and schools of pharmacy should consider the incorporation of oral assessments into their curriculum as a means for students’ to demonstrate knowledge of and competence in patient and interprofessional communication skills. This may help to contribute to the overall goal of developing knowledgeable and competent pharmacists capable of contributing to patient care in collaboration with other healthcare providers. Future research can evaluate the impact oral assessments have in students’ preparation for communication with healthcare professionals during their experiential education as they prepare for professional practice.