An Exploratory Mixed Methods Study of Standardized Patient Comments on Empathy and Student Communication Scores

Abstract

Objective. To quantify student pharmacists’ communication ability based on scores from standardized patient (SP) communication rubrics, describe and categorize SP comments about student empathy, and test the relationship between students’ communication scores and empathy.

Methods. A concurrent mixed methods research design was used to assess a graded performance-based assessment (PBA) of student pharmacists that had been conducted at one college of pharmacy. The PBA rubrics (n=218) completed by SPs contained 20 assessment items and space for open-ended feedback. Scoring categories for communication assessment included: yes, inconsistent, no, and not applicable (N/A). Descriptive statistics were calculated for rubric scores. Feedback from standardized patients was analyzed and used to categorize student interactions during the encounter as reflecting high empathy, mixed empathy, or low empathy. Kruskal-Wallis ANOVA was used to test the relationship between empathy category and communication score.

Results. Standardized patients had written comments on 141 of the 218 rubrics (64.7%). The mean communication score was 39.0±1.6 (range, 31-40) out of a maximum 40 points. The total scores for the low, mixed, and high empathy category transformations were 6 (4.3%), 95 (67.4%), and 40 (28.4%), respectively. The results of the Kruskal-Wallis ANOVA were significant, suggesting that communication scores were different between empathy categories.

Conclusion. There was a positive association between students’ scores on communication rubrics and student empathy categorization, with student pharmacists exhibiting different levels of clinical empathy. While the PBA of interest was not specifically focused on empathy, SPs frequently provided feedback about empathy to students, suggesting that showing empathy during the encounter was important.

INTRODUCTION

Empathy is a critical component of patient-centered communication, professional development, and pharmacy education and practice.^1-6 Empathy, as it applies to health professionals, can be described as a cognitive attribute that involves an understanding of a patient’s concerns, the capacity to communicate this understanding, and an intention to help.^5,7,8 The concept of clinical empathy is useful for describing dialogue in patient-provider clinical relationships.^9,10 Clinical empathy requires the provider to understand the patient’s perspectives, feelings, and meaning of their clinical situation; communicate their understanding to the patient and check its accuracy; and act on that understanding in a helpful and therapeutic way.^9,11-13 Empathy, when demonstrated in clinical encounters, has been shown to improve the patient experience, increase adherence to treatment recommendations, improve health outcomes, and enhance trust.^14-18 In addition to the positive effects on patient experiences and care, empathy also has been shown to provide personal benefits to providers, with evidence that providers who engage in empathetic communication experience greater professional fulfillment and decreased feelings of burnout.^9,19 With empathy having several implications for pharmacy practice, patient care, and provider well-being, pharmacists and student pharmacists stand to benefit from education and training focused on improving empathetic communication and clinical empathy. As such, empathy and empathetic communication have been included as a required component of accreditation standards related to providing patient-centered care and are skills that can be taught to health professionals.^2,7,20

To determine the effectiveness of empathy training for professional students, multiple studies have used cross-sectional assessments, with emphasis on evaluating educational interventions to increase the use of empathetic patient-centered communication while performing patient consultations.^20,21 Recent studies have focused on assessing and cultivating student pharmacist empathy.^22-25 One way of cultivating empathy in health care professional students is having them participate in role-playing activities, which frequently employ standardized patients (SPs) who engage with students in simulated clinical encounters and then assess the students’ communication abilities.²⁶ The focus of an SP in these encounters is not unlike that of an actual patient, making the SP well-equipped to provide valuable feedback that will assist students in their empathy and communication development.²⁷ Experiences with SPs are an essential component of comprehensive empathy assessment and may act as an educational intervention to improve empathy and self-reflection while measuring empathetic communication more objectively than self-assessment measures alone.^28-30

Given the increased emphasis on facilitating student acquisition of empathetic communication skills, pharmacy educators should keep in mind that empathy and empathetic communication skills actually decrease as individuals receive additional clinical education and experience.^31,32 Given the importance of clinical empathy and the use of SPs to facilitate the acquisition of and assess empathetic communication skills, exploring how SPs evaluate students on communication and empathy using existing communication assessment rubrics may provide additional insight on the importance, acquisition, and exhibition of clinical empathy within simulated clinical encounters. The objectives of this study were to quantify students’ communication ability using SP communication rubric scores, describe and categorize SP comments of student empathy, and test the relationship between quantitative rubric scores and empathy categorization using an integrative analysis of transformed qualitative comments.

METHODS

This was a retrospective evaluation of a graded performance-based assessment (PBA) in which SPs had been used with a convergent mixed methods research design at one college of pharmacy. The rationale for using mixed methods in this study was to expand and transform qualitative findings to understand how SP empathy perceptions influenced student communication scores and to explore the varying degrees to which students engaged in empathetic communication. A visual depiction of the mixed methods study design is presented in Figure 1. The study was conducted in accordance with institutional IRB policies and was granted an exemption as non-human subject research.

Figure 1.

Mixed Methods Study Design

The PBA scenario focused on a patient with several medications for hypertension and diabetes who was struggling with medication adherence. Students were tasked with obtaining information related to the medication regimen and difficulties with medications or disease state management. The primary objective of the PBA was to identify patient medication concerns and address medication non-adherence. Standardized patients were asked to evaluate students’ clinical and communication skills using a rubric developed and finalized by experts at six schools and colleges of pharmacy (Appendix 1).³³ Details on the specific development processes can be found elsewhere.³³ Efforts to further explore the validity and reliability of the rubric are ongoing. The SPs provided item scores and written feedback using communication and clinical rubrics and oral feedback immediately following the completion of the PBA encounter. The SPs were not notified that written feedback would be used to assess student empathy, minimizing the potential for bias. Standardized patients are trained to assess clinical and communication scores at the university’s college of medicine. The PBA communication rubrics used by SPs to assess two cohorts of third-year student pharmacists (218 students) were selected to use in this evaluation because these individuals were entering their advanced pharmacy practice experiences (APPEs) and this was their last PBA in which SPs were used. The two cohorts were similar, with all students completing comparable didactic and experiential training prior to the final PBA. Student pharmacist participants were deidentified by one of the course instructors before the analysis. Additionally, the clinical scenario and rubric used for assessment were identical for both cohorts. While all SPs completed assessment rubrics, only rubrics on which supplementary comments had been written were included in this analysis.

Appendix 1.

Communication Assessment Rubric used during the performance based assessment (PBA).³³*

For the quantitative data, total student communication scores were calculated from scores for 20 communication items across six domains: initiating the encounter (three items), establishing trust and respect (four items), using verbal communication skills (five items), using non-verbal communication skills (two items), managing the encounter (three items), and concluding the encounter (three items). While the original rubric was out of 20 points, items were transformed into whole numbers while maintaining comparable differences across all items to facilitate analysis. As such, a response of no was assigned a zero point value, inconsistent was assigned a one-point value (increased from one-half point value), and yes was assigned a two-point value (increased from a one-point value). Non-applicable (N/A) items were also assigned a two-point value. As a result, the maximum obtainable score on the communication assessment was 40 points. One item on the communication assessment rubric evaluated student empathy during the encounter; however, SPs were instructed to select N/A for student empathy as the encounter was not specifically focused on assessing empathy, and SPs have historically struggled to interpret and score the empathy item. Descriptive statistics and frequencies were used to evaluate communication rubric scores for all students. For the qualitative analysis, open-ended comments left by SPs providing communication feedback were uploaded into the data management software MAXQDA 2020 (VERBI Software). Two of the authors, a graduate student and faculty member who were not involved with the development or execution of the PBAs, independently performed a basic interpretive analysis on the written feedback provided by the SPs. These two study authors independently coded the text segments for all rubrics included in the analysis and met to discuss initial codes and specific terminology associated with SP comments related to student pharmacist empathy.

To integrate the two datasets, a second round of deductive coding was performed by the same study authors involved in the initial coding. The deductive coding was applied to the entirety of the SP open-ended comments from each student to categorize each student into one of three empathy categories (low empathy, mixed empathy, and high empathy). For each student, the empathy category was applied based on the prevalence and general affect of empathy-related terminology in the open-ended comments left by the SP. Student rubrics were categorized as “low empathy” when the majority of the comments reflected encounters where the SP reported the student missed opportunities to address their needs and/or communicated with them unempathetically (eg, “I was interrupted…when you asked about how I was doing with diet and exercise, I said I walked 3x week, and diet is going ok, but I am not interested in doing more about diet at this point. Even so, you said I should keep working on the diet.”). Student rubrics were categorized as “high empathy” when students received only positive feedback related to their ability to exhibit empathetic communication skills throughout the encounter (eg, “Student displayed strong concern for my well-being…I felt very engaged, no judging…”). Student rubrics were categorized as “mixed empathy” when SPs gave students both positive and negative feedback related to empathetic communication (eg, “…we spoke about different ways to elicit concerns from a patient and I am sure they will be able to do so in the future”) or when empathy-related terminology was not explicitly or implicitly mentioned in the open-ended comments (eg, “…not rushed, I think there is room to slow down a bit. It will help give you time to think.”) A subset of empathy categories and representative quotes were checked for accuracy and appropriate empathy categorization by three clinical pharmacy educators involved in PBA development. The clinical pharmacy educators were presented with a sample of uncoded text provided by SPs and were asked to independently code the text into the empathy categories, ensuring that the initial coding was an accurate assessment of student empathy category from the perspective of those immediately involved in clinical skill education and evaluation.

To test the relationship between rubric scores and empathy categorization for the integrated analysis, mean rank communication rubric scores were compared using Kruskal-Wallis ANOVA in SPSS Statistics for Windows, Version 25 (IBM). Post-hoc analysis was performed with Dunn’s pairwise tests performed on all three pairs of empathy categorization to identify significant differences in communication scores between empathy categories.

RESULTS

One hundred forty-one of 218 rubrics (64.7%) were included for analysis. Written, open-ended feedback was not available for 77 rubrics, preventing the ability to assign empathy categories. From the 141 rubrics, the average communication score for student pharmacists was 39.0±1.6, with a low score of 31 and a high score of 40.

The SPs provided a range of feedback on student communication. The SPs frequently commented on the student pharmacist’s nonverbal communication ability and what emotions or feelings the SP had during the encounter. Nearly all students received at least one positive comment from the SP and most students were given feedback on areas for improvement. Most of the feedback focused on students missing cues or opportunities to connect and communicate with the SP after the patient expressed a concern or need. Similarly, some students received feedback that their proposed clinical plan failed to consider the patient’s needs and preferences. In the following examples of SP comments, specific words reflecting negative experiences are underlined for emphasis and the associated empathy code is included in brackets following the statement. Example: “You did not ask why I came in today, instead you started with your agenda. I was interrupted…I am not interested in doing more about diet at this point. Even so, you said I should keep working on the diet.” [Low Empathy Comment]

The majority of SP comments indicated that students exhibited both high and low empathy during the encounters. Common deficiencies included SPs feeling as though students were rushing their encounter, speaking over them, and/or struggled to elicit their concerns. The SPs sometimes noted these deficiencies only occasionally happened within each encounter, with most reporting that they had some confidence that the students could correct these communication errors moving forward. Example: “There were a couple of times they spoke over me when explaining my medications…” [Mixed Empathy Comment]

Finally, some SP comments suggested they experienced exclusively positive encounters with their assigned student pharmacist. Positive SP feedback was often associated with encounters where the SP felt heard and supported, and did not feel judged. High-empathy comments frequently resulted from student pharmacists addressing patient concerns and helping the patient develop a plan to address them. Example: “[They] have a quiet confidence and openness about [them] that made me feel like I could trust [them] and talk openly about my prescriptions and lifestyle changes.” [High Empathy Comment]

Based on the SP comments included on 141 rubrics, 6 (4.3%) were categorized as low-empathy encounters, 95 (67.4%) were categorized as mixed, and 40 (28.4%) were categorized as exhibiting high-empathy during the encounter. Representative quotes for each of the three empathy categories are included in Appendix 2.

Appendix 2.

Empathy Categorization and Representative Quotes from Communication Rubrics

The results from the Kruskal-Wallis test showed a significant difference in communication score between at least one pair of empathy categories, (Kruskal-Wallis H = 24.62 df =2, p <.05) with a mean rank communication score of 12.1 for the low empathy category, 67.3 for the mixed empathy category, and 88.8 for the high empathy category. The Dunn pairwise test identified significant differences in communication scores between low empathy and high empathy categories, low empathy and mixed empathy categories, and high empathy and mixed empathy categories (p<.05, adjusted using Bonferroni correction). Descriptive statistics for empathy category and communication scores can be found in Table 2.

DISCUSSION

Overall, standardized patients gave the student pharmacists represented in this retrospective analysis high scores regarding their demonstration of empathy during a PBA. While the SP’s rubric scores suggested students used most of the graded communication elements during their encounter, the SP’s open-ended feedback tended to focus on the humanistic aspects of communication that were not formally assessed by the rubric. The integrated analysis we performed suggests there was a positive association between high student empathy and high scores on the communication rubric. Studies of actual patients supported the emphasis the SPs placed on empathy and its importance to a successful patient encounter. Higher levels of provider empathy have been associated with increases in standardized measures of patient experience and satisfaction with health services.^34,35 Outside of health care, higher levels of employee empathy indirectly contribute to perceptions of service quality, a benefit that may have implications pertaining to patient satisfaction, service experience, and health outcomes.^16-19

This study also highlights the potential for PBAs using SPs to evaluate communication skills, such as empathy, that can be difficult to capture with traditional checklist style evaluation rubrics. While SPs were instructed not to use the single-item empathy assessment on the communication rubric, SPs frequently provided written comments about their observations and feelings related to student empathy or lack thereof. Ultimately, SPs having a post-encounter discussion with the student pharmacist about how the SP felt during the encounter may be more beneficial than having SPs complete a single-item assessment of empathy.^36,37

The interpretive analysis of SP comments showed variation in students’ ability to engage in empathetic communication and clinical empathy. Employing aspects of clinical empathy during high stakes standardized patient encounters appears to be a high-level skill which students may only partially deliver and may need more opportunities to master. For example, based on written feedback, students appeared to use empathy more often to understand the patient’s perspective but less often to guide a therapeutic plan. Encounters for which the student pharmacist received low empathy scores lacked essential elements of clinical empathy, especially responding to and making use of cues from the SP when making recommendations.

While we did not include student demographic data in our analysis, other studies have reported that student pharmacists exhibit different levels of empathy based on several baseline characteristics, such as gender, altruism, age, and prior health care experiences.³ Further, professional students may exhibit low levels of empathy because of the stressful nature of assessments, a fear of making mistakes, and the difficulty of the clinical scenarios.^38,39 Thus, there is a need for future research to explore factors that may be contributing to student pharmacists’ difficulty in developing clinical empathy and demonstrating their mastery within standardized patient encounters.

Given the importance of student pharmacists demonstrating clinical empathy in patient encounters and the multitude of barriers that prevent them from doing so, schools and colleges of pharmacy should consider trying the following: helping students understand how clinical empathy relates to the process of providing patient care, encouraging students to use clinical empathy within skills assessments, and assessing students more consistently on clinical empathy abilities. First, to increase students’ use of empathetic communication and clinical empathy in high-stakes assessments, it may be important to use clinical empathy concepts to inform educational interventions and assessments within didactic coursework to help students make meaningful ties between clinical empathy and the Pharmacists’ Patient Care Process (PPCP).⁴⁰ This process, proposed by the Joint Commission of Pharmacy Practitioners, is a wholistic model of pharmaceutical care with a focus on patient-centricity. The process focuses on five steps to patient care: collect, assess, plan, implement, and follow-up. To help students value, develop, and employ clinical empathy skills, educational interventions should emphasize the importance of empathy throughout this process, using clinical empathy to guide patient-centered collection of data and assessment, and develop the clinical plan.^41,42

Second, providing students with more low-stakes opportunities to demonstrate and develop clinical empathy skills; putting increased emphasis on the importance of clinical empathy within PBAs. Doing so may encourage student pharmacists to use clinical empathy when engaging patients and making clinical recommendations that reflect a patient-centered approach. Making the demonstration of clinical empathy a required element of PBAs may encourage students to look for opportunities to use clinical empathy when it is their turn to be assessed.

Finally, to ensure that assessment results are an accurate representation of a student pharmacist’s ability to show clinical empathy, formal rubrics and SP training should emphasize all three components of clinical empathy: listening, communicating, and developing a patient-centered clinical plan. Open-ended SP comments could be used alongside the ratings to provide the student with authentic feedback.

This study has several limitations. This retrospective analysis used data gathered on two cohorts of third-year student pharmacists at a single school of pharmacy, which limits the generalizability of our findings and the ability to assess empathy development across student cohorts and institutions. These results may only apply to schools and colleges of pharmacy who currently use communication assessments similar to this study. Additionally, the authors only included those PBA rubrics with written feedback provided by SPs. Some communication rubrics did not have comments written on them, potentially altering the analysis and our findings. Comments made by SPs during those encounters may have been written on the back of the rubric but lost during the archiving process or feedback may have only been communicated verbally to students. Further, the clinical scenario and/or SP characteristics may have influenced SP assessment of communication and empathy. While rank ordering yielded a significant result, many of the scores were similar and did not exhibit normal distribution, suggesting a potential ceiling effect.

There are multiple opportunities for future studies to expand assessment of student empathy ability within PBAs. Future studies should explore how the evaluator rubric used for student pharmacist communication assessment compares to existing patient-provider communication frameworks and tools, such as The Four Habits Model and Patient-Centered Communication Tools (PaCT).^43,44 Given that ordinal data were unavailable for empathy assessment, future studies should explore how single-item empathy scores compare to qualitative responses. Instead of or in addition to SP comments, observational methods that use videotaping and validated communication coding schemes should be considered when planning future studies. While not modeled in the present study, an important area for future work is to consider how implicit biases related to ethnicity, gender, or other characteristics may affect SPs’ evaluations of students as these factors have been shown to influence learner self-evaluations and patient evaluations of empathy^30,45 More work is needed to evaluate student pharmacists’ clinical empathy skills longitudinally, specifically how these skills are affected by ongoing clinical education. and the effect of empathy on patient experience with pharmacy services in a real-life healthcare setting. Additional work should focus on differences in empathy based on SP and student pharmacist pairing, as well as considering additional empathy-related interventions to help students develop and employ clinical empathy in patient encounters. This may be done using mindfulness-based interventions,⁴⁶ professional development courses,³ and additional simulated clinical experiences.^22-25 Given the increased focus on the importance and value of pharmacists showing clinical empathy during pharmacy consultations,¹⁰ additional research exploring how clinical empathy can be taught and measured within high-stakes assessments in pharmacy school is needed.^47,48

CONCLUSION

Open-ended comments reflecting a student’s ability to demonstrate clinical empathy were positively associated with higher overall scores on a PBA communication rubric. The majority of SP comments indicated that students struggled to consistently exhibit empathetic communication throughout the simulated encounter. Schools and colleges of pharmacy should reconsider which methods should be used to teach and assess clinical empathy skills, and which will encourage students to employ these skills during PBAs and after they enter practice. With empathetic communication and clinical empathy integral to the patient encounter, exploring alternative tools for education and assessment may be useful in helping students acquire these skills.

Table 1.

Minimum, Maximum, and Median Communication Scores Across Rubrics Categorized by Empathy