Skip to main content
NCBI home page
The British Journal of Occupational Therapy logoLink to The British Journal of Occupational Therapy
. 2025 Mar 24;88(7):420–428. doi: 10.1177/03080226251314367

Intra-rater and inter-rater reliability of the Empathic Communication Coding System

Yu-Jeng Ju 1, Meng-Lin Lee 2, Jou-Chieh Chen 1, Yi-Ching Wang 1, Chih-Ping Chen 3, Han-En Tsai 4, Tzu-Yi Wu 3,*,✉, Ching-Lin Hsieh 1,3,5,*
PMCID: PMC12207621  PMID: 40599214

Abstract

Objects:

The Empathic Communication Coding System (ECCS) is a promising measure of such behaviors, yet its reliability remains largely unestablished. The aim of the study was to examine the intra-rater and inter-rater reliability of the ECCS.

Methods:

Transcripts of occupational therapy audio recordings were used to examine the reliabilities. Two occupational therapists were trained to evaluate the data for inter-rater reliability, and one of them evaluated the data twice for intra-rater reliability. Transcripts of 20 patients treated by 18 therapists were evaluated for intra-rater reliability. Transcripts of 25 patients treated by 22 occupational therapy interns were evaluated for inter-rater reliability. Adult rehabilitated patients with basic verbal communication ability were recruited. The percentage of agreement and Cohen’s κ were calculated as the indices of reliability.

Results:

The results showed acceptable intra-rater reliability but poor inter-rater reliability of the ECCS.

Conclusions:

The ECCS demonstrates potential as a reliable tool for assessing empathic behaviors over time when evaluated by the same raters.

Keywords: Assessment, empathy, Empathic Communication Coding System, occupational therapy, reliability

Introduction

Empathy is recognized as a core competence for medical professionals, including occupational therapists (Serrada-Tejeda et al., 2022). By understanding the emotions of patients through skills of empathy, occupational therapists can better provide emotional support, design appropriate therapeutic activities, and help patients participate in life as fully as possible. Recent findings underscore empathy’s significant impact on patient health (Dibbelt et al., 2009; Mercer et al., 2016; Moudatsou et al., 2020), therapeutic relationships (Moudatsou et al., 2020; Silverman, 2009), and the mental well-being of healthcare providers (Brazeau et al., 2010; Dyrbye et al., 2010). In addition, due to the time pressure faced in clinical work (e.g., productivity demands, documentation, and limited treatment sessions per patient), occupational therapists must engage in empathic interactions with patients more efficiently. As such, contemporary research highlights the necessity of integrating empathy training as a foundational element of occupational therapy education (Kelly, 2022; Serrada-Tejeda et al., 2022). Furthermore, according to the Intentional Relationship Model (Taylor et al., 2009), empathy is one of the six modes of the therapeutic use of self for occupational therapists. Therefore, empathy is essential in occupational therapy practice.

Selecting an appropriate measure of empathy is critical for its education and evaluation, yet doing so remains a challenging task. Empathy is a multidimensional concept including cognitive, affective, and behavioral dimensions (Cuff et al., 2016; Hojat, 2016; Sulzer et al., 2016). Among them, behavior is considered one relatively important dimension of empathy in clinical settings, as empathic interaction with patients could affect the outcome of health care (Hall et al., 2021; Halpern, 2003). However, few measures assess observable empathic behaviors (Bylund and Makoul, 2002; Humphris and Kaney, 2001; Mercer et al., 2004). Most commonly used measures do not focus on behavioral empathy. For example, widely used self-report measures such as the Jefferson Scale of Empathy (Fields et al., 2011; Hojat et al., 2001, 2002; Kane et al., 2007), the Interpersonal Reactivity Index (Davis, 1980), and the Questionnaire Measure of Emotional Empathy (Mehrabian and Epstein, 1972) are designed to primarily assess cognitive or affective dimensions of empathy in self-report form. Similarly, the Reading the Mind in the Eyes Test, recognized for evaluating emotion recognition abilities (Baron-Cohen et al., 1997, 2001), does not address behavioral empathy.

The Empathic Communication Coding System (ECCS) (Bylund and Makoul, 2002, 2005) was developed to evaluate empathic behaviors, and it has been used on various health professionals, including general internists (Bylund and Makoul, 2002, 2005), medical students (Foster et al., 2014, 2016), anesthesia residents (Waisel et al., 2020), dietitians (Goodchild et al., 2005; Parkin et al., 2014), hospice teams (Wittenberg-Lyles et al., 2012), and dignity therapists (Bylund et al., 2022). The ECCS is a third-person-rated coding system for quantifying health professionals’ responses to patients with seven empathic levels (Bylund and Makoul, 2005). The evaluation procedure of the ECCS comprises two steps: empathic opportunity identification and empathic behavior rating. The former involves identifying conversational opportunities where therapists are expected to express empathy, while the latter rates the levels of empathic behaviors demonstrated by clinicians in response to these opportunities.

The intra-rater reliability and inter-rater reliability of the ECCS are key psychometric properties, as the dependability of empathic behavior evaluations is heavily influenced by the consistency and agreement among raters. Poor intra-rater reliability would result in inconsistent evaluations from the same rater over time, undermining the consistency of the assessment process. Similarly, poor inter-rater reliability would lead to substantial discrepancies in evaluations between different raters, diminishing the validity of the measure and potentially impacting decision-making in clinical practice and research. However, to our best knowledge, no studies have focused on examining the intra- and inter-rater reliability of the ECCS. Therefore, there is an obvious need to verify the reliability of the ECCS. This study thus aimed to examine the intra- and inter-rater reliability of the ECCS.

Methods

Data

We extracted audio recordings of occupational therapy sessions from a database of an ongoing occupational therapist intern training program (Huang, 2022). Informed consent was obtained from the patients, interns, and clinical supervisors and the consent was written. The first two sessions of each patient’s treatment were audio recorded and extracted. Patients were included if they met the following criteria: (1) basic cognitive ability (passed two items of the language dimension in the mini-mental state examination (Folstein et al., 1975), (2) basic verbal communication ability (could briefly describe his/her daily activities), (3) over 20 years old, and (4) new patients of the intern. Each session was about 30 minutes. Thus, each patient would have a 1-hour long audio recording. The recordings were then transcribed for analysis. This study was approved by the Institutional Review Board of a medical center (the approval number: 202006114RINC; the year of approval: 2020).

The transcripts were further screened for analysis with the following exclusion criteria: (1) The patient did not speak Mandarin primarily and (2) treatments were taken over by supervisors and not conducted by interns. After applying these criteria, transcripts of 35 patients treated by 30 intern therapists remained and were included in the analysis. To examine the intra-rater reliability, a random selection of data from 20 patients was designated as dataset A for one rater (the sixth author). To examine the inter-rater reliability, a random selection of data from 25 patients was designated as dataset B for two raters (the sixth author and third author). Ten patients were in both datasets A and B.

Empathic Communication Coding System

The evaluation procedure of the ECCS consists of two steps, empathic opportunity identification and empathic behavior rating (Bylund and Makoul, 2002, 2005). An empathic opportunity is a unit of conversation for analysis with three possible categories: emotion, progress, or challenge. Specifically, an empathic opportunity is initiated by patients describing their current emotions (i.e., emotion), positive developments in their physical condition that have improved their quality of life (i.e., progress), or negative effects of physical or psychosocial problems that have impacted their quality of life (i.e., challenge). For example, a statement in which a patient describes feeling desperate after experiencing an incidental stroke would be identified as an empathic opportunity.

For each empathic opportunity, health professionals’ responses are rated on seven levels (0: Denial/disconfirmation, 1: Perfunctory recognition, 2: Implicit recognition, 3: Acknowledgment, 4: Pursuit, 5: Confirmation, 6: Shared feeling or experience) (Bylund and Makoul, 2005). For example, not responding to the patient’s expression of desperation and moving on to the next clinical evaluation would be rated as level 0. Behaviors at higher levels indicate more empathy (Bylund and Makoul, 2002, 2005).

In addition, two conditions were coded as missing values for responses to empathic opportunities. First, the patient did not allow time for a response (Bylund and Makoul, 2005). Second, the supervisor interrupted the conversation and did not allow time for the intern to respond.

Procedure

Training of raters

Two occupational therapists (the sixth author and the third author), each with less than 1 year of clinical service experience, underwent a 3-month training period to evaluate therapists’ empathic communication responses utilizing the ECCS. We used 22 transcripts retrieved from another database (Hsieh, 2017) as the training dataset. The training dataset was collected from 2018 to 2019 and consisted of occupational therapy sessions for patients with physical disabilities conducted by registered occupational therapists. All patients were diagnosed with stroke, had basic communication and cognitive abilities, and were new patients to the occupational therapists.

During the training, the raters first read and discussed the ECCS rating rules (Bylund and Makoul, 2002, 2005) to gain a preliminary understanding of empathic opportunity identification and empathic behavior rating. Then they independently identified empathic opportunities and rated empathic behaviors using qualitative analysis software (ATLAS.ti 9) for the training dataset.

In the first month, after each transcript was rated by the two raters, a group meeting involving two raters and one author (the eighth author) was held to discuss the results to reach a consensus. In the second and third months, weekly group meetings were held to discuss two to three transcripts until a consensus was reached. Practical examples from the discussions were added to the coding book after each discussion. The flowchart of the rater training process can be found in the Appendix Figure 1.

Empathic opportunity identification and empathic behavior rating

Intra-rater rating

One rater (the sixth author) conducted empathic opportunity identification and empathic behavior rating with dataset A twice, 8 weeks apart. When the rater completed the first round of rating, she sent all the transcripts, results, and other related materials to the primary author. She did not access the transcripts again until the second round of rating.

Inter-rater rating

The two raters (the sixth author and the third author) conducted empathic opportunity identification and empathic behavior rating with dataset B independently. The raters did not contact each other during the rating periods.

Data analysis

The percentage of agreement and Cohen’s κ were used to measure intra- and inter-rater reliability in identifying empathic opportunities, with the former being easy to calculate but biased by guessing, and the latter controlling for guessing effects (Banerjee et al., 1999). The percentage of agreement is a simple way to measure how often two raters agree in their assessments. However, it does not account for agreements that could happen by chance. By considering the possibility of chance agreement, Cohen’s κ is a more advanced measure that adjusts the percentage of agreement. In our study, the percentage of agreement was calculated as the number of the same-type empathic opportunities divided by the total number of empathic opportunities.

Also, both percentages of agreement and weighted κ were used to measure intra- and inter-rater reliability in rating empathic behaviors. Weighted κ is similar to Cohen’s κ but is used when the categories have an order (like “disagree,” “agree,” “strongly agree”). It assigns different weights to disagreements, giving partial credit if the ratings are close rather than completely different (Banerjee et al., 1999). To calculate the percentage of agreement and weighted κ, the ratings of empathic behaviors of the same empathic opportunities from time 1 and time 2 (for intra-rater) or from different raters (for inter-rater) were used. Thus, only the empathic behaviors of the empathic opportunities identified at both time 1 and time 2, or by both raters, were used. Missing values of responses were excluded from this analysis. The percentage of agreement was calculated as the number of behaviors rated at the same level at both time points (for intra-rater) or by both raters (for inter-rater) divided by the total number of empathic behaviors included.

A percentage of agreement >80% was considered acceptable (Mokkink et al., 2010). Cohen’s κ and weighted κ > 0.4 were considered acceptable, and values >0.75 were excellent (Banerjee et al., 1999). κ values were conducted with the psych package (Revelle, 2022) in R (R Core Team, 2023). The weight matrix for weighted κ was quadratic.

Results

Demographic characteristics

Dataset A (for intra-rater reliability)

Dataset A included 20 transcripts and audio recordings from 20 patients. The patients were 56.9 years old on average (SD = 14.7). Most patients were diagnosed with stroke or spinal cord injury (45% and 45%, respectively, Table 1).

Table 1.

The patients’ characteristics.

Variable Dataset A (n = 20) Dataset B (n = 25)
Sex
 Male 12 (60%) 14 (56%)
 Female 8 (40%) 11 (44%)
Age
 Mean (SD) 57 (15) 63 (14)
 [Min, max] [19, 75] [19, 79]
Diagnosis
 Stroke 9 (45%) 9 (36%)
 SCI 9 (45% 12 (48%)
 Other 2 (10%) 4 (16%)
Open in a new tab

SD: standard deviation; SCI: spinal cord injury.

The patients were treated by 18 interns aged 21.9 years on average (SD = 0.7, ranging from 21 to 23). Seven of the interns were male. Two interns contributed two transcripts each, as they treated two patients.

Dataset B (for inter-rater reliability)

Dataset B included data from 25 patients. The patients were aged 63.0 years on average (SD = 14.2), and most were diagnosed with stroke or spinal cord injury (36% and 48%, respectively, Table 1).

The patients were treated by 22 interns aged 21.7 years on average (SD = 0.8). Of the interns, 3 were males and 19 were females. Separately, three interns had two transcripts because they treated two patients.

Intra-rater reliability

Empathic opportunity identification

A total of 77 empathic opportunities were identified from the transcripts across the 2 time points. At time 1, the rater (the sixth author) identified 62 empathic opportunities among 16 patients; 4 patients had no empathic opportunities identified. At time 2, the rater (the sixth author) identified 66 empathic opportunities among 16 patients; the same 4 patients had no empathic opportunities identified. As listed in Table 2, most empathic opportunities were initiated by statements of challenge at time 1 (n = 34) and time 2 (n = 31).

Table 2.

The types of empathic opportunities identified by the rater at time 1 and time 2.

Categories of empathic opportunities Time 2 Total
Emotion Progress Challenge Not identified
Time 1 Emotion 20 0 0 1 21
Progress 0 5 0 2 7
Challenge 0 0 26 8 34
Not identified 8 2 5 0 15
Total 28 7 31 11 77
Open in a new tab

Numbers in cells indicate the number of opportunities.

Fifty-one empathic opportunities were consensually identified at both time 1 and time 2, and the percentage of agreement was moderately acceptable (51/77, 66%). The Cohen’s κ was 0.51 (95% CI [0.37, 0.65]).

Empathic behavior rating

As listed in Table 3, most responses were rated as level 3 or level 4 at time 1 and time 2. After the removal of 1 missing value, 50 empathic behaviors were qualified for the calculation of reliability indices. The percentage of agreement was 84% (42/50) and weighted κ was 0.74 (95% CI [0.56, 0.93]).

Table 3.

The results of empathic behavior rating for intra-rater analysis.

Response level Intra-rater
Time 1 Time 2
6 0 (0%) 0 (0%)
5 12 (19%) 13 (20%)
4 25 (40%) 27 (41%)
3 21 (34%) 23 (35%)
2 2 (3%) 2 (3%)
1 0 (0%) 0 (0%)
0 1 (2%) 0 (0%)
Missing 1 (2%) 1 (2%)
Open in a new tab

Missing values were coded in two conditions. First, the patient did not allow time for a response. Second, the supervisor interrupted the conversation and did not allow time for the intern to respond.

To investigate potential rating differences in behaviors responding to different types of empathic opportunities, we further calculated the intra-rater reliability indices of empathic behavior rating on emotion and challenge empathic opportunities, respectively. The behaviors for progress empathic opportunities numbered only 5 and were not further analyzed. For emotion-empathic opportunities, the percentage of agreement for empathic behavior ratings was 74% and the weighted κ was 0.57 (95% CI [0.25, 0.89]). For challenge empathic opportunities, the percentage of agreement for empathic behavior ratings was 89% and the weighted κ was 0.86 (95% CI [0.70, 1.00]).

Inter-rater reliability

Empathic opportunity identification

Across 25 patients’ data, 2 independent raters identified 97 empathic opportunities, with an average of 3.9 empathic opportunities per patient (SD = 3.2). No empathic opportunities were identified in the two patients’ transcripts. A maximum of 12 empathic opportunities were identified in the data of a single patient. Most patients’ data (n = 6, 24%) contained three empathic opportunities. As listed in Table 4, most empathic opportunities were initiated by statements of emotion for rater 1 (n = 41) and challenge for rater 2 (n = 26).

Table 4.

The types of empathic opportunities identified by raters 1 and 2.

Rater 2 Total
Emotion Progress Challenge Not identified
Rater 1 Emotion 14 0 0 27 41
Progress 0 1 0 6 7
Challenge 0 0 15 18 33
Not identified 1 4 11 0 16
Total 15 5 26 51 97
Open in a new tab

Numbers in cells indicate the number of opportunities.

Thirty empathic opportunities were consensually identified by both raters and the percentage of agreement was 31% (30/97). The Cohen’s κ was 0.08 (95% CI [−0.04, 0.20]). The other 67 empathic opportunities were identified by only one of the raters.

Empathic behavior rating

As listed in Table 5, most responses were rated as level 3 or level 4 by both raters. After the removal of 1 missing value, 29 behaviors were qualified for the calculation of reliability indices. The percentage of agreement was 45% (13/29) and weighted κ was 0.29 (95% CI [0.02, 0.57]).

Table 5.

The results of empathic behavior rating of inter-rater analysis.

Response level Inter-rater
Rater 1 Rater 2
6 1 (1%) 0 (0%)
5 8 (10%) 1 (2%)
4 26 (32%) 13 (29%)
3 28 (35%) 24 (53%)
2 5 (6%) 3 (7%)
1 0 (0%) 0 (0%)
0 5 (6%) 1 (2%)
Missing 8 (10%) 3 (7%)
Open in a new tab

Missing values were coded in two conditions. First, the patient did not allow time for a response. Second, the supervisor interrupted the conversation and did not allow time for the intern to respond.

Furthermore, for emotion-empathic opportunities, the percentage of agreement for empathic behavior ratings was 36% and the weighted κ was 0.27 (95% CI [0.03, 0.51]). For challenge empathic opportunities, the percentage of agreement for empathic behavior ratings was 57% and the weighted κ was 0.35 (95% CI [−0.21, 0.91]). Only one opportunity was related to progress and was not analyzed.

Discussion

We found acceptable intra-rater reliabilities for both empathic opportunity identification and empathic behavior rating. The findings indicate that trained raters can use the same standard to identify empathic opportunities and rate empathic behaviors on seven levels consistently, within an 8-week interval. In other words, when a rater rates the same transcript twice, the results are very likely to be similar, and not easily influenced by random errors such as timing, location, or transcript characteristics. Because this interval typically exceeds the duration of most treatments or empathy training programs, the ECCS thus could be considered a reliable tool for assessing changes in empathic behaviors over time evaluated by the same raters for both clinical practice and research.

Poor inter-rater reliability was found for the empathic opportunity identification. This indicates that different raters using the ECCS do not apply scoring standards similarly. As a result, for the same conversation, one rater may consider it an empathic opportunity, while another rater may not, and vice versa. A potential explanation is that the definition of empathic opportunity may not be appropriate or clear in occupational therapy scenarios. The ECCS defines an empathic opportunity as an explicit statement of patients’ emotions, progress, or challenges related to their quality of life (Bylund and Makoul, 2002, 2005). However, improving the quality of life is one of the main targets of therapist–patient communication in occupational therapy. If no additional criteria are added to differentiate the descriptions of a patient’s state of illness from a patient’s statement of empathic opportunities, most conversations across treatment sessions could be considered empathic opportunities. For example, if a therapist asks whether a patient can independently climb stairs, a patient may answer that he/she feels challenged due to the slower pace compared to before. This answer could be interpreted as either an empathic opportunity or merely a description of the patient’s current condition. Such ambiguity may lead to inconsistent identification of empathic opportunities between raters due to different subjective interpretations. To address this issue, a potential solution would be to revise the definition of empathic opportunities for occupational therapy scenarios, provide more specific and context-relevant examples, and clarify ambiguous situations.

We observed poor inter-rater reliability for the empathic behavior rating, potentially due to two complicating factors arising from the unclear guidelines, which impacted raters’ interpretations and scoring consistency. First, the transcripts only provided verbal communication content, omitting non-verbal cues. Second, the occupational therapy interns in our data might not have clearly expressed their intentions. For example, when a patient complains about the difficulties encountered when taking a shower, the therapist responds with a further inquiry regarding how or what the patient did to take a shower. This response can be rated as either level 4 (Pursuit) or level 2 (Perfunctory recognition) because this could also be considered an automatic, script-type response. Without viewing the facial expressions of the therapist or other non-verbal information, the raters had to rely on their subjective interpretations. To improve reliability, we suggest establishing more explicit guidelines for rating ambiguous behaviors and providing illustrative examples to differentiate between different levels and to address potential areas of confusion.

In addition to the previous three paragraphs discussing the main results of intra-rater and inter-rater reliability, we have synthesized the findings of this study and identified three main phenomena when using the ECCS in occupational therapy scenarios. First, we observed that the reliability of behavior ratings was generally higher for challenges than for emotions in both intra- and inter-rater reliability. This finding suggests that health professionals may respond differently to various types of empathic opportunities, which, in turn, affects the consistency of ratings. Specifically, health professionals might provide more nonverbal responses to emotions than to challenges, and these nonverbal responses were not captured in the transcript data. Consequently, although both emotions and challenges are common types of empathic opportunities, the reliability of ratings for behaviors in response to emotions was lower than that for challenges. To address this, training programs for raters should consider placing greater emphasis on emotion-type scenarios to improve rating reliability.

Second, our findings revealed that using the ECCS to assess therapists’ empathic behaviors may be influenced by rater subjectivity, leading to poor inter-rater reliability in identifying empathic opportunities and rating behaviors. This subjectivity may be influenced by various factors, including gender and cultural differences (Serrada-Tejeda et al., 2022). For instance, social expectations often encourage women to be more empathic and emotionally supportive, potentially making female raters more sensitive to emotional cues (Löffler and Greitemeyer, 2023; Shin et al., 2022). In addition, cultural backgrounds can influence how raters interpret the dialogue between therapists and patients, leading to diverse empathy assessments (Elfenbein and Ambady, 2002). These cultural and gender differences complicate the process of assessing empathy and highlight the difficulty of using a single, universal quantitative tool to measure empathy. Thus, intra-rater reliability was acceptable, relying on a single rater risks subjective bias. To improve consistency, incorporating qualitative elements in empathic measures may improve the accuracy and consistency of assessments.

Third, the unclear definitions and rating guidelines of the ECCS in occupational therapy scenarios might lead to varying subjective interpretations and consequently poor inter-rater reliability. To address this, we recommend modifications tailored to occupational therapy. For example, in the step of empathic opportunity identification, adding specific cues could enhance clarity beyond conceptual rules; in the step of behavior rating, more explicit guidelines and illustrative examples could help address areas of potential confusion. Previous studies have modified the ECCS in other clinical settings (Foster et al., 2014, 2016; Waisel et al., 2020; Wittenberg-Lyles et al., 2012), indicating the feasibility of such adjustments for occupational therapy. However, such adaptations will require validation in occupational therapy settings.

Before the ECCS is modified, we recommend assessing empathic behaviors using multiple raters to minimize subjective bias. By averaging the scores across all raters, this approach could enhance reliability. Nonetheless, multiple-rater evaluation may cause significant costs, and the optimal number of raters for reliable results remains undetermined. Further research on this matter is necessary. Besides, a future study is also recommended to investigate the correlation between ECCS and treatment outcomes to improve the utility of ECCS.

Limitations

There are several limitations in this study. First, small sample sizes of only 20 and 25 data points were examined for intra- and inter-rater reliabilities. Second, data were collected only from persons with physical disabilities receiving interns’ occupational therapy services. The generalizability of our results to other patient cohorts or registered therapists may be limited. Third, our data were collected from the first two sessions of treatment, in which very few empathic opportunities for progress were identified. Because our results indicate that the reliability of behavior rating might be different for different types of empathic opportunities, our results might not be generalizable to later treatment sessions due to possible changes in the proportions of empathic opportunity types. Fourth, our study only included occupational therapy students. Therefore, the results may not be generalizable to therapists from different fields or those with greater levels of experience. Finally, for both intra- and inter-rater reliability, the reliability of empathic opportunity identification limits the reliability of empathic behavior rating. If the reliability of empathic opportunity identification is low, it could have further limited the number of consensual opportunities available for calculating the reliability of behavior rating.

Conclusion

We found acceptable intra-rater reliability of the ECCS, supporting it as a reliable tool for assessing changes in empathic behaviors over time when the evaluations are administered by the same raters. However, we also found poor inter-rater reliability of the ECCS. Modifying the ECCS for occupational therapy scenarios and averaging the results from multiple raters are proposed as two separate approaches to improve inter-rater reliability.

Empathic behavior is a crucial ability for occupational therapists. The ECCS has poor inter-rater reliability and acceptable intra-rater reliability. Unclear ECCS definitions in occupational therapy warrant modifications, such as adding cues for empathic opportunity identification and providing explicit behavior rating guidelines. Assessing empathy with multiple raters to mitigate bias is recommended.

Key findings

  • Acceptable intra-rater reliability was found for both empathic opportunity identification and empathic behavior rating.

  • Poor inter-rater reliability revealed subjective differences in identifying and rating empathic opportunities and behaviors.

What the study adds

This study shows acceptable intra-rater but poor inter-rater reliability for the ECCS, emphasizing the need to modify definitions for occupational therapy and consider averaging scores from multiple raters to enhance inter-rater reliability.

Appendix

Figure 1.

Figure 1.

Open in a new tab

The flowchart of the rater training process.

Footnotes

Author’s Note: Meng-Lin Lee is also affiliated to School of Medicine, National Tsing Hua University, Hsinchu, Taiwan.

Research ethics: This study was approved by the Research Ethics Committee, National Taiwan University Hospital (the approval number: 202006114RINC; the year of approval: 2020).

Consent: Informed consent was obtained from the patients, interns, and clinical supervisors, and the consent was written.

Patient and public involvement data: During the development, progress, and reporting of the submitted research, Patient and Public Involvement in the research was included in the conduct of the research

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was partly supported by a grant from the National Science and Technology Council in Taiwan (110-2511-H-002-016-MY2).

Contributorship: Yu-Jeng Ju: idea generation, drafting, data analysis.

Meng-Lin Lee: drafting, providing valuable feedback and suggestions for improvement.

Jou-Chieh Chen: critical revision of the manuscript, empathic behaviors rating.

Yi-Ching Wang: providing valuable feedback and suggestions for improvement.

Chih-Ping Chen: providing valuable feedback and suggestions for improvement.

Han-En Tsai: idea generation, critical revision of the manuscript, empathic behaviors rating.

Tzu-Yi Wu: providing valuable feedback and suggestions for improvement.

Ching-Lin Hsieh: idea generation, critical revision of the manuscript, providing valuable feedback and suggestions for improvement.

References

  1. Banerjee M, Capozzoli M, McSweeney L, Sinha D. (1999). Beyond kappa: A review of interrater agreement measures. Canadian Journal of Statistics 27: 3–23. [Google Scholar]
  2. Baron-Cohen S, Jolliffe T, Mortimore C, et al. (1997). Another advanced test of theory of mind: Evidence from very high functioning adults with autism or Asperger syndrome. Journal of Child psychology and Psychiatry 38: 813–822. [DOI] [PubMed] [Google Scholar]
  3. Baron-Cohen S, Wheelwright S, Hill J, et al. (2001). The “Reading the Mind in the Eyes” Test revised version: A study with normal adults, and adults with Asperger syndrome or high-functioning autism. The Journal of Child Psychology and Psychiatry and Allied Disciplines 42: 241–251. [PubMed] [Google Scholar]
  4. Brazeau C. M. L. R., Schroeder R., Rovi S., Boyd L. (2010). Relationships Between Medical Student Burnout, Empathy, and Professionalism Climate: Academic Medicine 85: S33–S36. [DOI] [PubMed] [Google Scholar]
  5. Bylund CL, Makoul G. (2002). Empathic communication and gender in the physician–patient encounter. Patient Education and Counseling 48: 207–216. [DOI] [PubMed] [Google Scholar]
  6. Bylund CL, Makoul G. (2005). Examining empathy in medical encounters: An observational study using the empathic communication coding system. Health Communication 18: 123–140. [DOI] [PubMed] [Google Scholar]
  7. Bylund CL, Taylor G, Mroz E, et al. (2022). Empathic communication in dignity therapy: Feasibility of measurement and descriptive findings. Palliative & Supportive Care 20: 321–327. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Cuff BM, Brown SJ, Taylor L, et al. (2016). Empathy: A review of the concept. Emotion Review 8: 144–153. [Google Scholar]
  9. Davis MH. (1980). A multidimensional approach to individual differences in empathy. Journal of Personality and Social Psychology 10: 85. [Google Scholar]
  10. Dibbelt S, Schaidhammer M, Fleischer C, Greitemann B. (2009). Patient-doctor interaction in rehabilitation: The relationship between perceived interaction quality and long-term treatment results. Patient Education and Counseling 76: 328–335. [DOI] [PubMed] [Google Scholar]
  11. Dyrbye LN, Power DV, Massie FS, Eacker A, Harper W, Thomas MR, Szydlo DW, Sloan JA, Shanafelt TD. (2010). Factors associated with resilience to and recovery from burnout: A prospective, multi-institutional study of US medical students. Medical Education 44: 1016–1026. [DOI] [PubMed] [Google Scholar]
  12. Elfenbein HA, Ambady N. (2002). On the universality and cultural specificity of emotion recognition: a meta-analysis. Psychological Bulletin 128: 203. [DOI] [PubMed] [Google Scholar]
  13. Fields SK, Mahan P, Tillman P, et al. (2011). Measuring empathy in healthcare profession students using the Jefferson Scale of Physician Empathy: Health provider—Student version. Journal of Interprofessional Care 25: 287–293. [DOI] [PubMed] [Google Scholar]
  14. Folstein MF, Folstein SE, McHugh PR. (1975). “Mini-mental state”. A practical method for grading the cognitive state of patients for the clinician. Journal of Psychiatric Research 12: 189–198. [DOI] [PubMed] [Google Scholar]
  15. Foster A, Chaudhary N, Kim T, et al. (2016). Using virtual patients to teach empathy: A randomized controlled study to enhance medical students’ empathic communication. Simulation in Healthcare 11: 181–189. [DOI] [PubMed] [Google Scholar]
  16. Foster A, Harms J, Ange B, et al. (2014). Empathic communication in medical students’ interactions with mental health virtual patient scenarios: A descriptive study using the empathic communication coding system. Austin Journal of Psychiatry and Behavioral Sciences 1: 6. [Google Scholar]
  17. Goodchild CE, Skinner TC, Parkin T. (2005). The value of empathy in dietetic consultations. A pilot study to investigate its effect on satisfaction, autonomy and agreement. Journal of Human Nutrition and Dietics 18: 181–185. [DOI] [PubMed] [Google Scholar]
  18. Hall JA, Schwartz R, Duong F, et al. (2021). What is clinical empathy? Perspectives of community members, university students, cancer patients, and physicians. Patient Education and Counseling 104: 1237–1245. [DOI] [PubMed] [Google Scholar]
  19. Halpern J. (2003). What is clinical empathy? Journal of General Internal Medicine 18: 670–674. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hojat M. (2016). Empathy in Health Professions Education and Patient Care. Springer Cham. [Google Scholar]
  21. Hojat M, Gonnella JS, Nasca TJ, et al. (2002). Physician empathy: Definition, components, measurement, and relationship to gender and specialty. American Journal of Psychiatry 159: 1563–1569. [DOI] [PubMed] [Google Scholar]
  22. Hojat M, Mangione S, Nasca TJ, et al. (2001). The Jefferson Scale of Physician Empathy: development and preliminary psychometric data. Educational and Psychological Measurement 61: 349–365. [Google Scholar]
  23. Hsieh C-L. (2017). Decision Making Model Taken by Occupational Therapists and Their Patients. Taipei: Occupational Therapy Symposium of National Taiwan University. [Google Scholar]
  24. Huang S-L. (2022). Development and Evaluation of a Communication Skills Training Program for Occupational Therapy Interns in Physical Dysfunction Practice. Taipei: National Taiwan University. [Google Scholar]
  25. Humphris GM, Kaney S. (2001). The Liverpool brief assessment system for communication skills in the making of doctors. Advances in Health Sciences Education: Theory and Practice 6: 69–80. [DOI] [PubMed] [Google Scholar]
  26. Kane GC, Gotto JL, Mangione S, et al. (2007). Jefferson Scale of Patient’s Perceptions of Physician Empathy: Preliminary psychometric data. Croatian Medical Journal 48: 81–86. [PMC free article] [PubMed] [Google Scholar]
  27. Kelly C. (2022). Improving empathy of occupational therapy students through reading literary narratives. Journal of Occupational Therapy Education 6: 4. [Google Scholar]
  28. Löffler CS, Greitemeyer T. (2023). Are women the more empathetic gender? The effects of gender role expectations. Current Psychology 42: 220–231. [Google Scholar]
  29. Mehrabian A, Epstein N. (1972). A measure of emotional empathy. Journal of Personality 40: 525–543. [DOI] [PubMed] [Google Scholar]
  30. Mercer SW, Higgins M, Bikker AM, Fitzpatrick B, McConnachie A, Lloyd SM, Little P, Watt GC. (2016). General practitioners’ empathy and health outcomes: A prospective observational study of consultations in areas of high and low deprivation. The Annals of Family Medicine 14: 117–124. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Mercer SW, Maxwell M, Heaney D, et al. (2004). The consultation and relational empathy (CARE) measure: Development and preliminary validation and reliability of an empathy-based consultation process measure. Family Practice 21: 699–705. [DOI] [PubMed] [Google Scholar]
  32. Mokkink LB, Terwee CB, Gibbons E, et al. (2010). Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) checklist. BMC Medical Research Methodology 10: 82. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Moudatsou M, Stavropoulou A, Philalithis A, Koukouli S. (2020). The role of empathy in health and social care professionals. Healthcare 8(1): 26. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Parkin T, de Looy A, Farrand P. (2014). Greater professional empathy leads to higher agreement about decisions made in the consultation. Patient Education and Counseling 96: 144–150. [DOI] [PubMed] [Google Scholar]
  35. R Core Team (2023). R: A Language and Environment for Statistical Computing [Manual]. R Foundation for Statistical Computing. Available at: https://www.R-project.org/ (accessed 20 May 2023).
  36. Revelle W. (2022). Psych: Procedures for Psychological, Psychometric, and Personality Research [Manual]. Available at: https://CRAN.R-project.org/package=psych (accessed 20 May 2023).
  37. Serrada-Tejeda S, Martínez-Piedrola RM, Huertas-Hoyas E, et al. (2022). Empathy in occupational therapy students: A cross-sectional study at a Spanish university. BMJ Open 12: e058821. [DOI] [PMC free article] [PubMed] [Google Scholar]
  38. Shin HS, Park H, Lee Y-M. (2022). The relationship between medical students’ empathy and burnout levels by gender and study years. Patient Education and Counseling 105: 432–439. [DOI] [PubMed] [Google Scholar]
  39. Silverman J. (2009). Teaching clinical communication: A mainstream activity or just a minority sport? Patient Education and Counseling 76: 361–367. [DOI] [PubMed] [Google Scholar]
  40. Sulzer SH, Feinstein NW, Wendland CL. (2016). Assessing empathy development in medical education: A systematic review. Medical Education 50: 300–310. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Taylor RR, Lee SW, Kielhofner G, et al. (2009). Therapeutic use of self: A nationwide survey of practitioners’ attitudes and experiences. The American Journal of Occupational Therapy 63: 198–207. [DOI] [PubMed] [Google Scholar]
  42. Waisel DB, Ruben MA, Blanch-Hartigan D, et al. (2020). Compassionate and clinical behavior of residents in a simulated informed consent encounter. Anesthesiology 132: 159–169. [DOI] [PubMed] [Google Scholar]
  43. Wittenberg-Lyles E, Debra PO, Demiris G, et al. (2012). Conveying empathy to hospice family caregivers: Team responses to caregiver empathic communication. Patient Education and Counseling 89: 31–37. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from The British Journal of Occupational Therapy are provided here courtesy of SAGE Publications

RESOURCES