Assessment of Pharmacy Students' Communication Competence Using the Roter Interaction Analysis System During Objective Structured Clinical Examinations

Abstract

Objective

To determine the value of using the Roter Interaction Analysis System during objective structured clinical examinations (OSCEs) to assess pharmacy students' communication competence.

Methods

As pharmacy students completed a clinical OSCE involving an interview with a simulated patient, 3 experts used a global rating scale to assess students' overall performance in the interview, and both the student's and patient's languages were coded using the Roter Interaction Analysis System (RIAS). The coders recorded the number of utterances (ie, units of spoken language) in each RIAS category. Correlations between the raters' scores and the number and types of utterances were examined.

Results

There was a significant correlation between students' global rating scores on the OSCE and the number of utterances in the RIAS socio-emotional category but not the RIAS business category.

Conclusions

The RIAS proved to be a useful tool for assessing the socio-emotional aspect of students' interview skills.

INTRODUCTION

Communication competence is one of the generic competencies health care professionals are expected to have. Because communication competence can be improved through training, colleges and schools of pharmacy have implemented programs to develop better communication skills in students.¹ The ability to assess students' communication competence is essential to improving their communication skills²; however, development of an accurate assessment tool has been difficult.

Performance-based assessments in which students interact with a simulated patient often are used to determine pharmacy students' communication competence. Usually competence is evaluated as part of an objective structured clinical examination (OSCE).^3-7 However, there are some problems with the OSCE rating system because it depends on the raters' impression of the performance of the students. Well-designed rubrics have been developed to address variability in raters' scoring.^8,9 In addition, training of raters is necessary to reduce subjectivity in the assessment process. However, even if the raters are well trained using a standardized method of assessment, there is no format for presenting the raters' assessments. If such a format could be developed, some of the features that define “good” communication skills could be elucidated.

The Roter Interaction Analysis System (RIAS)¹⁰ is a method for coding medical dialogue that has been applied for the objective description of physician-patient communications.^11-13 We applied the RIAS to analyze the dialogue between pharmacy students and simulated patients in an OSCE and to explore the correlation between OSCE scores and RIAS scores to determine the characteristics related to communication competence in students.

METHODS

In Japan, OSCEs of pharmacy students cover 3 areas: (1) preparation of medicines (dispensing powdered medicine, dispensing liquid medicine, etc), (2) inspection of prepared medicines (checking errors in dispensing, etc), and (3) the medical interview and communication (patient reception, drug counseling, etc). Students are required to take the clinical examination at the end of the fourth year of a 6-year program. For this study, we focused on evaluating students' communication competency in conducting medical interviews with simulated patients. This trial was designed to establish a system for conducting routine OSCEs,¹⁴ training raters, and promoting research to assess students' communication competence in the Faculty of Pharmaceutical Sciences, Kyoto University. The faculty council of the Department of Pharmaceutical Sciences established a committee for planning and reviewing the examination trial. The study was approved by the Institutional Review Board.

Fifteen students (9 undergraduate and 6 postgraduate) volunteered to participate in the study, and informed consent was obtained from all of them. Undergraduate and postgraduate students were included to ensure that a range of communication competency levels were represented in the study.

The patient case used for the OSCE was that of a diabetic patient who was hospitalized for diabetes patient education. The setting for the OSCE was the pharmacy student's interview of the patient upon admission to gather information regarding the patient's medical history, with particular focus on drug therapy and related matters such as allergic events. Each interview was limited to 5 minutes. The 3 raters who evaluated the students were teaching staff members involved in communication education in both the Faculties of Medicine and Pharmaceutical Sciences, Kyoto University, who were certified as official raters by the Common Achievement Test Organization in Japan. The raters used a 6-point global rating scale¹⁵ (6 = outstanding, 5 = excellent, 4 = good, 3 = fair, 2 = poor, and 1 = unsatisfactory) to assess each student's overall performance in the interview. A rating of outstanding indicated that the student's performance did not differ from that of a real pharmacist. Four rater trainees also were present during the OSCEs and each assessed 7 or 8 students. Although each student was evaluated by 5 raters (3 experts and 2 trainees) at a time, only the scores of the 3 expert raters were included in this study. Each of the students was evaluated independently by all of the raters at the same time.

Three simulated patients who had been trained for more than 3 years were selected from among 30 simulated patients registered with Kyoto University. They were trained specifically for this study on 2 separate days prior to the OSCEs. On the first day, they received the case scenario and learned the details of the history of the disease and the patient's background. They also were challenged with questions that would likely be posed by the students and learned how to respond to these questions during the simulation. The patient actors conducted role plays and then evaluated each other to confirm that their responses reflected the case scenario. On the second day, the simulated patients/patient actors performed a role play with the trainer based on the scenario that would be used in the OSCE. The 3 expert raters were not engaged in the simulated patient training. Informed consent was obtained from each of simulated patients.

All of the interviews were videotaped. We used the RIAS to code the utterances of the patients and students separately and to clarify the characteristics of the dialogue. As defined above, utterances are units of spoken language or statements.¹⁶ We grouped each utterance into one of 41 mutually exclusive categories comprised of 15 socio-emotional and 26 business categories (Appendix 1). The coders wrote down all of the words spoken by the students and the patients so that the accuracy of the coding could be confirmed. The coding was conducted by 2 coders who completed an official training course for RIAS and were certified in Japan as RIAS coders. These 2 coders did not participate in the OSCE as raters. Prior to study analysis, inter-coder agreement in assigning utterances to RIAS categories was confirmed based on Spearman's correlation coefficients for the categories with a frequency greater than 2 per simulation.¹⁷

The average of the 3 expert raters' scores for each student's general performance was used as the global rating score for each student. Correlations between the cumulative number of utterances coded by RIAS and OSCE scores were evaluated by Spearman's correlation coefficient. We defined statistical significance as a p of less than 0.05.

RESULTS

The overall correlation coefficients of the RIAS scores between the 2 coders for each category were 0.95 for utterances by students and 0.96 for utterance for patients, respectively. Among the 14 categories for which the average number of utterances by the student and the patient was greater than 2, the number of utterances coded by the 2 coders correlated significantly in 13 categories (Table 1).

Table 1.

Correlations of the RIAS Scores Between Two Coders for Each RIAS Category^a

Abbreviations: RIAS= Roter Method of Interaction Process Analysis System

^aSum of the RIAS scored by a student and a simulated patient were used.

The correlation coefficients of the global rating scores for students for the 3 expert raters were 0.51 (p = 0.052), 0.75 (p < 0.005), and 0.72 (p < 0.005); showing sufficient correlation to justify the use of the average score given by the 3 raters.

The number of utterances of students correlated well with students' overall OSCE scores in the socio-emotional category (Figure 1, R=0.66; p < 0.01) but not in the business category (Figure 2, R = 0.43; p = 0.109). The number of utterances by the simulated patients in the socio-emotional category correlated well with students' scores (Figure 3, R = 0.72, p < 0.005; Figure 4, R = 0.31, p = 0.265) suggesting that the dialogue of students with good scores accompanied statements related to socio-emotional elements, such as agreement, laughing, and approval. The number of utterances by both patients and students concerning business matters was not significantly related to students' global scores on the OSCEs.

Figure 1.

Relationship between the students' global rating score and number of utterances in the socio-emotional category (R= 0.662; P < 0.01).

Figure 2.

Relationship between the students' global rating score and number of utterances in the business category (R = 0.4305; P = 0.1092)

Figure 3.

Relationship between the students' global rating score and number of utterances of simulated patients (SP) in the socio-emotional category (R= 0.722; P < 0.005).

Figure 4.

Relationship between the students' global rating score and number of utterances of simulated patients (SP) in the business category (R = 0.307; P = 0.2656).

DISCUSSION

The dialogue of this study was unique compared with other RIAS studies in that the interview setting and the interview itself were between pharmacy students and simulated patients.^18,19 RIAS is sometimes used to analyze the diverse statements uttered by physicians and patients. In this study, the content of the interview was specific to the pharmacy setting. However, this enables us to transcribe the whole interview and to complete double coding (coding by 2 independent coders) compared to other researchers who have had to use random sampling of the recorded interview to ensure double coding and maintain the reliability of RIAS coding. In this study, the coding was satisfactory to describe the characteristics of each interview.

In assessing the students' communication skills, detailed checklists or global rating scales are often used. Detailed checklists might be useful to assess relatively fundamental items, for example, to confirm whether the student stated their name to the patient at the beginning of the interview. However, assessing more advanced items such as empathy in the limited time of an OSCE is more difficult. In such cases, occurrence of interrater disagreement is crucial.^20,21 Global rating scales can be more reliable and valid instruments and more feasible to use than checklists; and can be used to evaluate the general features of students' interviews, including the subtleties and nuances.^22,23 Also, the global rating scale is suitable to assess students' communication ability in the context of an OSCE investigating the validity of assessment.²⁴

To assess communication competency in the context of an OSCE, the global rating scale appears to be more appropriate than detailed checklists.²⁴ However, the global rating scale may be more subjective and the score for each element of a student's performance could be affected by the rater's personal impression. The rater's experience and the level of evaluation in the global rating evaluation system also must be considered. At present, the most important and realistic strategy is to establish a training system for the raters. In the future, it will be necessary to determine more objective guidelines for raters to use in evaluations.

In our study, among the utterances of students, the utterances of the socio-emotional category rather than the business category correlated with the global rating scales. Statements rich in socio-emotional content correlate well with high interview scores, although the purpose of the interview was to get information from patients. Interestingly, not only students' utterances but also those by patients correlated well with high global rating scores. Thus, a student's ability to make patients more talkative may be related to the student receiving a high score for dialogue.

To determine the features of the interview in a simulated clinical setting, the RIAS possibly could be useful for rating real clinical activities as well as OSCEs, and in the teaching of communication skills. Verhoeven and colleagues²⁵ reported a separate written assessment component to an OSCE. This type of assessment also can help us to evaluate students' ability to obtain accurate and complete information from a patient.

Because we limited the number of students to 15 and tested only 1 interview scenario, care must be taken in applying the statistical findings of this study to other settings. Although we ensured correlation among the global rating scores of medical communication skills experts, it might be more reliable to investigate this further with more OSCE raters. It will be particularly important in future studies to evaluate multiple interview scenarios.

CONCLUSION

We found that “good” communication skills as reflected by high global scores on the OSCE were anchored by a larger number of the socio-emotional utterances by students, and that this seemed to result in more utterances by the simulated patient. A “good” interview in the OSCE has been defined as one consisting of a numerous variety of utterances, especially in the socio-emotional category. RIAS could be a potentially useful tool to assess the features of pharmacy student interviews with simulated patients, and may be a means of conducting objective performance-based evaluations for communication competence.

Appendix 1. Categories in RIAS and Their Abbreviations

Only the RIAS categories used in this study are listed.

Adapted from The Roter Method of Interaction Process Analysis System (RIAS) by Noro et al.²⁶