Improvement in medical students’ communication and interpersonal skills as evaluated by patient satisfaction questionnaire after curriculum reform

Abstract

Fifteen years of undergraduate medical education curriculum reform at Saga Medical School was evaluated by measuring medical students’ communication and interpersonal skills with a patient satisfaction questionnaire developed by the American Board of Internal Medicine. A multiphase cross-sectional study was conducted at the General Medicine Clinic of Saga Medical School Hospital in phase I (1998–1999), phase II (2001–2002), and phase III (2009–2010). A total of 1,963 patient ratings for 437 medical students’ performance was analyzed. The average scores of phases II and III were significantly higher than for phase I. The average score of female students showed a significant difference between phases I and II, but no difference between phases II and III. The average score of male students showed no difference between phases I and II, but significant difference between phases II and III. The phase II curriculum introduced basic clinical skills and examination and improved female students’ performance. The phase III curriculum was effective for male students because it emphasized the clinical skill program more and introduced problem-based learning. Curriculum reform at Saga Medical School is considered to have made good progress in improving students’ clinical competence and patient-centered attitudes.

Introduction

Japanese medical education at under- and postgraduate levels has undergone systemic reform in the past 2 decades. The undergraduate curriculum was converted from traditional teacher-centered and examination-driven passive learning to student-centered and problem-oriented active learning to improve medical students’ clinical competence and patient-centered attitudes.^(1,2) In 2001, the Subcommittee for Research and Development of Medical Education Programs at the Ministry of Education, Culture, Sports, Science, and Technology issued Model Core Curriculum for Medical Education that outlined essential core components of the undergraduate medical education program in knowledge, skills, and attitude domains. Problem-based learning (PBL) and clinical clerkship were emphasized as the main teaching and learning strategies in undergraduate medical education.⁽³⁾

In 2005, the common achievement test (CAT) system was established prior to the clinical clerkship year and has been shown to strongly influence the preclinical curriculum.⁽⁴⁾ The CAT is designed to test the quality of medical students before allowing them to graduate to clinical clerkship. The CAT consists of two parts: 300 multiple-choice questions (MCQs) in computer based-testing (CBT) to measure knowledge, and a six- to eight-station objective structured clinical examination (OSCE) to measure clinical skills and patient-centered attitudes.⁽⁵⁾ To meet the CAT-OSCE requirements, many medical schools have expanded their basic clinical skills training program, which covers medical interview and physical examination skills in the preclinical curriculum. Well-equipped clinical laboratories were established and a related clinical skills training program including simulated patient (SP) participation has been developed.^(6–8)

An urgent issue for further development is to evaluate the effectiveness of educational reform. Workplace-based evaluation by focusing on actual student performance and patient benefits means that Kirkpatrick’s third or fourth level is insufficient for Japanese medical education.^(9,10) We have focused on the change in medical students’ performance in parallel to the curricular reform implemented at Saga Medical School (SMS) from the late 1990s to the late 2000s.^(11,12) The tool we used to evaluate students’ performance was a patient satisfaction questionnaire (PSQ) developed by the American Board of Internal Medicine (ABIM).⁽¹³⁾ The ABIM-PSQ was developed as a validated instrument to assess physicians’ communication skills and interpersonal skill by actual patients. We evaluated the effectiveness of 15 years of medical educational curriculum reform by comparing PSQ scores between phases in this study.

Methods

Saga Medical School educational reform

Fig. 1 shows the framework of the SMS curriculum. The first 4 years are allocated to preclerkship medical education, including the liberal arts, basic science, and theoretical aspects of clinical medicine. In their fifth and sixth year, students participate in a clinical clerkship at the university hospital and related institutions. Integrated organ/system-based curriculum and clinical clerkship had been adopted by the SMS since it was established in 1978. However, since the late 1990s, teaching strategies, the clinical skill-training program in the preclerkship curriculum, and the student evaluation system had been in a process of reform. The reform process was divided into three characteristic phases (Table 1).

Fig. 1.

Basic framework of Saga Medical School curriculum.

Table 1.

Outline of curriculum reform in Saga Medical School

	Phase I (late 1990s)	Phase II (early 2000s)	Phase III (late 2000s)
Preclerkship curriculum
Curriculum structure	Organ system-based integrated curriculum	Organ system-based integrated curriculum	Organ system-based integrated curriculum
Teaching strategy	Systematic lecture	Systematic lecture	PBL and lecture
Clinical skill training	20 h	35 h	81 h
Requirement for participating clinical clerkship
Assessment of medical knowledge	Full credit in preclerkship curriculum	Full credit in preclerkship curriculum	Full credit in preclerkship curriculum and CAT-CBT
Assessment of clinical skills and attitudes	OSCE: two stations	OSCE: five stations	CAT-OSCE: six stations
Clinical clerkship in 5th and 6th year
Duration	55 weeks	55 weeks	55 weeks
Student assessment of clinical clerkship	MCQs	MCQs and four stations OSCE	MCQs and four stations OSCE

In phase I during the late 1990s, the main teaching strategy for preclerkship medical education was based on systematic lectures and paper examination. The basic clinical skills training program in the preclerkship curriculum was 20 h. In 1997, the OSCE was introduced at the end of the fourth year. However, the OSCE stations were composed of only two physical examinations without a medical interview. As the OSCE was an experimental trial and used more formative than summative assessments, students could participate in the clerkship program with the achievements of full credits in the preclerkship curriculum. After 55 weeks of clinical clerkship, a paper examination composed of 210 MCQs is conducted as the summative assessment of the clerkship program.

In phase II during the early 2000s, the clinical skills program was expanded up to 35 h. The OSCE prior to clinical clerkship was expanded to five physical examination stations and used as a summative assessment. Although an SP group was established at the SMS and participated in medical interview training, the role of the SP was to demonstrate medical models with a clinician teacher. Medical students learned interview skills mainly by role-play. The advanced OSCE was composed of a series of four stations (medical interview, physical examination, laboratory work, and chart writing) and was introduced as the summative assessment of the clerkship program in addition to MCQs.

In phase III during the late 2000s, 55 weeks of PBL was adopted as the main teaching strategy for clinical science in the third and fourth years.⁽¹⁴⁾ The total hours of lectures in the third and fourth years was reduced from 1,177 h to 527 h to protect students’ self-learning time. The clinical skills program was expanded up to 81 h. The clinical skills program included 19 h of interview training with a well-trained SP. The SP group provided interview training not only for building good patient–doctor relationships and information gathering to make a diagnosis, but also patient education and motivation. In 2005, SMS joined the Common Achievement Test Organization and introduced CBT and OSCE as the summative assessment. During the research period, SMS did not make major revisions to its admission policy and entrance exam selection process.

Setting and design

This multiphase cross-sectional study was conducted at the General Medicine Clinic (GMC) at SMS Hospital in 1998, 1999, 2001, 2002, 2009, and 2010. This study was approved by the Ethics Committee of SMS Hospital.

Participants

A 2-week clinical clerkship at the GMC is mandatory for all medical students. Patients attending the GMC are all patients on their initial visit without referral to a specialty department, and also include patients requiring comprehensive and continuous care from the university hospital. From 1978 to 2004 at GMC, medical students performed their clerkship in their final (sixth) year and from 2005, students began their clerkship in their fifth year.

The patient satisfaction survey was conducted at GMC in 1998, 1999, 2001, 2002, 2009, and 2010. We obtained patient satisfaction scores from new patients interviewed and examined by medical students at GMC. We used 1998 and 1999 data as a representative sample in phase I, 2001 and 2002 data as representative of phase II, and 2009 and 2010 as representative of phase III. In total, we obtained PSQ scores from 1,963 outpatients for 437 medical students. Table 2 summarizes the characteristics of medical students and outpatients.

Table 2.

Characteristics of participants

	Total	Phase I (1998–1999)	Phase II (2001–2002)	Phase III (2009–2010)
Medical student number	437	123	139	175
Gender (male)	252 (57.7%)	72 (58.5%)	81 (58.2%)	99 (56.6%)
School grade		6th year	6th year	5th year
Patient number	1963	405	657	901
Gender (male)*	889 (45.3%)	198 (48.9%)	271 (41.2%)	420 (46.7%)
Age (range)**	53.3 (15–92)	49.3 (15–90)	49.8 (15–89)	57.5 (15–92)

*A significant difference among phases; chi-square test, p<0.05; **one-way ANOVA, p<0.001.

Instrument

To measure student performance, we selected the PSQ developed by ABIM. The official Japanese-language translation of PSQ has been approved for use in Japan by the ABIM.⁽¹¹⁾

Though the original ABIM-PSQ includes ten items with a five-point Likert scale (poor, fair, good, very good, and excellent), only six items were selected for adaption to the undergraduate curriculum at SMS (Table 3). Questions 1, 2, and 4 are items to evaluate the student’s patient-centered humanistic attitude, and questions 3, 5, and 6 are items for evaluating students’ communication skills. Selection of these items in this study was justified because the ABIM-PSQ developers reported that any items scoring more than four out of ten would not deteriorate score reliability.⁽¹³⁾ The alpha value was 0.92, which suggests a high degree of internal consistency for these six items.

Table 3.

Six items of Patient Satisfaction Questionnaire developed by American Board of Internal Medicine

1. Greeting you warmly; calling you by the name you prefer; being friendly; never crabby or rude

2. Treating you like you’re on the same level; never ”talking down” to you or treating you like a child

3. Letting you tell your story; listening carefully; asking thoughtful questions; not interrupting you while you’re talking

4. Showing interest in you as a person; not acting bored or ignoring what you have to say

5. Encouraging you to ask questions; answering them clearly; never avoiding your questions or lecturing you

6. Using words you can understand when explaining your problems and treatment; explaining any technical medical terms in plain language

Data-gathering process

Within 2 weeks of GMC clerkship program, students independently interviewed several newcomer patients with indirect faculty supervision. Immediately after the initial student–patient encounter, a research assistant explained our research objectives to patients and asked patients to anonymously complete the PSQ provided. Time taken to complete the PSQ was flexible and patients were ensured privacy. We gave patients a guarantee that their PSQ answers would not affect their treatment and/or management. The completed questionnaires were collected by the research assistant. After these processes were completed, faculty physicians interviewed and examined those patients. Patients aged less than 15 years were excluded. Patients who were cognitively or physically unable to complete the PSQ were also excluded.

Medical students did not have access to the PSQ and were reassured that their PSQ score would not influence their academic achievements.

Statistical analysis

Participants’ characteristics across the three phases were compared using the chi-square test and one-way analysis of variance (Table 1). Patients’ gender and age showed significant differences between three phases. Because the ratio of male patients in phase II was lower than in other phases and patient age in phase III was higher, these factors were included as covariates to obtain our main result.

To obtain the difference in PSQ score between the three phases, repeated measures analysis of covariance (RM-ANCOVA) was conducted after converting the PSQ scores into a continuous scale (poor, 1; fair, 2; good, 3; very good, 4; and excellent, 5). The difference between the six items was a within-subject factor, the phase (I, II, and III) and student gender were between-subject factors, and patient age and gender were covariates. Huynh–Feldt epsilon correction was used to test subject effects involving more than one degree of freedom. The Bonferroni post-hoc test was conducted to examine the difference between factors. Statistical analysis was performed with SPSS software (ver. 22.0; IBM SPSS, Armonk, NY). A p value of less than 0.05 was considered significant.

Results

A total of 1,963 patients’ ratings for 437 medical students’ performance were analyzed. Table 4 shows the average score of the six PSQ items between the three phases. RM-ANCOVA revealed significant difference between the six PSQ items (F_{(4.3, 8452.6)} = 155.6, p<0.001). The post-hoc test indicated that the average score of question 5 ”Encouraging patients to ask questions and answering clearly” was ranked lowest (item score, 3.36 ± 0.85, p<0.001). Question 6 ”Explanations with plain language” was ranked second lowest (item score, 3.49 ± 0.80, p<0.001). In the between-subject factors, phase, student gender, and the interaction between phase and student gender were significant after removing covariate effects. The average scores for phase II (3.58 ± 0.75) and phase III (3.61 ± 0.73) were significantly higher than the score for phase I (3.38 ± 0.86, p<0.001), but no difference was found between the scores for phases II and III (p = 0.65).

Table 4.

Overall scores of Patient Satisfaction Questionnaire

PSQ Items	Total	Phase I	Phase II	Phase III
1. Greeting and friendliness	3.57 (0.71)	3.51 (0.74)	3.59 (0.72)	3.58 (0.71)
2. Respect for patients	3.61 (0.72)	3.57 (0.76)	3.61 (0.74)	3.63 (0.68)
3. Listening carefully	3.61 (0.74)	3.50 (0.81)	3.67 (0.72)	3.62 (0.71)
4. Showing interest personally	3.65 (0.73)	3.55 (0.72)	3.65 (0.72)	3.69 (0.72)
5. Encouraging to ask questions and answering clearly	3.36 (0.85)*	2.95 (0.97)	3.43 (0.80)	3.49 (0.77)
6. Explanations with plain language	3.49 (0.80)**	3.22 (0.93)	3.50 (0.78)	3.61 (0.72)
Overall average	3.55 (0.77)	3.38 (0.86)	3.58 (0.75)	3.61 (0.73)

The Cronbach’s alpha value was 0.92, suggesting a high degree of internal consistency of six items. *Significantly lower than any other questions; RM-ANCOVA, p<0.001. **Significantly lower than question 1 to 4; RM-ANCOVA, p<0.001.

The average score of female students was significantly higher than for male students (3.57 ± 0.73 vs 3.46 ± 0.72, p<0.01). To examine the student gender-specific phase effect, we performed the RM-ANCOVA and post-hoc tests separately for each gender. Fig. 2 shows the female students’ scores from phases I to III. RM-ANCOVA showed significant difference in phase (F_{(1, 848)} = 19.26, p<0.001). Post-hoc tests indicated significant difference between phases I and II (3.35 vs 3.68, p<0.001), but no difference between phases II and III (3.68 vs 3.66, p = 0.50). A post-hoc test for question 1 ”greeting and friendliness” showed significant improvement between phases I and II (3.56 ± 0.73 vs 3.73 ± 0.72, p<0.05), but significant deterioration between phases II and III (3.73 ± 0.72 vs 3.59 ± 0.69, p<0.05) was noted. Questions 2 to 6 showed significant improvement between phases I and II, but no difference between phases II and III.

Fig. 2.

Improvement of Patient Satisfaction Questionnaire scores of female student in a time-dependent manner. The average score of female students showed significant difference between phases I and II (3.35 vs 3.68, p<0.001), but no difference between phases II and II (3.68 vs 3.66, p = 0.50). For each item, question 1 showed significant improvement between phases I and II (3.56 ± 0.73 vs 3.73 ± 0.72, *p<0.05), but significant deterioration between phases II and III (3.73 ± 0.72 vs 3.59 ± 0.69, *p<0.05) was noted. Questions 2 to 6 showed significant improvement between phases I and II (**p<0.01, ^†p<0.001), but no difference between phases II and III.

The average score of male students also shows significant difference in phase (F_{(1, 1105)} = 11.0, p<0.05) as shown in Fig. 3. The Bonferroni post hoc test showed no difference between phases I and II (3.37 vs 3.47, p = 0.24), but significant difference between phases II and III (3.47 vs 3.61, p<0.01). For each item, only question 5 showed significant improvement between phases I to II (2.93 ± 0.99 vs 3.31 ± 0.80, p<0.001) and phases II and III (3.31 ± 0.80 vs 3.48 ± 0.76, p<0.01). Question 6 showed improvement between phases II and III (3.41 ± 0.76 vs 3.60 ± 0.72, p = 0.001).

Fig. 3.

Improvement of Patient Satisfaction Questionnaire scores of male student in a time-dependent manner. The average score of male students showed no difference in average score between phases I and II (3.37 vs 3.47, p = 0.24), but significant difference between phase II and III (3.47 vs 3.61, p<0.01). For each item, question 5 showed significant improvement between phases I to II (2.93 ± 0.99 vs 3.31 ± 0.80, *p<0.001) and phases II and III (3.31 ± 0.80 vs 3.48 ± 0.76, **p<0.01), and question 6 showed improvement between phases II and III (3.41 ± 0.76 vs 3.60 ± 0.72, *p = 0.001).

Discussion

The main finding of our study is that SMS students’ performance as evaluated by the ABIM-PSQ in our outpatient clinic showed significant improvement from phase I (1998–1999) to phase II (2001–2002) and phase III (2009–2010). However, the process of improvement was different between genders. Female students showed remarkable improvement in phase II, but no improvement in phase III. By contrast, male students showed no significant improvement in phase II, but meaningful improvement in phase III. Additionally, the improvement in male students was mainly by addressing item score issues arising from question 5 ”Encouraging patients to ask questions and answering clearly” and question 6 ”Explanations with plain language”, which were the lowest two scores for male communication skills in the six PSQ items.

Because there was no systematic revision in clinical clerkship between phases I and II, the improvement shown by female students in phase II was considered to be caused by our preclerkship program reform. The reform of phase II emphasized standardized clinical skill training and evaluation in addition to the framework of a traditional lecture-based examination-oriented learning. Therefore, female students’ improvement in phase II may be because they may be able to apply their newly acquired basic clinical skills to actual practice in clinical clerkship. Gude and colleagues reported that the improvement of communication skills during internship occurred only in female physicians and speculated that the gender difference was because male physicians were slow learners compared with female students in acquiring communication skills.⁽¹⁵⁾

The phase III curriculum was characterized by a change to problem-oriented self-directed learning with more emphasis on basic clinical skill training, which improved male students’ performance especially in communicating with patients to help them understand their health problems and to make decisions about medical management. This finding may suggest that it is possible that PBL affected male students’ performance. PBL is a learning strategy that not only builds a knowledge base, but also is adaptable enough to solve patient problems or to identify learning issues, and uses small group discussion for students to share their thinking processes with their colleagues. Previous research shows that students who were taught the PBL curriculum performed better in communication skills and hypothesis generation.^(16,17) Male students’ communication skills and clinical reasoning may be developed by repetitive group discussions over the 55 weeks of the PBL curriculum at SMS.

Patient satisfaction is a useful perspective for evaluating education quality.^(18,19) The importance of communication and interpersonal skills has been emphasized as one of the main competency domains because physician communication and interpersonal skills are associated with improved patient satisfaction, better health outcomes, greater adherence to treatment, and more active self-management of chronic illnesses.^(20–25) From this perspective, curriculum reform to improve students’ clinical competence and patient-centered attitude has made fairly good progress.

However, previous research showed that patients’ evaluation was positively biased.^(18,26) This is also evidence that the quality and reputation of the institution tends to produce skewed patient satisfaction scores.⁽²⁷⁾ The possibility that the quality of medical practice and SMS hospital’s reputation affected patients’ rating is impossible to deny. We could not control these factors, which are limitations for this survey. A 360-degree evaluation of students in further research on curriculum evaluation should be introduced, which includes a multiple evaluation tool and evaluators in addition to patient satisfaction. Another limitation of our research is that it was conducted in a single educational institution in Japan, which limits the generalizability of our findings.

Conclusion

Evidence was found that curriculum reform improved clerkship students’ communication and interpersonal skills using Kirkpatrick’s third or fourth level evaluation. Curriculum reform at SMS was performed based on the Model Core Curriculum for Medical Education; therefore our findings may be useful for other Japanese medical schools that have introduced similar curriculum reforms to improve medical students’ clinical competence and patient-centered attitudes.

Abbreviations

ABIM

American Board of Internal Medicine

CAT

common achievement test

CBT

computer based-testing

GMC

General Medicine Clinic

MCQs

multiple-choice questions

OSCE

objective structured clinical examination

PBL

problem-based learning

PSQ

patient satisfaction questionnaire

RM-ANCOVA

repeated measures analysis of covariance

SMS

Saga Medical School

SP

simulated patient

Conflict of Interest

No potential conflicts of interest were disclosed.