Impact of Student Ethnicity and Primary Childhood Language on Communication Skill Assessment in a Clinical Performance Examination

Abstract

Background

Clinical performance examinations (CPX) with standardized patients (SPs) have become a preferred method to assess communication skills in US medical schools. Little is known about how trainees’ backgrounds impact CPX performance.

Objective

The objective of this paper is to examine the impact of student ethnicity, primary childhood language, and experience of diversity on the communication scores of a high-stakes CPX using SPs.

Design

This research was designed as an observational study.

Participants

The participants of this study were third-year medical students at one US medical school.

Measurements and Main Results

The measurements used in this study were CPX scores from mandatory exam, student demographics and experience with diversity measured by self-report on a survey, and Medical College Admission Test (MCAT) and United States Medical Licensing Examination (USMLE) scores. A total of 135 students participated. Asian and black students scored lower than white students on the communication portion of the CPX by approximately half a standard deviation (Asian, 67.4%; black, 64.4%; white, 69.4%, p < .05). There were no differences by ethnicity on history/physical exam scores. Multivariate analysis controlling for MCAT verbal scores reduced ethnic differences in communication scores (Asian–white mean differences = 1.95, p = 0.02), but Asian–white differences were eliminated only after sequential models included primary childhood language (difference = 0.57, p = 0.6).

Conclusions

Even after controlling for English language knowledge as measured in MCAT verbal scores, speaking a primary childhood language other than English is associated with lower CPX communication scores for Asian students. While poorer communication skills cannot be ruled out, SP exams may contain measurement bias associated with differences in childhood language or culture. Caution is indicated when interpreting CPX communication scores among diverse examinees.

INTRODUCTION

Physician communication skills are critical to effective patient care. The amount of curricular time devoted to communication has increased in recent years, and this emphasis has been matched by enhanced assessment of clinical performance.^1–3 Comprehensive clinical examinations using standardized patients (SPs) are now the norm for assessing communication skills in most medical schools⁴ and, since 2004, are a component of the USMLE step 2 licensing process.

Doctor–patient communication skills are particularly crucial in cross-cultural interactions. Studies of racial/ethnic differences in health care have highlighted the role of communication in health care disparities.⁵ Legislators and health advocates have called for an assessment of practicing physicians’ cross-cultural communication skills and have argued that comprehensive clinical examinations be required not only of trainees, but also of practicing physicians.⁶ Implicit within this argument is the belief that SP examinations capture important components of communication between clinicians and patients of diverse ethnicity. Yet, while the psychometric properties of SP exams are well established,⁷ little is known about the relationship between student/physician background and performance on comprehensive clinical exams. The data that do address this topic derive primarily from studies done in the UK, or with international medical graduates, and may not generalize to US students.^8–12

We conducted a study of third-year US medical students to determine the relationship between student ethnicity, primary childhood language, experience with diversity, and performance on the communication portion of an SP exam, reasoning that students with more exposure to diversity would be more sensitive to communication complexity and, hence, perform better.

METHODS

We combined data from several sources at one US medical school. We conducted a survey of medical students midway through their third year. The students subsequently took a mandatory comprehensive clinical performance exam (CPX) administered at the end of their third year. The Medical College Admission Test (MCAT) and United States Medical Licensing Examination (USMLE) step 1 scores were used as measures of baseline academic performance (MCAT exams are taken before medical school admission, and USMLE step 1 examinations are taken toward the end of the second year of medical school). The medical school institutional review board approved the study.

We developed a framework for how medical student ethnicity could be associated with communication after reviewing the literature on SP exams and medical student attitudes about communication, which suggests that experience with diversity may positively influence performance.^11,13 Our survey was designed to capture experiences that have been posited to influence communication skills. The survey contained items on student demographics including self-reported ethnicity, childhood experience with racial diversity, primary childhood language, international travel experience, and interest in community service and caring for underserved populations. Ethnicity was assessed by self-report. We asked students to check “all that applied” from a list of ethnic categories, or choose “other” and insert an ethnicity. Participation in the survey was voluntary. The survey was administered electronically, and students were informed that their answers were confidential.

The CPX Exam

The CPX is a required exam consisting of eight stations, with SPs presenting common ambulatory medicine problems, such as headache or diabetes. The exam was developed by the eight schools comprising the California Consortium for the Assessment of Clinical Competence. The eight cases used 24 SPs, three to each station. Eighteen SPs were white (cases 1–6), three were African–American (case 7), and three were Latina (case 8). The exam was conducted entirely in English. After each 15-min interaction, SPs scored students using checklists for the eight cases. The number of history and physical examination checklist items varied by case, but each case included an assessment of communication called physician–patient interaction (PPI) using the common ground instrument, which consists of seven items on a six-point Likert scale¹⁴ (see Table 1). Checklists for the CPX consisted, for each student after the eight stations, of a total of 104 history taking, 51 physical examination, 37 information sharing, and 56 PPI items. Standardized patients participated in 20 h of training over multiple sessions. Trainers conducted random assessments of checklist accuracy throughout the examination. Checklist accuracy in the consortium exceeds 95%.¹⁵

Table 1.

Common Ground Instrument: PPI Items

PPI assessment	Description
Appeared professionally competent	Seemed to know what he/she was doing, inspired my confidence, appeared to have my interests at heart
Effectively gathered information	Collected information in a way that seemed organized, began with several open-ended questions and progressed through interview using a balanced ratio of open- to closed-ended questions, summarized periodically
Listened actively	Paid attention to both my verbal and nonverbal cues, used facial expressions/body language to express encouragement, avoided interruptions, asked questions to make sure he/she understood what I said
Established personal rapport	Introduced self warmly; verbally/nonverbally showed interest in me as a person, not just my condition; avoided technical jargon
Appropriately explored my perspective	Encouraged me to identify everything that I needed to say
Addressed my feelings	Acknowledged and demonstrated interest in my expressed and/or unexpressed feelings and experience
Met my needs	Worked toward a plan which addressed both the diagnosis and my concerns about my illness

SP checklist: Outstanding, very good, good, needs improvement, marginal, and unacceptable

Analysis

To assess checklist-item reliability, we calculated Cronbach alpha for each station and across stations. Cronbach alpha assesses measurement error due to content sampling and, in the terminology of generalizability theory, represents the generalizability coefficient for items.¹⁶ However, it should be noted that in the context of this study, there are other potentially important sources of measurement error that could be estimated with a larger sample size and a more complex reliability design, such as measurement error due to cases.

We scaled communication (PPI) and history/physical exam scores from 0–100. We first examined bivariate associations between student characteristics and the mean communication score, averaged across the eight clinical cases, using t tests. Then, to assess the independent effects of student characteristics, we conducted a repeated-measures analysis with the communication score for each student SP dyad as the unit of analysis. Recognizing that the communication scores were clustered both at the level of the student (intraclass correlation of 11%) and at the level of the SP (intraclass correlation of 32%), we used mixed effects models with independent, “crossed” random effects for both students and SPs. In this analysis, we considered key demographic variables and those factors from our conceptual model that were statistically associated with communication or history/physical exam scores in bivariate analysis at the p < 0.1 level (Table 2). The repeated measures approach allowed us to examine the influence of the concordance of student and SP ethnicity, a within-subject predictor that varies for each student across clinical cases; we also assessed the interaction between student ethnicity and MCAT verbal score. In addition, we used nested models to examine mediation of the overall association of the ethnicity and communication score. Mediation in this analysis is captured by attenuation of the estimated ethnic differences when the proposed mediators are added to the model. We controlled for results on the communication portion of the exam by repeating all analyses for the history/physical exam score. Finally, we repeated all the above analyses using each clinical case and each communication item as separate outcomes to determine whether the scores on any one case or any single communication item drove the overall results.

Table 2.

Communication and History/Physical Exam Scores by Student Characteristic (N = 135)

	Number of patients	Percent	Mean PPI score	P value	Mean history and physical exam score	P value
Gender				0.05		0.3
Female	78	59	69.0		59.8
Male	57	41	67.5		58.6
Age				0.4		0.6
23–27	95	70	68.6		59.5
28–40	40	30	67.9		58.9
Ethnicity				0.04		0.8
White	64	47	69.4		59.7
Asian/PI	44	33	67.4		59.2
Black	4	3	64.4		57.2
Latino	11	8	67.9		60.4
Other	12	9	68.1		57.4
Primary language childhood				0.002		1.0
English	77	57	69.4		59.3
Other	58	43	67.0		59.3
MCAT verbal				0.04		0.6
5–9	36	26	66.8		58.4
10–11	71	52	68.7		59.7
12–13	28	21	69.4		59.5
USMLE				0.5		0.003
Step 1	135		227.5		227.5
Parent in health profession				0.5		0.3
Yes	40	30	68.7		60.3
No	95	70	68.2		58.9
Family’s economic background				0.7		0.1
Working class	28	21	67.9		57.5
Middle class	67	49	68.4		59.2
Well-to-do	40	30	68.7		60.8
Linguistic diversity childhood				0.3		0.02
Not at all	24	18	66.8		60.8
Minimal	38	28	69.3		58.1
Somewhat	39	29	68.7		61.6
Very	26	19	68.2		56.7
Extremely	8	6	67.9		57.6
Racial diversity childhood				0.7		0.4
Not/Minimal	56	42	68.3		59.3
Somewhat	44	32	68.8		60.3
Very/Extremely	35	26	67.9		58.1
Importance of care for				0.2		0.3
Underserved	32	24			60.3
Somewhat	103	76	69.3		59.0
Extremely			68.1
Lived abroad			69.5
Never, 3 months	30	22	67.7	0.05	58.5	0.1
3 months–1 year	40	29	69.3		60.9
1–3 years	36	27	66.9		59.8
Over 3 years	29	21			57.3
Community service involvement				0.9		0.7
None/a little	18	13	68.0		58.2
Some	48	35	68.4		59.6
A great deal	69	51	68.5		59.4
Humanities vs basic Science				0.5		0.5
Person	33	24			60.3
Mainly science	75	56	69.2		59.3
Balanced	27	20	68.1		58.3
Mainly humanities			68.1

PPI, common ground instrument for physician–patient interaction

RESULTS

A total of 135 of the 136 students who participated in the CPX completed the full survey (Table 2), including 44 Asians (33%), 4 blacks (3%), and 11 Latinos (8%). Almost half the class (43%) spoke a language other than English in their childhood home (34:44 Asians, 9:11 Latinos, 1:4 blacks, 6:64 whites, and 8:12 others). Cronbach alpha for the communication score at each station ranged from 0.79 to 0.89 and across stations was 0.89.

The association between student characteristic and mean communication and mean history/physical exam scores are presented in Table 2. Students who self-identified as Asian or black scored lower than whites on the communication portion of the examination (Asians, 67.4; black, 64.4; white, 69.4; p < .05 for each pair-wise comparison), but not on the history/physical exam portion. The Latino–white difference did not reach statistical significance (Latino, 67.9; white, 69.4; p = 0.28). Whereas there was no difference in communication scores by student age, women scored higher than men (69.0 vs 67.5, p = 0. 05), and students whose primary childhood language was English scored higher than those whose primary language was not English (69.4 vs 67.0, p = 0. 002). Parents’ profession, family economic background, exposure to racial diversity in childhood, interest in working with underserved populations, community service, and self-identification with humanities vs basic sciences were not associated with communication scores.

MCAT verbal score between 5 and 9 was associated with lower communication score, although not with history/physical exam score, while USMLE step I score was associated with history/physical exam, but not communication score. MCAT biology and physics scores were not associated with PPI, but were associated with history/physical exam score (data not shown).

Sequential models are displayed in Table 3. After adjusting for age and gender (model 1), the relationship between ethnicity and communication score was maintained: Asians and black students continued to score lower than white students. When MCAT verbal scores were entered into the model (model 2), along with ethnicity, age, gender, and MCAT verbal score, the white–Asian difference persisted, but the Black–white difference was eliminated. Model 3, which maintained the demographic and the MCAT verbal variable and added primary childhood language, rendered the white–Asian difference insignificant. Repeating the analysis with primary childhood language, added before MCAT verbal scores, did not alter the results. Table 3 also includes results of the same sequential modeling on the mean history/physical exam score. Ethnicity was not associated with the score in history/physical exam model 1 (shown), and negative confounding, which would conceal an association, was eliminated by sequential models (data not shown).

Table 3.

Sequential Multivariate Models for Communication Scores and History/Physical Exam Scores (N = 135)

Variable list	PPI adjusted model I			PPI adjusted model II			PPI adjusted model III			History and physical exam adjusted model I
	Mean difference	95%CI	P value	Mean difference	95%CI	P value	Mean difference	95%CI	P value	Mean difference	95%CI	P value
Gender
Female	1.48	(0.04, 2.92)	0.04	1.33	(−0.11, 2.77)	0.07	1.29	(−0.13, 2.71)	0.07	0.96	(−1.36,3.28)	0.5
Male
Age
23–27	1.18	(−0.42, 2.78)	0.2	1.27	(−0.33, 2.87)	0.1	1.23	(−0.35, 2.81)	0.1	1.09	(−1.50, 3.67)	0.4
28–40
Ethnicity
White
Asian/PI	−2.14	(−3.79, −0.49)	0.01	−1.95	(−3.61, −0.29)	0.02	−0.57	(−2.64, 1.50)	0.6	−1.35	(−4.02, 1.31)	0.3
Black	−4.45	(−8.64, −0.26)	0.04	−3.40	(−7.69, 0.90)	0.1	−3.12	(−7.37, 1.13)	0.2	−3.00	(−9.76, 3.74)	0.4
Latino	−1.70	(−4.34, 0.94)	0.2	−0.54	(−3.41, 2.34)	0.7	0.90	(−2.22, 4.03)	0.6	−0.25	(−4.51, 4.00)	0.9
Other	−1.34	(−3.94, 1.25)	0.3	−1.03	(−3.63, 1.58)	0.4	0.14	(−2.64,2.92)	0.9	−3.55	(−7.73, 0.64)	0.1
MCAT verbal
5–9				−2.16	(−4.41, 0.08)	0.06	−2.09	(−4.31, 0.12)	0.06
10–11				−0.64	(−2.46, 1.19)	0.5	−0.60	(−2.40, 1.20)	0.5
12–13
Primary language childhood									0.03
English							2.02	(0.17, 3.88)
Other

Model I = gender, age, and ethnicity; model II = model I + MCAT verbal; model III = model II + primary childhood language; PPI, common ground instrument for physician–patient interaction

When Asians were stratified by primary childhood language, there was no difference in MCAT verbal scores between those who spoke English as their primary childhood language and those who did not. For students whose primary childhood language was English, however, there was no association between Asian ethnicity and communication score; the association was strong for those whose primary childhood language was not English. Inserting an interaction term for ethnicity and MCAT verbal score did not change the analysis, which confirmed the stratified results. There was no association between SP and student ethnicity with concordance terms in the model or in stratified analysis. We repeated the analyses using the PPI communication score for each case, and then each PPI item as an outcome. The strength of association between ethnicity and PPI was similar across all cases and items.

DISCUSSION

Our study examining the association between student background, experience of diversity, and communication score has three main findings indicating a complex interaction between ethnicity and student performance on a comprehensive assessment using SPs. First, we found that Asian and black students scored lower on the communication items than their white counterparts, with a trend toward lower scores for Latinos as well. Second, we found that while accounting for MCAT verbal scores eliminated communication score differences for Black students, Asian–white differences were only eliminated after controlling for primary childhood language. Finally, we found that other measures of experience with diversity were not associated with better communication scores. Taken together, these findings contradict our initial hypothesis that students with a greater experience with diversity would be better performers on the communication portion of a clinical practice exam.

The ethnic score difference we observed was equivalent to slightly over half a standard deviation, usually considered a moderate effect size. It is difficult to interpret the importance of this difference, but one study found that an intensive 2-year communication skills curriculum raised SP exam scores by slightly under one standard deviation, suggesting that the difference we observed is on the order of magnitude achieved through curricular attempts to improve communication.¹⁷

Differences between black and white students appear to be associated with skills captured by MCAT verbal scores. More research, including a greater number of African American students, is needed to explore the relationship between communication skill and MCAT verbal score, noted in this study and in other studies.¹⁸

Why do Asian students score lower on the communication portion of the CPX, even after accounting for MCAT verbal scores? There are two possible explanations: These students may be worse communicators, or the CPX may contain some form of measurement bias. Without a gold standard test, or actual patient outcomes, neither hypothesis can be excluded. A simple explanation would be that the students’ communication ability is hampered because of lack of fluency in English, consistent with prior studies showing the effect of English proficiency on CPX scores of international medical graduates.¹¹ However, the students in this study are all fluent in English, with mean MCAT verbal scores above the 80th percentile of MCAT examinees.¹⁹ The sequential models instead suggest an important role for primary childhood language.

That childhood language, rather than English knowledge, is associated with communication scores is intriguing, yet subject to multiple interpretations. Childhood language may be associated with subtle language and nonverbal communication issues, not captured in the written examination setting by MCAT verbal scores, which surface in student SP communication. Alternatively, childhood language may be a proxy for cultural differences in communication. Culture influences multiple aspects of physician–patient communication, including expectations, interpersonal agendas, and assumptions.²⁰ The PPI checklist may inadvertently codify cultural norms specific to the dominant culture. Checklist items such as “active listening” or “establishing personal rapport” may be particularly culturally determined, although we found no evidence in our study that these items were more highly associated with student ethnicity. SP training, in general, may also favor certain cultural practices, such as smiling or acknowledging feelings. Moreover, other communication behaviors that are culturally determined, such as the physical distance between participants, avoidance of uncertainty, or use of time, may influence communication scores despite not being listed as checklist items. Whereas culturally determined communication practices might resonate with culturally similar patients, the SP sample in this study was not very diverse, and the standardization process of SP training may inadvertently eliminate the inherent diversity of patient responses.

Finally, it is possible that the communication behaviors measured by SPs differ from an actual patient’s experience of communication and, hence, are an incomplete measure of communication. Whereas this discrepancy may be true for all patients, the difference between physician behavior and patient experience has been highlighted in studies with ethnically diverse patients. In a study examining the relationship between patient and physician ethnicity and communication behavior, ratings for physician participatory decision-making style did not differ by physician ethnicity, but patients with racially concordant physicians rated their visits as significantly more participatory than patients in racially discordant relationships.²¹ Similarly, in a study using videotapes and transcripts to rate physicians’ shared decision-making behaviors, ethnically diverse patients’ subjective experience of collaboration—how they felt about making a decision with their doctor—often did not match what would be expected from the observed communication behavior.²² These studies challenge educators to capture other important facets of communication, in addition to communication behavior. Moreover, they emphasize the need to create culturally heterogeneous SP scenarios and exam tools that can be used to assess student communication performance.

Our study has several limitations. It presents results from one class at one medical school at one point in time; results may not generalize to other students or to practicing physicians. However, the fact that the study took place at a medical school with high MCAT verbal scores strengthens the finding that factors other than English ability are at play in communication scores. The small number of black and Latino students limits interpretation of findings related to these groups. We caution that, although the differences we saw are statistically significant, they are drawn from four students, and although there was a trend toward lower scores for Latino students as well, we cannot draw strong conclusions due to the small numbers of Latinos in our sample. The number of Asian students did not allow us to distinguish possible differences by country of ancestry. Similarly, the lack of diversity among the SPs, in particular, the lack of Asian SPs, does not allow us to completely rule out an effect of SP ethnicity on student scores. Although other studies that have examined this question have found no or minor interaction effects between SP and student ethnicity, more research with greater numbers of diverse SPs is needed before an association between SP ethnicity and student score can be ruled out.⁸ Finally, we cannot determine whether differences in scores translate to differences in actual patient care or patients’ experience of communication.

Despite the enthusiasm for communication skills assessment, a growing literature has urged caution.^23,24 Our study, which, to our knowledge, is the first to examine how the background of high-performing US students impacts CPX scores, raises important questions that should prompt further research on communication skills assessment with diverse students, SPs, and scoring methods.