Medical school selection criteria as predictors of medical student empathy: a cross-sectional study of medical students, Ireland

Abstract

Objectives

To determine whether performance in any of the Health Professions Admissions Test (HPAT) sections, most specifically the interpersonal understanding section, correlates with self-reported empathy levels in medical students.

Setting

The study was conducted in University College Cork, Ireland.

Participants

290 students participated in the study. Matching HPAT scores were available for 263 students. All male and female undergraduate students were invited to participate. Postgraduate and international students were excluded.

Primary and secondary outcome measures

Primary measures: HPAT-Ireland and Jefferson Scale of Physician Empathy (JSE) scores were compared including subsection analysis. Secondary measures: comparisons were made between groups such as gender and year of programme.

Results

A total of 290 students participated. Males scored significantly higher than females for total HPAT-Ireland (U=7329, z=−2.04, p<0.05), HPAT-Ireland section 1 (U=5382, z=−5.21, p<0.001) and section 3 scores (U=6833, z=−2.85, p<0.01). In contrast, females scored significantly higher than males on HPAT-Ireland section 2 (U=5844, z=−4.46, p<0.001). Females demonstrated significantly higher total JSE scores relative to males (mean score ± SEM: 113.33±1.05vs 109.21±0.95; U=8450, z=−2.83, p<0.01). No significant association was observed between JSE scores and any of the HPAT-Ireland measures (all p>0.05). There was no effect of programme year on JSE scores (all p>0.05).

Conclusion

The introduction of the HPAT-Ireland test was partly designed to identify students with strong interpersonal skills. A significant finding of this study is that JSE values did not correlate with HPAT-Ireland scores. This study suggests no clear link between scores on a selection test, the HPAT-Ireland, which is designed to assess several skill domains including interpersonal skills, and scores on a psychometric measure of empathy, at any point during medical education.

Strengths and limitations of this study.

• The results of this study add to the body of research in the area of medical entrance methods, particularly in Ireland.

• The use of a widely used, reliable tool, the Jefferson Scale of Physician Empathy (JSE) as a measure of empathy strengthens the results of this study.

• Use of a longitudinal assessment of JSE scores would provide a more valid approach to the questions of whether there is an empathy decline across years spent in medical school

• The response rate varied across the various year groups, ranging from 50% to 74%, increasing the possibility of response-rate bias.

Introduction

Empathy is regarded as one of the most important competencies required by health professionals and is considered an important focus of medical education curricula.^{1 2} Physician empathy has been positively correlated with better patient comprehension,³ more accurate diagnosis,⁴ increased treatment adherence,⁵ and decreased emotional distress and improved quality of life among patients.^{6 7} It has been proposed that empathy is not a unitary concept and that it is highly influenced by contextual factors in practice.⁸ The Jefferson Scale of Physician Empathy (JSE) is an instrument which has been developed and employed to measure the empathy levels of healthcare professionals.^{1 9} Explanatory factor analysis of the scale suggests a multidimensional construct comprising three factors: ‘perspective-taking’, ‘compassionate care’ and ‘standing (walking) in patient's shoes’.^{9 10} A more recent analysis of the factor structure of the JSE scale suggested an additional fourth factor, ‘metacognitive effort’, which measures the level of insight possessed by the student regarding the need to think like the patient.¹ It is, at present, unclear which of these factors is more strongly associated with improved patient outcomes.

Research in medical education has primarily employed the JSE instrument to demonstrate a strong relationship between empathy levels and ratings of clinical competence in medical students and trainees.^{11 12} Empathy levels have also been linked to motivation to study medicine¹³ as well as increased peer ratings in relation to professionalism¹⁴ and leadership¹⁵ among medical students. Empathy has also been shown to be moderately associated with intention to pursue a career in a people-oriented rather than a procedure-oriented specialty after graduation.^{16 17} Another notable finding reported in the medical student empathy literature concerns the reported decline in empathy during undergraduate training;¹⁸ however, Costa et al¹⁹ argue this decline has been linked with diminution in the quality of healthcare provision and is likely attributable to a constellation of factors including elements of the hidden curriculum and inadequate role models. In light of data linking physician and student empathy with better clinical performance, empathy has been defined as an important graduate outcome for undergraduate curricula in USA and Canada.^{20 21} Guidelines for teaching professionalism within the undergraduate curriculum in Ireland and the UK have also tried to incorporate the concept of empathy.²²

There is considerable diversity in the methods used in medical school admission and selection. While academic achievement is a heavily weighted selection criterion, specialised aptitude tests are also widely used internationally.²³ In the literature on medical school selection practices, there has also been some discussion about the utility of methods such as the situational judgement test for assessing non-academic attributes including empathy which are considered important within the profession.^{24 25} Since 2009, the Irish medical school application process ranks applicants on the basis of secondary level grades as well as performance in the Health Professions Admission Test-Ireland (HPAT-Ireland), which shares a similar structure to the Undergraduate Medicine and Health Sciences Admission Test (UMAT), used in Australia and New Zealand. HPAT-Ireland was introduced with the intention of broadening access to medical school, increased emphasis on non-academic attributes and bringing Irish medical schools’ recruitment policy in line with international norms.²⁶ A 2006 national educational report entitled ‘Medical Education in Ireland—A New Direction’, recommended a more diverse entry mechanism to medical education in Ireland and proposed that an alternative mechanism ‘must still ensure that students selected have the intellectual and emotional capability for a demanding course and profession’.²⁷ The HPAT tool was noted to be reliable and valid by the National Evaluation group.²⁸ Information about the development of the various test items is not readily available. This has previously been noted by a study relating to Irish medical school entry mechanisms.²⁶

The HPAT contains three separate sections: logical reasoning and problem-solving (analysis of graphically presented information), interpersonal understanding (demonstrating awareness of thoughts, feeling and intentions of scenario characters) and non-verbal reasoning (pattern shape detection and sequence prediction).

Predictive validity of any selection test is its ability to predict subsequent performance in medical school. Previous research has shown that HPAT scores were a poor predictor of clinical or communication skills performance.²⁶ In the current study, we sought to determine whether performance in any of the HPAT sections, but perhaps most specifically the interpersonal understanding section, would correlate with self-reported empathy levels in undergraduate medical students.

Methods

Study design and procedure

A cross-sectional design was employed in the present study. The sample consisted of a cohort of medical students across years 1–5 of the undergraduate medical programme, who had sat the HPAT selection test prior to obtaining a place in medical school. Paper-based questionnaires were distributed to all year groups during lectures during the academic year 2014/2015. All students who had sat the HPAT examination were invited to participate, thereby excluding all international students and students registered on the graduate-entry medical programme. Student registration numbers were used to obtain HPAT scores from the university database.

Questionnaires were provided to students at the beginning of a lecture and collected after the end of the lecture. There was no incentive to complete the questionnaire. Data pertaining to gender, age, year of programme and year HPAT was undertaken were collected. Desired future career specialty was also noted, but the responses were more incomplete and these data were not analysed or reported. For the purpose of the analysis, students not present in class on the day were also considered non-responders to the questionnaire.

Study instruments

Empathy was evaluated using the student version of the Jefferson Scale of Physician Empathy—JSE–S.¹⁰ This self-report survey tool includes 20 items rated on a 7-point Likert-type scale with anchors 1: Strongly disagree and 7: Strongly agree. It has been widely used for the study of empathy in clinicians and in medical students.^{1 10}

The JSE has been shown to be internally consistent²⁹ and to have been positively correlated with ratings of clinical competence.¹¹ The student version of the JSE, which was used in this study, has been shown to be reliable.³⁰The JSE was significantly and positively correlated with sociability subscale scores of the Zuckerman-Kuhlman Personality Questionnaire.¹⁶

All candidates who sit HPAT-Ireland sign a waiver allowing their results to be analysed for research purposes. The study was approved by the Clinical Research Ethics Committee of the Cork Teaching Hospitals.

Data analysis

All data were entered into and analysed using SPSS version 21.0 for Windows (SPSS, Chicago, Illinois, USA). Descriptive statistics, mean, SEM and median (where appropriate) were used to describe continuous variables and frequencies to describe categorical variables. Mann-Whitney U tests and one-way analysis of variance (ANOVA) were employed to compare HPAT-Ireland and JSE scores between groups (eg, gender, year of programme). Pearson's correlation coefficient analysis was employed to examine the relationship between continuous baseline and outcome variables. A correlation coefficient of ≥0.20 or ≤−0.20 was defined as a threshold for scientific significance. Multiple linear regression analysis was used to identify significant predictors of JSE score variation in the present sample.

Results

Demographics

A total of 290 students participated in this study, providing a response rate of 59.2% (290/490). Of this sample, 53.4% (n=155) were female and distributed across the following age (years) categories: 17–21 (51%), 22–24 (44.8%), <24 (4.2%). The dataset included HPAT-Ireland scores collected between 2009 and 2014, in a student sample across years 1–5 of the undergraduate programme (year 1: n=68, 71%; year 2: n=45, 45%; year 3: n=50, 49%; year 4: n=59, 59%; year 5: n=68, 74%). Of the 290 surveys collected, matching HPAT scores were available for 263 students, which was the final sample size available for correlational and/or regression analyses. One-way ANOVA examined the influence of year in which HPAT was taken on both total score and individual HPAT section values and revealed a significant main effect for the factor ‘year’ for total HPAT-Ireland score (F(5, 257) = 13.16, p<0.001), HPAT-Ireland section 1 (logical reasoning and problem-solving) (F(5, 257) = 3.71, p<0.01), section 3 (non-verbal reasoning) (F(5, 257) = 19.67, p<0.001), but not HPAT-Ireland section 2 (interpersonal understanding) scores (p>0.05). Posthoc Bonferroni analyses confirmed that HPAT-Ireland total and HPAT-Ireland section 3 were significantly lower in 2014 relative to all years between 2010–2013 inclusive (all p<0.05) and the same pattern was observed for the year 2010, where HPAT-Ireland total and HPAT-Ireland section 3 scores were significantly lower than comparable 2012–2014 values (all p<0.05).

HPAT-Ireland and JSE scores: relationship with demographic and educational variables

Descriptive statistics for HPAT-Ireland and JSE variables are outlined in table 1. Males scored significantly higher than females for total HPAT-Ireland (U=7329, z=−2.04, p<0.05), HPAT-Ireland section 1(logical reasoning and problem-solving) (U=5382, z=−5.21, p<0.001) and section 3 (non-verbal reasoning) scores (U=6833, z=−2.85, p<0.01). In contrast, females scored significantly higher than males on HPAT-Ireland section 2 (interpersonal understanding) (U=5844, z=−4.46, p<0.001). Females demonstrated significantly higher total JSE scores relative to males (mean score ± SEM: 113.33±1.05 vs 109.21±0.95; U=8450, z=−2.83, p<0.01). One-way ANOVA comparisons revealed no effect of programme year on JSE scores (p>0.05; table 1).

Table 1.

Descriptive statistics for the variables of interest

Variable	N	M (SEM)	Median
HPAT-Ireland section 1: logical reasoning and problem-solving (Max=100)	263	61.31 (0.47)	61.0
HPAT-Ireland section 2: interpersonal understanding (Max=100)	263	59.21 (0.46)	59.0
HPAT-Ireland section 3: non-verbal reasoning (Max=100)	263	66.12 (0.52)	66.0
Total HPAT-Ireland (Max=300)	263	184.93 (16.0)	185.0
Total JSE (Max=140)	290	111.42 (0.72)	112.0

HPAT-Ireland, Health Professions Admissions Test-Ireland; JSE, Jefferson Scale of Physician Empathy; M, mean.

HPAT-Ireland and JSE scores: results of correlational analyses

Table 2 provides a summary of the results of the correlational analyses. A weak but statistically significant positive correlation was observed between HPAT-Ireland section 3 (non-verbal reasoning) scores and HPAT-Ireland section 1 (logical reasoning and problem-solving) (r=0.21, p<0.05). A weak but statistically significant negative correlation was observed between HPAT-Ireland section 3 scores and HPAT-Ireland section 2 (interpersonal understanding) (r=−0.20, p<0.05). Unsurprisingly, HPAT-Ireland total scores strongly correlated with each section subscore: HPAT-Ireland section 1 (r=0.67, p<0.0001), HPAT-Ireland section 2 (r=0.43, p<0.001) and HPAT-Ireland section 3 (r=0.58, p<0.001). No significant association was observed between JSE scores and any of the HPAT-Ireland measures (all p>0.05; table 2).

Table 2.

Correlations between HPAT-Ireland and JSE scores

	JSE	Total HPAT-Ireland	HPAT-Ireland 3	HPAT-Ireland 2
HPAT-Ireland section 1: logical reasoning and problem-solving	−0.082	0.672†	0.201*	−0.060
HPAT-Ireland section 2: interpersonal understanding	0.0550	0.431†	−0.195*	–
HPAT-Ireland section 3: non-verbal reasoning	−0.014	0.584†	–	–
Total HPAT-Ireland	−0.021	–	–	–

Correlation coefficients correspond to results of Pearson's r analysis.

*p<0.05.

†p<0.001.

HPAT-Ireland, Health Professions Admissions Test-Ireland; JSE, Jefferson Scale of Physician Empathy.

Multiple regression analysis

Multiple regression analysis was undertaken to investigate the nature of the relationship between empathy, HPAT scores and demographic or educational variables (gender and year of programme). Table 3 provides a summary of the factors affecting JSE score variation in the present sample. Consistent with the correlational analysis, none of the selected variables emerged as a significant predictor of medical student empathy. Tables 4 and 5 depict the gender analysis.

Table 3.

Factors affecting JSE score variation

	B	Standard Error B	β	p Value
HPAT-Ireland section 1: logical reasoning and problem-solving	−0.06	0.10	-0.06	0.377
HPAT-Ireland section 2: interpersonal understanding	0.04	0.10	0.04	0.524
HPAT-Ireland section 3: non-verbal reasoning	0.03	0.09	0.03	0.630
Sex	0.08	1.57	0.08	0.238
Programme year	−0.08	0.49	-0.08	0.221

Model fit: R²=0.02, F=1.07, p=0.38. B=unstandardised beta co-efficient, β = standardised beta co-efficient, p-value= significance value

Table 4.

Gender analysis (female)

		JSE total	HPAT total	HPAT 3	HPAT 2
HPAT section 1	Pearson's correlation	−0.079	0.701†	0.246†	−0.004
	p Value	0.346	0	0.003	0.965
	N	143	143	143	143
HPAT section 2	Pearson's correlation	0.124	0.463†	−0.261†
	p Value	0.142	0	0.002
	N	143	143	143
HPAT section 3	Pearson's correlation	−0.04	0.543**
	p Value	0.639	0
	N	143	143
HPAT total	Pearson's correlation	0.004
	p Value	0.963
	N	143

Correlation coefficients correspond to results of Pearson's r analysis.

*p<0.05.

†p<0.001.

HPAT, Health Professions Admissions Test; HPAT-Ireland section 1, logical reasoning and problem-solving; HPAT-Ireland section 2, interpersonal understanding; HPAT-Ireland section 3, non-verbal reasoning; JSE, Jefferson Scale of Physician Empathy.

Table 5.

Gender analysis (male)

		JSE total	HPAT total	HPAT 3	HPAT 2
HPAT section 1	Pearson's correlation	−0.02	0.653†	0.079	0.069
	p Value	0.828	0	0.392	0.452
	N	120	120	120	120
HPAT section 2	Pearson's correlation	−0.104	0.507†	−0.047
	p Value	0.256	0	0.61
	N	120	120	120
HPAT section 3	Pearson's correlation	0.051	0.611
	p Value	0.578	0
	N	120	120
HPAT total	Pearson's correlation	−0.026
	p Value	0.775
	N	120

Correlation coefficients correspond to results of Pearson's r analysis.

*p<0.05.

†p<0.001.

HPAT, Health Professions Admissions Test; HPAT-Ireland section 1, logical reasoning and problem-solving; HPAT-Ireland section 2, interpersonal understanding; HPAT-Ireland section 3, non-verbal reasoning; JSE, Jefferson Scale of Physician Empathy.

Discussion

The introduction of the new entry mechanism for medical school to Ireland, via completion of the HPAT-Ireland test was partly designed to identify students with strong interpersonal skills, who would be able to demonstrate empathy as future clinicians. These results, which have failed to show any association between selection test scores (including subscales designed to measure interpersonal understanding) and individual variation on a psychometric measures of empathy, adds to a growing literature questioning the validity of the HPAT-Ireland test as a selection tool. For example, Quinn and colleagues³¹ administered a modified version of the HPAT-Ireland to a sample of hospital consultants, non-consultant hospital doctors and medical students. They reported no group differences in test performance, suggesting that increased clinical experience is uncorrelated with what is assessed across the three sections of this selection tool. A more recent study examined the relationship between HPAT-Ireland scores and communication and clinical skills subtest scores in a Year 1 Objective Structured Clinical Examination.²⁶ These authors also reported no relationship between selection test scores and scores for either subtest.

HPAT-Ireland and JSE scores: gender differences

The present study demonstrated that males scored significantly higher than females on HPAT subsections 1 (logical reasoning and problem-solving) and 3 (non-verbal reasoning). Existing published evidence has demonstrated that there are slight gender differences in total HPAT-Ireland scores, generally favouring males but varying from year to year.³² Similar gender differences have been reported in relation to other medical schools selection tests, for example, UK Clinical Aptitude Test (UKCAT), Medical College Aptitude Test (MCAT), UMAT.^{33 34} With respect to specific subsection score differences, the present results are in agreement with previous data which have shown that males perform better on measures of non-verbal reasoning, while females score higher on items measuring interpersonal understanding.^35–37 Women consistently score higher than men in relation to JSE scores.^{10 11} In this study, females also scored higher than males using the JSE-S scale.

JSE scores and evidence for decline of empathy in medical school

The results of the present cross-sectional analysis demonstrated that JSE values did not vary significantly across the curriculum. Specifically, total JSE scores were comparable among both students who had recently entered and those at the point of exit from the undergraduate programme. Earlier studies have shown erosion of empathy in medical school,^{18 38} particularly during the transition from the preclinical to the clinical cycle; this has been attributed to several factors, including lack of role models, high academic workload, time pressure.³⁸ However, the present results are in agreement with the conclusions arising from a review of 11 longitudinal studies which interrogated changes in self-reported empathy levels in students at various stages of medical training.³⁹ This review concluded that there was no definitive evidence to suggest that empathy levels declined during medical education and that studies purporting to show such deficits were compromised by inappropriate conceptualisation and measurement of empathy (ie, self-report measures including the JSE) in a patient care context. This explanation does not fully explain the current study results (ie, lack of association between HPAT-Ireland and empathy), where the JSE measure was also employed. Rather, the absence of an empathy decline may represent students’ increasing awareness of the recent emphasis placed by professional bodies including the Irish Medical Council on the importance of empathy and compassion in the development of professionalism.²²

HPAT-Ireland and criterion validity

An ability to communicate with patients and understand their concerns is considered important when assessing global clinical competence. JSE scale scores have been shown to be positively correlated with ratings of clinical competence.¹¹ Other instruments for measuring empathy such as the Interpersonal Reactivity Index (IRI),^{40 41} were reported to have modest relationships with the JSE scores among medical students.⁴² In a study involving medical students, it was reported that the scores of the JSE were significantly and positively correlated with sociability subscale scores of the Zuckerman-Kuhlman Personality Questionnaire.¹⁶ Therefore, when reflecting on the observed lack of association between HPAT and JSE scores, the criterion validity (which reflects the extent to which a measure is related to an outcome) of the HPAT-Ireland test must be questioned. In a recent review of the construct validity of HPAT-Ireland, it was noted that criterion validity is a major question mark for more recently devised selection tests including HPAT-Ireland's performance.⁴³ These authors suggested that selection tests containing more science-related, knowledge-based items, such as MCAT, may be more predictive in relation to the early years of the undergraduate programme, where there is a greater focus on biomedical sciences. In contrast, they propose that HPAT-Ireland, which assesses a broader range of reasoning and interpersonal skills, may be more predictive in the clinical years.⁴³

Study limitations

One of the limitations of the present study is the use of a cross-sectional design in a single educational institution and therefore may not be representative of the national experience. Use of a longitudinal assessment of JSE scores would provide a more valid approach to the questions of whether there is an empathy decline across years spent in medical school. Additionally, the response rate varied across the various year groups, ranging from 50% to 74%, increasing the possibility of response bias. Data pertaining to the non-responders were not collected or analysed and therefore it is not evident if this group would vary significantly from those studied. Selection bias could exist in this study given that convenience sampling during lectures was used. It is possible that an increased response rate or the analysis and comparison of data related to non-responders could have potentially affected the results of the study.

All measures of empathy, including the JSE, are at best a proxy of empathic behaviour. It might also be argued items in the JSE are transparent and thus susceptible to social desirability response bias, that is, they could be answered in a way that is recognised as more socially acceptable. However, the JSE was administered in ‘non-penalising’ situation where the purpose was described as research. Respondents were assured that their responses would be confidential and would be used only for research purposes approved by the Research Ethics Committee. This assurance, in itself, may reduce respondents’ tendency to give socially desirable responses. The developers of JSE scale note that ‘the pattern of relationships in their validity studies, particularly the convergent and discriminant validities, suggests that social desirability response bias, even if operative, did not substantially distort the expected relationships’.¹⁰ These authors also conducted a study to investigate the influence of faking ‘good’ responses on JSE scores.¹⁶ In that study, they administered the S-Version of the JSE and other personality tests, including the ZKPQ to 422 first-year medical students who matriculated at Jefferson Medical College. Analysis of covariance was employed to control the effect of giving false responses on the research outcomes, using the ‘infrequency’ score as a covariate, and they reported no substantial change in the general pattern of results. These findings suggest that social desirability response bias does not distort the validity of the JSE score.

A recent study comparing the JSE and the IRI, another widely used empathy scale, across students in five countries, showed weak correlation between these two scores. This article concluded that these scales measure different constructs of empathy.⁴⁴ The relationship between the HPAT and IRI has not been assessed, and there is no readily available data from the manufacturers of the construct of the HPAT.

Conclusion

In a systematic review of the literature regarding use of empathy tests in medicine, it was concluded that while evidence of reliability, internal consistency and validity were observed for several instruments, no existing empathy measure was regarded as sufficiently reliable and valid for pretraining admission selection.⁴⁵ While some authors argue in favour of the implementation of empathy assessments during the medical school admission process,⁴⁶ the current data suggest no clear link between scores on a selection test, the HPAT-Ireland, which is designed to assess several skill domains including interpersonal skills and scores on a psychometric measure of empathy, at any point during medical education.