Does Emotional Intelligence at Medical School Admission Predict Future Academic Performance?

Abstract

Purpose

Medical school admissions committees are increasingly considering noncognitive measures like emotional intelligence (EI) in evaluating potential applicants. This study explored whether scores on an EI abilities test at admissions predicted future academic performance in medical school to determine whether EI could be used in making admissions decisions.

Method

The authors invited all University of Ottawa medical school applicants offered an interview in 2006 and 2007 to complete the Mayer–Salovey–Caruso EI Test (MSCEIT) at the time of their interview (105 and 101, respectively), then again at matriculation (120 and 106, respectively). To determine predictive validity, they correlated MSCEIT scores to scores on written examinations and objective structured clinical examinations (OSCEs) administered during the four-year program. They also correlated MSCEIT scores to the number of nominations for excellence in clinical performance and failures recorded over the four years.

Results

The authors found no significant correlations between MSCEIT scores and written examination scores or number of failures. The correlations between MSCEIT scores and total OSCE scores ranged from 0.01 to 0.35; only MSCEIT scores at matriculation and OSCE year 4 scores for the 2007 cohort were significantly correlated. Correlations between MSCEIT scores and clinical nominations were low (range 0.12–0.28); only the correlation between MSCEIT scores at matriculation and number of clinical nominations for the 2007 cohort were statistically significant.

Conclusions

EI, as measured by an abilities test at admissions, does not appear to reliably predict future academic performance. Future studies should define the role of EI in admissions decisions.

Medical school admissions committees are tasked with selecting the most suitable candidates for entrance to their programs. Traditionally, admissions committees have focused on cognitive measures, including grade point average (GPA), the Medical College Admission Test in North America, and the Graduate Medical School Admissions Test in the United Kingdom and Australia. These cognitive measures have demonstrated predictive validity and reliability.¹ Although admissions committees have long acknowledged the importance of such cognitive skills in assessing applicants, more recently, they have come to recognize the importance of noncognitive skills as well.² In addition, the Accreditation Council for Graduate Medical Education’s six core competencies are patient care, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice.³ In Canada, the CanMEDS framework includes not only medical expert and scholar roles but also professional, manager, health advocate, communicator, and collaborator roles.⁴

To better understand applicants’ noncognitive skills, such as their communication, interpersonal, and professionalism abilities, researchers have suggested exploring the construct of emotional intelligence (EI).⁵ EI is defined as the ability to monitor one’s own and others’ emotions, to discriminate among them, and to use the information to guide thinking and actions.⁶ Outside health care, EI has been linked to individuals’ academic success, social skills, job satisfaction, and improved interpersonal relations.^7,8 Within health care, EI is considered important because understanding patients’ emotions and controlling one’s own emotions are essential to maintaining effective doctor–patient relationships and to working successfully in teams. EI also may be relevant to the competencies of professionalism and systems-based practice, which require good communication skills and teamwork.⁹ A recent systematic review of studies with empirical data on EI in physicians or medical students revealed that higher EI scores contributed to improved doctor–patient relationships, increased empathy, and improved teamwork and communication skills, as well as better stress management, organizational commitment, and leadership skills.⁹ In addition, researchers have studied specifically how to increase EI in medical students.¹⁰ Together, these findings suggest that using EI in admissions decisions could have value.

However, very few studies have examined the use of EI in assessing applicants to medical school or in predicting their future performance. Carr,¹¹ for example, found no association between EI and traditional admissions criteria. Leddy and colleagues¹² found similar results—no relationship between EI at admissions, as measured by the Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT), GPA, interview scores, and autobiographical sketch scores. If admissions committees are to use EI in high-stakes decision making, we need more information regarding the measure’s predictive validity. Specifically, we must understand if EI correlates with future academic performance and, if so, what type of performance EI predicts.

EI is a set of four distinct yet related abilities, including perceiving emotions, using emotions, understanding emotions, and managing emotions.⁵ To communicate effectively with patients, physicians must develop a rapport, trust, and an ethical therapeutic relationship, as well as accurately elicit and synthesize relevant information.⁴ Such successful communication requires physicians to complete a complex process in which they perceive their patients’ emotions, manage their own reactions, and use those emotions to facilitate their performance.⁵ Recently, McNaughton¹³ suggested that EI is a skill that is observable and measurable through assessment methods such as the objective structured clinical examination (OSCE). In an OSCE, students must communicate effectively with standardized patients while skillfully acquiring information. Thus, we expect that a student’s EI would correlate with his or her future performance on a clinical skills test, such as an OSCE, but would not correlate with his or her future performance on written examinations of knowledge.

The methods used to assess EI are relevant. In a recent systematic review of EI studies in medicine, all researchers measured EI using self-report questionnaires.⁹ Many raised significant concerns about relying on scores based on individuals’ subjective judgments of their own abilities because individuals could readily inflate their scores in a high-stakes environment, such as during the admissions process. Several authors demonstrated that faking on self-report tests changed the rank order of applicants.^14,15 Some authors suggest that using an abilities test, such as the MSCEIT v2.0, to measure EI is preferable.^11,16 Such a test directly assesses an individual’s ability to perceive emotions accurately, use emotions to facilitate thought, understand emotions, and manage emotions. It also has robust psychometric properties, is affordable, and is available in several languages.¹⁷

No gold standard exists for measuring EI, and experts in the field disagree on the construct of EI, how to measure it, and the reliability and validity of measurement tools.¹⁸ In addition, some have argued that the lack of conceptual clarity around the construct itself limits what we can interpret from EI scores.¹⁸ Thus, further research is needed to improve our understanding of this challenging construct.

The purpose of our study was to explore the use of EI as a measure of applicants’ noncognitive skills at admissions. We focused on determining the degree to which scores on an abilities test of EI predicted future academic performance on assessments of students’ cognitive and noncognitive skills in medical school.

Method

Admissions process

The University of Ottawa Medical School annually selects 160 applicants for admission from a large pool (mean n = 2,900). In a previous study, we described this process in detail.¹² In summary, on the basis of the weighted GPA (wGPA) and autobiographical sketch score, the admissions committee offers approximately 18% of applicants an interview. They calculate the wGPA using a weighting ratio of 3:2:1 for applicants’ GPAs in the three most recent years of full-time undergraduate studies, and then they determine a cutoff score. For eligible applicants, trained faculty then assess the autobiographical sketches in six areas, including education, employment, volunteering, extracurricular activities, awards, and research contributions. Three independent raters subsequently individually interview and score the selected applicants and determine each applicant’s final score through an iterative process.

Participants

We invited all applicants in 2006 and 2007 who were offered an interview to participate in the study. Those who agreed to participate completed the MSCEIT v2.0 in English or French immediately after their admissions interview in March of either 2006 or 2007, approximately six months prior to their potential matriculation. We then invited participants who were accepted to the university to complete the MSCEIT v2.0 again at matriculation. We proctored all deliveries of the test, and an outsider vendor, Multi-Health Systems Inc., scored the completed tests.

MSCEIT v2.0

The MSCEIT v2.0 is an abilities-based measure of EI that consists of 141 items and is accessible online.¹⁷ Each item is scored by a general consensus method in which a respondent’s answer is scored according to the proportion of the reference sample (n = 5,000) of the adult population that endorsed the same MSCEIT answer. The respondent’s raw item scores are compiled to generate a total EI score. The MSCEIT score is reported in a way that is similar to that of intelligence scales, with 100 as the mean reference score and 15 as the standard deviation. An individual with a MSCEIT score above 115 is thought to demonstrate a high level of EI, whereas someone with a score below 85 is considered to have a level of EI that potentially could cause interpersonal problems.

Written examinations of knowledge

At the end of each of 13 preclerkship learning blocks, all students completed multiple-choice and short-answer questions to assess their knowledge. A team of content experts constructed the examinations and linked content to specific learning objectives. Faculty generated each student’s overall score for that block by combining the student’s scores on the written, laboratory, and individual and population health examinations. In this study, we calculated participants’ mean overall scores for the six year 1 and seven year 2 learning blocks.

During clerkships, faculty also assessed students’ knowledge using written examinations. They calculated each student’s overall score for each rotation as a composite of their scores on the multiple-choice and short-answer sections. In this study, we calculated participants’ mean overall score for the eight core clerkship rotations in year 3.

OSCEs

Over the course of their four-year program, all medical students completed three OSCEs, one in each of years 2, 3, and 4. Each OSCE consisted of 9 or 10 stations that tested students’ clinical skills, such as history taking, physical examination, and management, and covered content from family medicine, internal medicine, psychiatry, pediatrics, surgery, and obstetrics–gynecology. Each station was seven minutes in length and was scored by a physician examiner who used a standardized content checklist and a communication rating scale. Content experts designed the stations, and a committee of chief examiners—four faculty members with expertise in OSCE case writing—reviewed them. All physician examiners contributed to the standard-setting process using the modified borderline group method.¹⁹

The communication rating scale used to score students’ performance included four items—(1) interpersonal skills (listens carefully; treats patient as an equal); (2) interviewing skills (uses words patient can understand; organized; does not interrupt; allows patient to explain); (3) patient education (provides clear, complete information; encourages patient to ask questions; answers questions clearly; confirms patient’s understanding/opinion); and (4) response to emotional issues (recognizes, accepts, and discusses emotional issues; controls own emotional state). Physician examiners rated each student on each item using a 5-point scale from poor to excellent, for a maximum potential score of 20 points. The physician examiners were faculty members who regularly taught medical students and participated in an orientation and training prior to the examination. Almost all had previous experience as OSCE examiners. Trained standardized patients portrayed the cases but were not involved in the scoring. Each OSCE station score comprised the mean checklist score (worth 80% of the total score) and the communication rating scale (worth 20% of the total score). The scores for each station were added to provide an overall OSCE score.

Clinical nominations and failures

In every clerkship rotation, preceptors could recognize students for their outstanding clinical performance on the basis of the preceptor’s impression of the student’s knowledge, skills, and behavior in providing patient care and the student’s role as a member of the health care team. On the basis of the aggregate information from all preceptors for that rotation, the clerkship director made the final decision regarding each nomination on the medical student performance evaluation (MSPE). In our study, we tallied the number of nominations over the two years of clerkship documented in each student’s MSPE and any failures on written examinations, OSCEs, or preceptor evaluations over the four years of medical school.

Analysis

We calculated descriptive statistics and Pearson correlations to compare participants’ MSCEIT scores with their written examination and OSCE scores as well as their scores on the communication rating scale. With a potential sample size of 160 participants in each cohort, there would be enough power to detect a significant correlation as low as 0.22. In our study, we compared the two cohorts separately because examinations and assessments could differ from one academic year to the next. To protect our calculations from an increase in the familywise error rate associated with multiple correlations, we applied a Bonferroni correction to each analysis involving one or the other cohort. All data were anonymous, and personal information for all participants was protected. The Ottawa Hospital research ethics board approved our study.

Results

The percentage of applicants who completed the MSCEIT in 2006 was 70% (333/475) and in 2007 was 67% (326/490). Participants were similar to nonparticipants in gender and eventual offer of admission. Participation in the study (once admitted to the university) ranged from 70% (105/151) to 79% (120/151) for the 2006 cohort and 66% (101/153) to 69% (106/153) for the 2007 cohort. In Table 1, we include the MSCEIT scores for both cohorts and both time periods (at interview and at matriculation). The split-half reliability coefficient for the MSCEIT was 0.87/0.89, and Cronbach alpha was 0.86/0.87, for the 2006 and 2007 cohorts, respectively. MSCEIT scores all fell within the “usual” range, as described by the test developers, with no significant differences between the cohorts.

Table 1.

Mean (Standard Deviation [SD]), Minimum, and Maximum Scores on the Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT), Written Examinations, and Objective Structured Clinical Examinations (OSCEs), University of Ottawa Faculty of Medicine^a

Table 1 also includes the mean, standard deviation, minimum, and maximum scores for the written and OSCE examinations and the communication rating scale. The year 1, 2, and 3 written examination scores fell within the expected range. The year 2, 3, and 4 OSCE scores demonstrated typical means ranging from 67% to 80%, and the distribution of scores was similar to that in previous years.

Table 2 includes the results of our correlations analysis between students’ MSCEIT scores and their written examination scores. We found no significant correlations between scores on the MSCEIT and those on the written examinations for either cohort.

Table 2.

Correlations Between Scores on the Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT) and on Written Examinations, University of Ottawa Faculty of Medicine

Table 3 includes the results of our correlations analysis between students’ MSCEIT scores and their OSCE scores. For the 2006 cohort, we found no significant correlation between total OSCE scores and MSCEIT scores or between communication subscores and MSCEIT scores. For the 2007 cohort, the pattern was similar, with largely small and nonsignificant correlations. The only statistically significant positive correlation we found was between the year 4 total OSCE score and the MSCEIT score at matriculation (r = 0.35, P < .001).

Table 3.

Correlations Between Scores on the Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT) and on the Objective Structured Clinical Examinations (OSCEs), University of Ottawa Faculty of Medicine

In the 2006 cohort, 37 participants (26%) never received a clinical nomination, 39 (28%) received a single clinical nomination, and 65 (46%) received two or more nominations. In the 2007 cohort, 39 participants (28%) never received a nomination, 39 (28%) received a single nomination, and 63 (45%) received two or more nominations. We found only one significant correlation between MSCEIT scores and clinical nominations—between MSCEIT scores at matriculation and clinical nominations for the 2007 cohort (see Table 4). The total number of failures on any examination (i.e., written, OSCE, and preceptor) was 26/140 (19%) for the 2006 cohort and 14/141 (10%) for the 2007 cohort. We found no significant correlations in either cohort between MSCEIT scores and the presence or absence of a failure over four years of medical school (see Table 4).

Table 4.

Correlations Between Scores on the Mayer–Salovey–Caruso Emotional Intelligence Test (MSCEIT) and Failures or Clinical Nominations, University of Ottawa Faculty of Medicine^a

Discussion

In this study, we sought to determine whether EI measured at admissions predicted subsequent cognitive and noncognitive skills in medical school. We found no correlations between EI and written examination scores that reached statistical significance. However, we were not surprised by this finding—a purely cognitive test did not predict or correlate with EI. Although other studies have had similar findings, they used self-reporting tools to measure EI, potentially leading to different outcomes, whereas we used an abilities test.⁹

If EI is necessary for successful doctor–patient relationships, then students with higher EI scores potentially would have higher clinical skills scores, including measures of communication. Our findings did not support this prediction. We found no correlation between EI and OSCE scores or communication scores for the 2006 cohort. For the 2007 cohort, we found a weak to moderate correlation between EI at matriculation and year 4 OSCE scores. MSCEIT scores at interview, therefore, did not predict students’ OSCE performance. A previous study examined the correlation between EI and OSCE scores and found a modest correlation (in the range of 0.17–0.2) for communication skills.²⁰ Two key differences in their methodology may explain the difference in outcomes. In their study, the authors measured communication skills using a checklist consisting of three items with dichotomous variables and EI using the Trait Meta-Mood Scale and the Davis’ Interpersonal Reactivity Index, two self-report tools.²⁰ In our study, we used a communication rating scale that was more comprehensive with four items (interpersonal skills, interviewing skills, patient education, and response to emotional issues) and employed a rating scale rather than dichotomous variables. In addition, our EI assessment tool was an abilities test, suggested by some to be a better measure of the EI construct.¹¹

Finally, we examined the correlation between MSCEIT scores and clinical nominations, which are a measure of excellence in clinical performance. We would argue that such excellence would necessitate many skills that contribute to EI, such as communication skills, teamwork relationships, and professionalism. Yet, we found no significant correlations between MSCEIT scores and clinical nominations, except for a weak correlation between MSCEIT scores at matriculation and clinical nominations for the 2007 cohort.

Limitations

Our study has a number of limitations. First, self-selection may have biased our samples, as applicants and students were not required to participate. Also, admissions procedures and assessment tools vary in each medical school. As we conducted our study at a single institution, it may limit the generalizability of our findings. In addition, controversy exists regarding both the construct of EI and its predictive validity. Some authors have argued that the lack of conceptual clarity informing different models and measures of EI should lead to caution in considering EI as a measure of success.¹⁸ Yet, we conducted our study in a rigorous fashion—measuring EI in two separate cohorts, at two time points, with an abilities test, and compared with standard measures of cognitive and clinical skills.

Conclusions

In conclusion, we found that EI did not correlate with written examination scores or examination failures and that it correlated inconsistently with OSCE scores and nominations for excellence in clinical performance. Thus, EI, as measured by an abilities test at admissions, does not appear to reliably predict future academic performance during medical school. To further explore the use of EI at admissions, the next phase of our study will compare EI with other academic attributes, such as professionalism, need for examination deferral or leave of absence, and need for assistance from the student affairs or wellness office.

Acknowledgments: The authors wish to thank T. Rainville, D. Clary, and M-H Urro for their administrative assistance, as well as Luanne Waddell and Vikki McHugh from the Ottawa Exam Center for their invaluable support.