Personality Testing May Identify Applicants Who Will Become Successful in General Surgery Residency

Abstract

Introduction

Identification of successful general surgical residents remains a challenging endeavor for program directors (PD) with a national attrition of approximately 20% per year. The Big 5 Personality traits and the Grit Scale have been extensively studied in many industries and certain traits are associated with professional or academic success. However, their utility in surgery resident selection is unknown.

Methods

We performed a retrospective review of all categorical surgery residents (n=34) at the University of Texas Medical Branch from 2015–2017. Current residents were classified into low performing (n=12) or non-low performing (n=22) based on residency performance and standardized test scores. Groups were assessed for differences in both conventional metrics used for selection and Big 5 and Grit Scores using bivariate analysis and Pearson’s correlation coefficient. Personality testing was administered to recent resident applicants (N=81). Applicants were ranked using conventional application information. We then examined the applicants’ personalities and their rank position with personality characteristics of non-low performing residents to determine if there was any correlation.

Results

The Big 5 personality test identified significantly higher extroversion, conscientiousness, and emotional stability scores in those residents classified as non-low performers. There was no significant difference in conventional metrics or in grit scores between non-low performers and low performers. Our final rank does not correlate well with personality traits of non-low performers.

Conclusions

The Big 5 test may prove to be a useful adjunct to the traditional residency application in identifying applicants who may become successful in general surgery residency.

1. Introduction

Over the past four decades, the attrition rate for general surgery residency has been higher than any other medical specialty, with 14-32% of residents leaving general surgery to pursue other specialties.^1-7 Although the surgeon lifestyle is the most commonly cited reason for leaving general surgery,⁷ the elimination of the pyramidal system and the implementation of duty hour regulations have failed to improve the attrition rate.¹ This suggests that lifestyle alone is insufficient to account for the high rate of attrition seen in general surgery programs.

Every year, Program Directors (PD) are challenged with identifying and selecting general surgery applicants most likely to succeed in their residency program(s). Among program directors, it is considered one of the more daunting tasks.⁸ Traditionally, standardized tests, letters of recommendation, honorary society membership induction (AOA), and short, unstructured interviews have provided the basis for which applicants are selected. However, multiple studies have shown this process to be ineffective.^9-11 If we were able to identify the best-fit applicants for general surgery programs, attrition rates should decrease.

The desire to identify the best-fit applicants is not unique to surgery: other high stress careers such as aviation have studied the use of personality tests in applicant selection.^12,13 The Big Five factor structure of personality allows for the standardization and comparative analysis of personality traits by categorizing personality into 5 traits: extroversion, agreeableness, conscientiousness, emotional stability, and openness (Table 1).¹⁴ Several studies have shown a relationship between specific personality characteristics and workplace performance or academic success.^15-18 Prior investigations using the Big Five Inventory (BFI) in surgery have predominantly focused on identifying the characteristics associated with surgeons, but have not sought to elucidate the difference between the personalities of successful and unsuccessful residents.^{15, 19-23} In addition, grit is a non-cognitive trait described in psychology literature as perseverance and passion for long-term goals.²⁴ Higher grit tends to be associated with individuals who will complete set goals despite obstacles and setbacks.²⁵ Grit was found to be a better predictor than either the Big 5 or demographic variables when evaluating retention in multiple settings including the military, workplace sales, high school, and marriage.^{24, 26} Furthermore, there is initial evidence that grit scores may be predictive of residents at risk of attrition in general surgery.^24,27,28

Table 1.

The Big-5 Inventory Domains

Personality Trait		Descriptive Terms
Extroversion	The tendency to seek out activity and the company of others—People with high levels of extraversion tend to be assertive and often seek leadership positions.	Gregarious, Assertive, Energetic, Adventurous, Enthusiastic, Outgoing
Agreeableness	How oriented towards community someone is—People with high levels of agreeableness tend to exhibit greater levels of altruism, trust, and modesty.	Cooperative, trustworthy Sympathetic
Conscientiousness	The tendency to regulate and control impulses in order to reach our goals—People with high levels of conscientiousness tend to delay gratification and are good at prioritizing tasks.	Organized, Self-discipline, Deliberate, Efficient, Goal-directed
Emotional Stability (Neuroticism)	How prone someone is to experience negative emotions —People with high levels of neuroticism tend to be anxious, tense, and experience more depression. Lower levels are associated with higher emotional stability.	Self-confident, Even tempered
Openness	Open to experience —People with high levels of openness tend to be more creative, with multiple different interests, always seeking to learn new things	Curious, Imaginative, Artistic, Unconventional; Inspirational, Seek knowledge

However, the data are lacking on how these tests may be used to identify and select successful surgery residency applicants. We hypothesized that personality testing would provide valuable additional information to help identify candidates best suited for surgery residency when compared with conventional application information alone.

2. Methods

2.1. Study Design and Sample

2.1a. Current Surgical Residents

In 2015, our department sought to improve the applicant selection process. As such, we implemented questionnaires as part of our residency application process to attempt to identify applicants who would fit best in our program. We found that the questionnaires improved our overall application process and sought to report our findings. We discussed our intentions with our institutional review board (IRB). Based on our discussions, our IRB deemed this study to be a quality improvement (QI) project aimed at improving our applicant selection process and not human subjects research. Next, a retrospective review was performed of all categorical general surgery residents (n=34) at the University of Texas Medical Branch (UTMB) from 2015-2017. The personality traits of current residents were evaluated using a combination of an application score (used to measure conventional metrics) with BFI testing and Grit scores. The application score was calculated for each resident by a single observer, the program director. Residents were then asked to complete the Ten Item Personality Inventory (TIPI) and Grit tests. Residents were grouped as low-performing or non-low performing based on remediation, attrition status, and Accreditation Council for Graduate Medical Education (ACGME) Milestones (an evaluation tool used by accredited residency or fellowship programs to assess residents in multiple domains with regard to their competency²⁹). Low performing residents (n=12) were determined to have at least one or more of the following: ACGME Milestones less than the 25^th percentile in one or more of the six domains, postgraduate year (PGY) remediation, or leaving the program (n=12). Non-low performing residents were those not in any of the aforementioned categories (n=22). The application score, TIPI, and Grit scores were then compared between low and non-low performing residents.

2.1b. General Surgery Residency Applicants

The 2016-2017 year UTMB general surgery residency applicants were asked to complete the TIPI and Grit tests. Participation was entirely voluntary and this information was not used to determine applicant ranking. We had a 100% response rate, with 81 of the 81 applicants completing the questionnaires. Each applicant was ranked based on traditional ranking measures (including the Electronic Residency Application Service (ERAS) content and interviews). After identifying non-low performing resident personality traits, we examined residency applicants with similar traits to determine if we ranked them highly on our rank list.

2.2. Measures

The Application Score

The application score was created to capture the pre-interview data utilizing information available from the ERAS application. Each person was given 0-3 points for type of medical school attended, United States Medical Licensing Examination (USMLE) Step I and II, class standing (i.e. rank), the caliber of their letters of recommendation (LOR), the content within the LOR (i.e. knowledge, problemsolving ability, work ethic, general demeanor and hand skills), content of the Dean’s letter, gaps in training, receipt of awards, and extracurricular activities, with a total possible score of 27 (Figure 1).

Figure 1.

Scoring Rubric for Application Score

*USFMG= United States Citizen Foreign Medical Graduate

**FFMG= Non US Citizen, Foreign Medical Graduate

+Letters = receives one point each if obtained a letter from a person with the above described title.

++ Receives one point if letter mentioned one of the above characteristics.

N/A= Not applicable

The Big 5

The Ten-Item Personality Inventory (TIPI) is a validated, shorter form of the standard 44- question Big-Five Inventory (BFI), which is used to describe the five personality traits (previously described in Table 1).^{15, 18, 30} The TIPI utilizes a total of 10 questions, two questions pertaining to each personality trait, the answers of which are averaged to calculate the personality trait score.³¹ Each question is answered on a scale of 1 (Disagree Strongly) to 7 (Agree Strongly) with the maximum score being 7 for each given personality trait. For example, those who have higher scores related to calmness are more emotionally stable. The TIPI has been shown to have both internal consistency as well as high correlation with other five-factor model instruments¹⁴ but is shorter and easier to complete.

The Grit Scale

The original Grit Scale is a 12- item point scale derived using a 5-point Likert scale for each question. The maximum score is 5 points with 5 being extremely gritty and 1 being not at all gritty. Thus, those with higher scores have more grit (i.e. 4 vs. 2). The derivation of these scales are described elsewhere.^{24, 25}

2.3. Statistical Analysis

Fisher’s exact test was used for descriptive statistics: race, gender, and medical school graduate status (U.S. vs. foreign medical graduate). The TIPI questionnaire and the Grit were analyzed for differences between residents in the low and non-low performing groups using student’s t-test. Next, low performers were further separated into two groups: residents who left the program (low performer-attrition) and those who remained but had low performance (low performer-current resident). Differences between non-low performing residents (n=22), low performing-current residents (n=7 of 12), and low performing-attrition residents (n=5 of 12) were assessed using a one-way ANOVA with multiple comparisons. For all tests, statistical significance set at p<0.05. Pearson’s correlation coefficient was used to evaluate the association between each of the Big Five trait categories, Grit, and the application score with regard to where applicants fell on the rank list. SAS software, Version 9.4 (SAS Institute Inc., Cary, NC, USA) was used for statistical analysis.

3. Results

Demographics

All categorical residents enrolled in our program, with at least one full year of clinical experience and milestone evaluation at the time of the study, were eligible to participate (N= 34). Thirty-three of 34 residents completed the TIPI (97% response rate), and 28 of 34 completed the Grit score (79% response rate). We evaluated the relationship between gender, race, medical school type (foreign medical graduate vs. U.S. graduate), average USMLE Score, and average American Board of Surgery In-Training Examination (ABSITE) scores with non-low (n=22) and low performance (n=12) status (Table 2). There was a significant difference in medical school characteristics between non-low and low performers, with foreign medical graduates (FMG) significantly more likely to be non-low performers (FMG 100% non-low performers vs. 0% low performers, compared to US graduates 56% non-low performers, 44% low performers, p=0.03). There were no statistically significant differences between the non-low performer and low performer groups in the following demographic data: sex (n=19 males and n=15 females, p=0.71) and race (Caucasian (n=14), African-American (n=6), Hispanic (n=6), Asian (n=6), Middle Eastern/North African (n=2), p=0.47). There were no significant differences (average ± SEM) between non-low and low performers on USMLE Step I (230.71 ± 3.03 vs. 230.31 ± 4.03, p=0.46) or Step II (237.24 ± 2.83 vs. 238.23 ± 4.02, respectively, p = 0.14) scores, or on average ABSITE (44.43±4.65 vs. 40.47 ±7.07, p=0.21) score.

Table 2.

Demographics, Conventional ERAS Data, and Test Scores

	Non-low Performer N (%)	Low Performer N (%)	p-value
Race
Caucasian	8 (57%)	6 (43%)	0.57
African American	3 (50%)	3 (50%)
Hispanic	4 (67%)	2 (33%)
Asian	5 (83%)	1 (17%)
Middle Eastern/ North African	2 (100%)	0 (0%)
Gender
Male	13 (68%)	6 (32%)	0.61
Female	9 (60%)	6 (40%)
Type of Medical School
U.S. Medical School	15 (56%)	12 (44%)	0.03
Foreign Medical School	7 (100%)	0 (0%)
USMLE Scores (Average)
Step I	230.71 ± 3.03	230.31 ± 4.03	0.46
Step II	237.24 ± 2.83	238.23 ± 4.02	0.14
ABSITE (Average)	44.43±4.65	40.47 ±7.07	0.21

Conventional Application Score

The conventional application score, showed no statistically significant differences (average ± SEM) between non-low and low performers (13.37 ± 0.66 vs. 15.33 ± 1.03, p= 0.12) (Table 3). The average applicant score was higher for those in the low performer-attrition group, but not statistically significant (low performer-attrition vs. non-low performer, 16 ± 1.64 vs. 13.28 ± 0.71, p = 0.25; low performer-attrition [n=5] vs. low performer-current resident [n=7], (16 ± 1.64 vs. 14.86 ±1.41, p=0.83) (Table 4).

Table 3.

Application Score, Big 5, and Grit among Non-low and Low Performers

	Non-low Performer (average ± SEM)	Low Performer (average ± SEM)	p-value
Application Score	13.37 ± 0.66	15.33 ± 1.03	0.12
Big 5
Extroversion*	4.77± 0.34	3.77± 0.40	0.03*
Agreeableness	5.07 ± 0.22	5.09 ± 0.34	0.48
Conscientiousness*	6.09 ± 0.14	5.55± 0.32	0.03*
Emotional Stability*	5.50± 0.25	4.41± 0.45	0.02*
Openness	5.28 ± 0.24	5.32± 0.30	0.45
Grit	4.01 ± 0.12	3.65± 0.19	0.06

^*Indicates statistical significance

Table 4.

Multiple Comparison ANOVA with non-low performers, low performer-current residents, and low performer-attrition.

			Non-low Performer	Attrition (Low-Performer)	Current Resident (Low Performer)	Attrition (Low-Performer)	Non-low Performer	Current Resident (Low Performer)
Application Score		Average	13.28 ±0.71	16 ± 1.64	14.86 ±1.41	16 ± 1.64	13.28 ±0.71	14.86 ±1.41
		p-value	0.25		0.83		0.54
Big 5	Extroversion	Average	5.1 ± 0.33	4.2 ± 0.54	3.42 ± 0.57	4.2 ± 0.54	5.1 ± 0.33	3.42 ± 0.57
		p-value	0.44		0.65		0.046*
	Agreeableness	Average	4.95 ± 0.21	4.6 ± 0.64	5.5 ± 0.29	4.6 ± 0.64	4.95 ± 0.21	5.5 ± 0.29
		p-value	0.76		0.31		0.47
	Conscientiousness	Average	6.05 ± 0.14	5.4 ± 0.46	5.67 ± 0.48	5.4 ± 0.46	6.05 ± 0.14	5.67 ± 0.46
		p-value	0.27		0.85		0.58
	Emotional Stability	Average	5.52 ± 0.26	3.4 ± 0.58	5.25 ± 0.44	3.4 ± 0.58	5.52 ± 0.26	5.25 ± 0.44
		p-value	0.003*		0.038*		0.87
	Openness	Average	5.24 ± 0.25	5 ± 0.45	5.58 ± 0.40	5 ± 0.45	5.24 ± 0.25	5.58 ± 0.40
		p-value	0.9		0.66		0.77
Grit		Average	4.06 ± 0.08	3.46 ± 0.24	3.8 ± 0.29	3.46 ± 0.24	4.06 ± 0.08	3.8 ± 0.29
		p-value	0.04*		0.45		0.44

Big 5 Personality Traits

There was a statistically significant difference (average ± SEM) between non-low performers and low performers in extroversion (4.83 ± 0.33 vs. 3.80 ± 0.40, p= 0.03); conscientiousness (6.10 ± 0.13 vs. 5.5 ± 0.32, p=0.04), and emotional stability (5.5 ± 0.24 vs. 4.41 ± 0.30, p=0.02). There was no statistically significance difference between non-low and low performers for openness (5.28 ± 0.23 vs. 5.31 ± 0.30, p=0.46) and agreeableness (5.0 ± 0.22 vs. 5.09 ± 0.34, p=0.41) (Table 3). When comparing residents in the low performer-attrition cohort versus those in the non-low or low performer-current resident cohorts, we found consistently lower emotional stability scores in the residents who left the program (low performer-attrition vs. non-low performers: 3.4 ± 0.58 vs 5.52 ±0.26, p <0.003; low performer-attrition vs. low performer-current residents: 3.4 ± 0.58 vs. 5.25 ± 0.44, p< 0.04) (Table 4).

Grit

There was no statistically significant difference between non-low performers and low performers in grit scores (4.01 ± 0.12 vs. 3.65 ± 0.19, p=0.06) (Table 3). However, when evaluating residents who left the program versus non-low performing residents, we found that they had statistically lower grit scores (low performer-attrition: 3.46 vs. non-low performers: 4.06, p < 0.04) (Table 4).

Resident Applicants

Pearson’s Correlation Coefficient demonstrated that the conventional application score (i.e. ERAS, residency interviews, etc.) was highly correlated with an applicants’ rank position (r = −0.89) (Table 5). There was no correlation between the Big five (Extroversion [r =0.072], Agreeableness [0.17], Conscientiousness [r= −0.30], Emotional Stability [r = −0.03], Openness [0.16]) or Grit (r= −0.01) with applicant rank (Table 5).

Table 5.

Pearson’s Correlation between Rank and Applicant Score, Big 5 Characteristics, and Grit

	Application Score	Big 5					Grit
		Extroversion	Agreeableness	Conscientiousness	Emotional Stability (Neuroticism)	Openness
Pearson’s Correlation Coefficient	−0.89	0.072	0.17	−0.30	−0.03	0.16	−0.01

Discussion

The results of this study demonstrate that components of the application score (USMLE score, letters of recommendation, etc.) are no different between non-low performing and low performing residents. Non-low performers were noted to have scored significantly higher on extroversion, conscientiousness, and emotional stability than those ranked in the lower performer cohort. While the grit scores between non-low performers and low did not reach statistical significance, this is likely due to small numbers in each cohort. In addition, we found that residents in the low performer-attrition attrition group exhibited less emotional stability than non-low performer and the low performer-current resident groups, and lower grit scores than non-low performers. Among our general surgery applicants, only conventional application scores correlated with their rank position. There was no correlation between those personality traits of our non-low performing residents and the applicants’ rank position for our general surgery program.

The use of standardized test scores, letters of recommendation, and medical school metrics for residency selection has been the gold standard for many years. Our finding of no difference between non-low performing and low performing residents in standard ERAS application metrics adds to the growing body of literature that suggests that traditional selection methods are inadequate.^9-11 Although many studies have evaluated the characteristics of surgeons versus trainees, students, and non-surgeon physicians,^{19, 20, 23} our study is the first to evaluate general surgery residents using the TIPI and Grit tests with respect to residency performance. The results of this study are consistent with the current literature and expand upon current knowledge regarding general surgery residents and their personality characteristics.^{19, 32, 33}

Similar to the study by McGeevy et al in 2002 who found surgery residents tend to score higher in extroversion, openness, and conscientiousness, and lower in neuroticism (i.e. higher in emotional stability)²³, in our study, non-low performing residents scored significantly higher in emotional stability, conscientiousness, and extroversion compared to low performers but not in openness. We believe this finding suggests that while emotional stability, consciousness, and extroversion may be critical to identifying non-low performers, openness may not. In the last sixteen years, there have been significant changes in surgical training including the ACGME duty hour restrictions, as well as the requirement that an attending be present for the critical portions of the surgical case. Such changes may explain the difference in openness scores between earlier studies and ours.

The higher levels of conscientiousness in non-low performers is not surprising considering that goal-directed behavior without impulsiveness is necessary when planning and conducting high stake activities, such as operations. For this reason, conscientiousness as a hallmark characteristic among surgery residents is well documented throughout the literature, even when compared with nonsurgical residents.^{19, 20, 23}

Emotional stability is also a valuable attribute to possess. With higher levels of stress for the average general surgery resident compared to community norms, exhibiting the ability to remain even-tempered without becoming rattled would seem to be an ideal characteristic of a successful resident ²⁰, who is managing critically ill patients in the intensive care unit (ICU), operating room (OR) or emergency room (ER).

As a group, surgeons are perceived and reported to have higher extroversion than other medical specialties.^{34, 35} Exhibiting the ability to communicate effectively and lead a team is important in surgery. Surgeons are typically the leaders within multidisciplinary teams, therefore an extroverted leader could be more beneficial to the success of a team. Thus, it is not surprising that those in our non-low performing group are more extroverted when compared to the low performing group.

Although ours is the first to look at Big 5 personality scores and residency performance in surgery, other investigators have utilized other inventories to identify the surgical personality. Foster et al utilized the World of Work Inventory Online (WOWI Online), a multidimensional personality assessment, to evaluate differences in non-low versus low performing residents and attending physicians.³³ They discovered that the personality characteristics of non-low performing residents and faculty surgeons were the same, while they both differed from low performing residents. Those surgeons that were identified as non-low performing scored significantly higher on 5 subscales: versatility, gregariousness, influencing, self-control, and rigor. Although using a different inventory, these characteristics closely align with those seen in the Big 5.

The identification of those candidates and residents at greatest risk for attrition may be useful not only in the selection process but also in early intervention and resident retention. More recent literature suggests that behavioral traits may be enhanced with the use of mental skills training^36-38. Furthermore, there is evidence that some interventions such as mental skills training may also improve resident performance in surgery^{36, 39, 40} The early identification of at-risk residents may provide an opportunity for early intervention. Although there is limited data specifically evaluating these techniques in surgery, it does merit consideration. Personality or performance issues are not new in surgery, and should not preclude one from receiving the necessary resources to aid in achieving success in a surgery residency program. The implications of personality are not limited to resident selection or fit, but also relate to future job satisfaction and burnout. The traits of extroversion, conscientiousness, and emotional stability have all been linked to job satisfaction among trauma surgeons and other professionals.^{15, 41} The presence of favorable personality traits in applicants selected for residency programs is important considering that residency has been identified as a peak time for burnout⁴² and that lacking extroversion and emotional stability has been associated with an increased risk of burnout in surgeons. It is difficult characterize burnout consistently as the pattern differs based on the level of training,^{42, 43} but its presence is overwhelmingly reported among those in surgical fields, occurring in up to 42% of surgeons and 69% of residents.^{44, 45} Much like identifying those at risk for low performance, the identification of those at risk for burnout offers early opportunity for intervention.

The use of grit as a marker for success has been studied across multiple fields, and is a better predictor of retention than both the Big 5 and demographic variables across multiple settings.²⁶ We found that the grit scores of the low performing-attrition group were lower than non-low performing residents, which is consistent with previously reported data.^{27, 28} In addition, our data suggest that grit scores may be helpful in differentiating non-low performing and low performing residents as a whole. However, this was a trend, which failed to reach statistical significance. Despite this, the importance of grit is readily evident: those with more grit are able to overcome obstacles and maintain focus on the long-term goal;²⁴i.e., the willingness to endure long work-hours, perplexing patient management, demanding operations, and personal life balance in order to become a surgeon. ^{27, 28}

Although the literature supports the idea that personality testing provides valuable information, there has been no report of general surgery residency programs or fellowships that have used personality testing to aid in the ranking of their prospective applicants. Ours is the first study to suggest that it may be useful in predicting performance success in general surgery residents. Our findings support the idea that non-low performing residents possess unique, measurable personality traits and that the current, commonly used evaluation methodology does not identify them. As a result, personality testing should be considered as a valuable adjunct in the evaluation of surgery residency applicants.

This study was subject to some limitations. Our data is representative of a single institution. We also excluded any resident, including our intern class, who had not completed one year of residency at our institution; these residents were excluded due to insufficient data by which to categorize them accurately as low or non-low performers. In addition, our rubric, adapted from the University of Illinois-Chicago, used for the conventional application score may not be generalizable at all general surgery programs. However, it is readily adoptable and may be modified, as necessary, by any residency program. Despite our findings and level of consistency of the data among similar studies, they may not be representative of general surgery residents in other programs. Although our response rate was high (97 %) for TIPI scale and (79%) for the Grit scale, this study is underpowered, limiting the generalizability of this data as well as the potential for subgroup analysis. Additional studies with more subjects are necessary to confirm these preliminary findings.

Conclusion

Psychological tests of personality, such as the TIPI and the Grit, are readily available adjuncts to the conventional interview process. Using these supplementary tests may aid program directors in identifying those individuals psychologically equipped to endure the rigorous training environment of surgery residency; leading to a better match between applicants and programs as well as reduced resident attrition.