A Survey of the Complex General Surgical Oncology Fellowship Programs Regarding Applicant Selection and Rank

Abstract

Background.

There is significant demand for training in Complex General Surgical Oncology (CGSO) fellowships. Previous work has explored objective quantitative metrics of applicants that matriculated to CGSO fellowships; however, ambiguity remains concerning academic benchmarks and qualitative factors that impact matriculation.

Study Design.

A web-based survey was sent to each ACGME/SSO-approved CGSO fellowship training program. The survey was comprised of 24 questions in various forms, including dichotomous, ranked, and five-point Likert scale questions.

Results.

Twenty-nine of 30 program directors (97%) submitted complete survey responses, representing 64 of the 65 CGSO fellowship positions (99%) currently offered. Programs received a mean of 73 applications per cycle (range 50–125) and granted a mean of 26 interviews (range 2–45). Seventy-two percent of programs had an established benchmark for ABSITE score percentile before offering a candidate an interview, with 62% of those programs setting that benchmark above the 50^th percentile. The majority of programs also had established benchmarks for quantity of first author publications (mean: 2.3) and all publications of any authorship (mean: 4.4). An applicant’s interview was ranked as the most important factor in determining inclusion on the program’s rank list. The ability to work as part of a team, interpersonal interaction/communication abilities, and operative skills were rated as most important applicant characteristics, whereas an applicant’s personal statement was ranked as least important.

Conclusions.

After established academic benchmarks have been met, a multitude of factors influences ranking of applicants to the CGSO fellowship, most of which are assessed at the interview.

Application to fellowship for specialized post-residency education is becoming nearly ubiquitous amongst general surgery trainees.^1–3 Candidates interested in the complex surgical and multidisciplinary care of cancer patients may apply for fellowship in Complex General Surgical Oncology (CGSO), which established a separate board certification in 2011.^4,5 Although the number of Society of Surgical Oncology (SSO) recognized CGSO programs has grown from just eight in 1986 to 30 in 2018, the demand for training in CGSO continues to exceed the number of available fellowship positions by a significant margin.^4–6

Our group previously used data from the Electronic Residency Application System (ERAS) to describe the 2015 and 2016 CGSO fellowship applicant pools and determine objective applicant characteristics associated with matriculation into this competitive fellowship.⁴ We found that education at a U.S. allopathic medical school, training in a university-based surgical residency, and residency affiliation with a CGSO fellowship or National Cancer Institute designated Comprehensive Cancer Center (NCI-CCC) were associated with matriculation. We also demonstrated that matriculants produced more publications and achieved publication in higher impact journals than did nonmatriculants.⁴

While these findings were enlightening for surgical educators and the growing body of residents interested in CGSO fellowship training, a significant amount of data could not be captured in this analysis. Specifically, qualitative information, such as recommendation letters, interview performance, and perceived character traits, were not evaluable. Moreover, the use of academic benchmarks in evaluating applicants could not be garnished from ERAS, and data regarding American Board of Surgery In-Training Examination (ABSITE) scores were not available. We sought to evaluate the impact of these previously unknown, but potentially important, variables by surveying the program directors at each of the 30 ACGME (Accreditation Council for Graduate Medical Education)/SSO-accredited CGSO fellowship programs. Our specific goals were to discover established academic benchmarks, the applicant qualities of greatest importance to program directors, and factors associated with offering an interview and ranking an applicant highly.

METHODS

Data Acquisition

Approval was granted for the collection of deidentified survey data by the National Cancer Institute research compliance office. A web-based survey was sent to each CGSO fellowship recognized jointly by the ACGME and SSO inclusive of 30 programs as of September 2018.⁷ Either the program director or the associate program director for each program was invited to participate with the stipulation that only one response per program would be accepted. Surveys were available October 2018–November 2018 via a web-link in the distributed emails. No incentives were offered for participation.

Data were collected using the web-based survey collection tool, SurveyMonkey (SurveyMonkey Inc. Palo Alto, CA; www.surveymonkey.com). The survey was comprised of 24 questions in various forms, including dichotomous yes/no questions, open-ended questions, ranked selections, and five-point Likert scales, which ranged from “Not at All Important” to “Critical to further consideration”. An additional Likert scale evaluated the effects of certain applicant characteristics on likelihood of offering an interview and responses ranged from “Much less likely” to “Much more likely.” Question content addressed characteristics of the overall application process, factors in an application that affect decisions both before and after the interview, and finally questions regarding which factors are most important when making rank list decisions. Open-ended questions were used to identify the number of positions that each program offered as well as interview slots and average number of applications. Open-ended questions also were used to obtain specific details in response to previous answers. Ranked selections were used to evaluate the importance of listed factors (ABSITE score, interview performance, letters of recommendation, publications/research experience, and personal phone calls supportive of the applicant) compared directly to one another. The complete survey is available as Supplemental Fig. 1.

Response Analysis

Survey responses were deidentified but were linked based on how many positions a respondent stated that their CGSO program offered per year. Nominal and dichotomous data are presented as percentages of total program responses (prog) for each question. We have additionally displayed data on how many CGSO fellowships positions (pos) are represented by those particular respondents. Percentages represent number of programs responding except as otherwise indicated. Likert-scale questions were calculated using weighted answer choices as well, where the answer choice “Not at all important” was assigned a weight of one, and “Critical to further consideration” was weighted to a value of five. An identical calculation was used in evaluating the influence of applicant characteristics on likelihood of an interview with “Much less likely” assigned a value of one and “Much more likely” assigned a value of five. From this, a mean rating was calculated on aggregated responses and helped to establish the overall importance assigned to a factor. The mode of the available ratings was also calculated and displayed. A sixth answer choice of “Not Applicable” was included in all Likert scales and those responses marked in the affirmative were not included in analysis. Ranked responses were calculated using weighted answer choices as well; however, the highest ranked choice was assigned a value of one and the lowest a value of five to reflect ordinal rank.

RESULTS

General Information

Twenty-nine of 30 programs completed the survey (97% response rate), which accounted for 99% (N = 64/65 pos) of the available fellowship positions per year. Respondents reported receiving a mean of 73 applications (range: 50–125) and granting a mean of 26 interviews (range 2–45) per application cycle. This resulted in a mean interview rate of 37% (range 2–81%; Table 1). Seventy-two percent of respondents (N =21 prog, 51 pos) reported that multiple staff members work together to decide which applicants should be invited for an interview.

TABLE 1.

Survey and fellowship program characteristics

Response rate
CGSO fellowship programs (prog)	29/30 (97%)
CGSO fellowship positions (pos)	64/65 (99%)
Mean number of applications received per program	73 (range 50–125)
Mean number of applications received per position	49 (range 11–125)
Mean number of interviews granted	26 (range 2–45)
Mean interview rate (interviews granted/applications received)	37% (range 2–81%)

Factors Used to Evaluate CGSO Applicants for an Interview Invitation

Most programs reported using an applicant’s average ABSITE percentile scores (72%, N =21 prog, 42 pos), number of total publications (83%, N =24 prog, 49 pos), and number of first author publications (66%, N =19 prog, 42 pos) as benchmarks to meet prior to invitation for an interview (Fig. 1). Of the programs that required a minimum ABSITE score, 62% (N = 13 prog, 32 pos) required scores above the 50^th percentile (Fig. 1). All 21 programs requiring a benchmark reported that increasing ABSITE scores in successive years or consistently good performance can compensate for one previous suboptimal score. Additionally, those programs with benchmarks tended to place more importance an applicant’s most recent scores compared to those from earlier PGY residency levels or research years.

FIG. 1.

Factors used in evaluating CGSO applicants in which benchmark criteria for selection was defined. a Percentage of respondents that indicated an applicant’s average ABSITE score was used as an established benchmark before selecting candidates for an interview. Bar graphs display percentage of respondents answering yes who then further defined the following average score range as their benchmark. b Percentage of respondents that indicated an applicant’s quantity of published scientific manuscripts was used as an established benchmark before selecting candidates for an interview. This includes consideration of all manuscripts of any authorship position. Bar graphs display percentage of respondents answering yes who then further defined the following value as their benchmark. Mean benchmark, 4.4 (range 1–10) c Percentage of respondents that indicated an applicant’s quantity of first authorship published manuscripts was used as an established benchmark before selecting candidates for an interview. This includes consideration of manuscripts only when the applicant was first author. Bar graphs display percentage of respondents answering yes who then further defined the following value as their benchmark. Mean benchmark, 2.3 (range 1–5). pos displays the number of CGSO fellowship positions offered by the responding programs

Of the 24 programs that required a minimum number of publications of any authorship, 54% (N = 13 prog, 16 pos) required at least 5 publications before considering applicants for an interview (Fig. 1). The mean number of required publications by responding programs was 4.4. Of the programs that required a minimum number of first-author publications, 42% (N = 8 prog, 21 pos) required at least one first-author publication before considering applicants for an interview (Fig. 1). The mean number of required first-author publications by responding programs was 2.3. Eighty-three percent (N = 24 prog, 57 pos) of programs reported that the impact factor of journals that applicants publish in appropriately reflects the quality of their research work.

Using a Likert scale, this survey also investigated how particular factors influenced the likelihood of offering an interview to an applicant (Fig. 2). Spending time at the institution for clinical rotation (4.21 ± 0.68) or for research (4.21 ± 0.62) were the factors that influenced a decision most positively, whereas previous failed CGSO fellowship application (2.03 ± 0.87) made obtaining an interview less likely (Fig. 2). Eighty-six percent (N = 25 prog, 57 pos) of programs reported that when considering the overall application, an area of deficiency can be compensated by an above average performance in another area.

FIG. 2.

Applicant characteristics and influence on the likelihood of an interview invitation. Respondents were asked to evaluate each applicant factor on a scale of 1 Much less likely, 2 Less likely; 3 No effect; 4 More likely; and 5 Much more likely in regard to their influence on offering an interview. Mean aggregate ratings are displayed for each factor. Solid line displays the neutral rating

Factors Used to Rank Applicants after an Interview was Completed

Respondents were asked to consider 20 factors and use a Likert scale to determine how important each factor is to ranking an interviewed applicant highly (Fig. 3). The top factors that were important to CGSO programs were teamwork ability (mean 4.72 ± 0.45), interpersonal/communication skills (mean 4.59 ± 0.57), and interaction with other applicants/nonphysician staff (mean 4.32 ± 0.82), whereas the least important factor in rank determination was personal statement (mean 2.59 ± 1.18; Fig. 3).

FIG. 3.

Importance of applicant characteristics when determining a rank list after a completed interview. Respondents were asked to evaluate each applicant factor on a scale of 1 Not at all important; 2 Slightly important; 3 Moderately important; 4 Very important; and 5 Critical to further consideration in regard to ranking an applicant after the interview was completed. Mean aggregate ratings are displayed as bars for each factor. ⚫ represents the mode response of each factor

Respondents were asked to directly compare ABSITE score, interview performance, letters of recommendation (LoR), publications/research experience, and personal phone calls/communication from trusted surgeons in order of use when making the decision to rank an applicant highly (Table 2). Interview performance was most commonly ranked first as the most important, and publications/ research experience were most commonly ranked second. Factors in Table 2 are listed in descending order of their average ranking.

TABLE 2.

Rank scale of importance for listed factors when compared directly

	1st	2nd	3rd	4th	5th
Interview	18 (62%)	4 (14%)	5 (17%)	2 (7%)	0 (0%)
	38 pos	8 pos	14 pos	4 pos	0 pos
Publications and Research	4 (14%)	10 (34%)	7 (24%)	7 (24%)	0 (0%)
	10 pos	19 pos	27 pos	6 pos	0 pos
Letters of Recommendation	3 (10%)	8 (28%)	10 (34%)	6 (21%)	2 (7%)
	3 pos	14 pos	14 pos	28 pos	5 pos
Personal Phone Calls recommending applicant	4 (14%)	6 (21%)	4 (14%)	8 (28%)	7 (24%)
	13 pos	19 pos	6 pos	13 pos	13 pos
ABSITE scores	0 (0%)	1 (3%)	3 (10%)	5 (17%)	19 (66%)
	0 pos	4 pos	3 pos	10 pos	45 pos

Displays the numbers of responding programs ranking each factor in stated position when compared directly to the other listed factors. Numbers may not add to 100% as some respondents chose not to rank all factors

“pos” displays the number of CGSO fellowship positions offered by the responding programs

Bold displays the aggregate final ranking of each factor

DISCUSSION

CGSO fellowship positions continue to be among the most competitive and sought-after in surgical fellowship training. An average of 74 applications were submitted to each CGSO training program in 2018, and the current demand for positions greatly exceeds the number of positions offered.^4,8 In fact, 63% of applicants go unmatched to CGSO fellowships.⁴ As such, data regarding academic benchmarks and desired characteristics of applicants are likely to be highly valuable to the aspiring surgical oncologist. This study was designed with input from the leadership of multiple CGSO fellowship programs in the hopes of offering insight and transparency to a competitive application process. This study documents benchmarks for ABSITE percentile scores and quantities of both first-author publications and publications of any authorship by the majority of fellowship programs prior to offering an interview. Moreover, the perceived ability to work in a team and communication skills on interview day carried significant importance and influenced decisions regarding ranking.

ABSITE scores have been traditionally identified as one of the few objective markers in which to evaluate a surgical resident’s general knowledge prior to taking the board examinations.⁹ Despite this, an applicant’s average percentile score was ranked as the least important factor in ranking an applicant compared with the interview, LoR, publication record, and communication from other surgeons regarding an applicant. Our finding is similar to a study of Hepato-pancreato-biliary (HPB) fellowships, which found ABSITE in the bottom third of importance rating and ranked below interview, LoR, and research experience during applicant evaluation.¹⁰ Despite this, the average ABSITE percentile of applicants in their study was above the 75^th percentile. The authors suggested that, “As these qualities [high ABSITE scores] are so common amongst the matching fellows, the low rating may be a reflection of the limited usefulness in differentiating candidates and not of their importance to becoming a successful HPB surgeon.”¹⁰ Indeed, it is possible that a similar trend exists in the CGSO population. The majority of respondents in our study indicated a minimum score to be met prior to offering an interview and many of them set this score above the 50^th percentile. It would stand to reason then, that once a score benchmark is met and an applicant demonstrated to have above average surgical knowledge, ABSITE percentile loses its value in differentiating CGSO candidates. In addition, ERAS currently allows users to upload their own scores and does not require separate independent verification of scores from the American Board of Surgery before submitting an application, nor do they require a standard quantity of scores or PGY levels to be submitted.¹¹ More than three-quarters of programs in our study indicated that they would be in favor of a standardized system with verification, a modification that may give ABSITE percentiles more value in future applicant appraisals.

The majority of programs also had expected benchmarks for both number of first-author publications and number of publications of any authorship. With most respondents in our survey stating that they preferred applicants pursuing a career path in academics/research, these results are not surprising. The results also are consistent with our previous work using ERAS, which demonstrated applicants that matched into CGSO fellowship programs published significantly more manuscripts than those who did not (median 10 vs. 4.5) and were more likely to achieve publication in a high impact journal.⁴ Other work has demonstrated that the majority of general surgery residents author either one or zero manuscripts during residency emphasizing the point that academic productivity is exceptional in the CGSO applicant pool, yet is necessary to be consistent with the benchmarks that training programs expect.¹² It is therefore likely that dedicated research time during residency is prerequisite and should be a strong consideration for those residents who intend to enter the CGSO fellowship match.

Interview performance was identified as the most important factor in rank list determination. Program directors consistently expressed importance for applicant qualities that were discerned from interviewing: the ability to work in a team, communication skills, an applicant’s decision-making capabilities and interaction with others. Additionally, nearly three-quarters of programs reported that a supportive personal phone call from a surgical colleague made them more likely to rank the applicant highly. Finally, respondents also expressed that applicants who spent time at their institution in either a research or clinical capacity were more likely to be offered an interview. These findings indicate the significant importance of personal connection and in-person evaluation of an applicant when faculty are trying to find the “right fit” for their program. Conversely, disadvantage exists for having applied to CGSO fellowship previously and failing to secure a position: This was associated with less likelihood of being offered an interview in the current cycle and is again consistent with our previous data in which only one of the seven applicants who applied twice was able to eventually obtain a CGSO fellowship position.⁴

Finally, the above findings prompt questions about how to further improve the application process. As stated before, many respondents felt that standardization of ABSITE reporting could offer improvement, but open-ended questions also identified other opportunities. Multiple respondents expressed desire for letters of recommendation to follow a standardized form as many letters currently espouse exceptional abilities in the majority of applicants. This seemingly creates difficulty in the ability to distinguish which applicants are truly exceptional, an admittedly difficult task, but one that may be eased with a consistent format. Other respondents expressed desire for “better screening” of applicants before site visits for interviews: applicants could meet all program representatives in one place at the SSO annual symposium or discuss program details over video conference. Limiting the number of interviews would result in less time away from general surgery training and greatly reduced expenses. Finally, one respondent summarized the goal of studies, such as ours, as a way forward for improvement: “Each program could be more transparent about their criteria for eligibility into their program so that applicants know if they are competitive enough to apply.”

Although our study provides unprecedented insight into the application process for CGSO fellowships, it is not without limitations. Most prominently, this work is vulnerable to the inherent limitations and subjective nature of survey studies. Each individual respondent may have their own interpretation of a question or an internal rating scale of what they believe qualifies as important. This limits the utility of in-depth statistical analysis and the ability to derive statistically significant results when comparing responses. Additionally, despite the fact that this survey was performed in an anonymous fashion, survey respondents are still prone to the Hawthorne effect: respondents’ answers may be subconsciously or even consciously altered due to the fact that they know responses will eventually be reviewed by the investigators. Despite these subjective limitations, our study seeks to offer descriptive data while acknowledging that the application and interview process for CGSO fellowship itself is also partially a subjective exercise.

CONCLUSIONS

We believe that this study, in combination with our previous review of the objective ERAS data, offers powerful insight into the CGSO application process. A multitude of factors influences evaluation of applicants to the CGSO fellowship after established academic benchmarks have been met. Ability to work in a team, communication skills, academic achievements, and the capability to convey all these characteristics on interview day carry significant importance and influence decisions regarding applicant rank. Our analysis of CGSO fellowship application and ranking should serve in part to guide aspiring surgical oncologists and offer some transparency to future applicants.