Performance in Irish Selection and Future Performance in Surgical Training

This study attempts to determine the association between measured candidate factors at the time of an Irish selection and assessment outcomes in surgical training.

Key Points

Question

Is there an association between measured candidate factors at selection and future performance in surgical training?

Findings

A cohort study was conducted including 303 successful applicants to the Royal College of Surgeons in Ireland Core Surgical Training program. Performance at the time of selection, measured across technical and nontechnical domains in a multimodal fashion, was significantly associated with future performance in the workplace and in simulation-based assessments.

Meaning

In this study, a multimodal selection framework could be used to recruit surgical trainees in a standardized and transparent manner, conforming to a predictive validity paradigm.

Abstract

Importance

Selection processes for surgical training should aim to identify candidates who will become competent independent practitioners and should aspire to high standards of reliability and validity.

Objective

To determine the association between measured candidate factors at the time of an Irish selection and assessment outcomes in surgical training, examined via rate of progression to Higher Specialist Training (HST), attrition rates, and performance as assessed through a multimodal framework of workplace-based and simulation-based assessments.

Design, Setting, and Participants

This retrospective observational cohort study included data from all successful applicants to the Royal College of Surgeons in Ireland (RCSI) national Core Surgical Training (CST) program. Participants included all trainees recruited to dedicated postgraduate surgical training from 2016 to 2020. These data were analyzed from July 11, 2016, through July 10, 2022.

Exposures

Selection decisions were based on a composite score that was derived from technical aptitude assessments, undergraduate academic performance, and a 4-station multiple mini-interview.

Main outcomes and measures

Assessment data, attrition rates, and rates of progression to HST were recorded for each trainee. CST performance was assessed using workplace-based and simulation-based technical and nontechnical skill assessments. Potential associations between selection and assessment measures were explored using Pearson correlation, logistic regression, and multiple linear-regression analyses.

Results

Data were available for 303 trainees. Composite scores were positively associated with progression to HST (odds ratio [OR], 1.09; 95% CI, 1.05-1.13). There was a weak positive correlation, ranging from 0.23 to 0.34, between scores and performance across all CST assessments. Multivariable linear regression analysis showed technical aptitude scores at application were associated with future operative performance assessment scores, both in the workplace (β = 0.31; 95% CI, 0.14-0.48) and simulated environments (β = 0.57; 95% CI, 0.33-0.81). There was evidence that the interpersonal skills interview station was associated with future performance in simulated communication skill assessments (β = 0.55; 95% CI, 0.22-0.87).

Conclusions and Relevance

In this study, performance at the time of Irish national selection, measured across technical and nontechnical domains in a multimodal fashion, was associated with future performance in the workplace and in simulated environments. Future studies will be required to explore the consequential validity of selection, including potential unintended effects of selection and ranking on candidate performance.

Introduction

Obtaining a postgraduate surgical training position is a competitive process worldwide.^1,2 Selection processes vary widely.³ Common criteria informing selection include letters of recommendation,^4,5,6 prior academic performance,⁷ results in national licensing examinations¹ and professional examinations,⁸ student performance evaluations,⁹ research output,^4,5 prior experience,¹⁰ personal statements,¹¹ and performance at interview.^5,9,10,12 Selection can be considered as a high-stakes assessment process and should conform to high standards of validity.¹³ In particular, processes should follow a predictive paradigm, aiming to identify candidates who will become competent medical professionals.¹⁴ Industry evaluations of selection processes further describe return on investment across 3 domains; administrative efficiency, the future performance of selected candidates, and subsequent attrition rates.¹⁵ Screening processes are costly¹ and attrition rates from general surgery programs can be as high as 30%.¹⁶ Deficits in medical knowledge can lead to delayed progression¹⁷ and shortfalls in interpersonal skills or professionalism may be even more challenging to rectify.¹⁸ Of further consideration for surgical specialties is technical skill or aptitude.¹⁹ A multifactorial approach to trainee selection could theoretically select for high-performing residents.

While a number of studies have examined factors associated with success at an undergraduate level,^20,21 the validity of selection metrics in postgraduate training are less well explored.^13,22 Predicting the future performance of surgical trainees is challenging.¹³ A 2013 meta-analysis of selection strategies across all medical and surgical specialties showed minimal or no correlation between interviews, reference letters, or deans’ letters and future resident performance.²³ Performance in the US Medical Licensing Examination correlates weakly to moderately with future in-training examination performance, with an observed weaker correlation between US Medical Licensing Examination scores and performance in surgical specialties.²⁴ While performance in the Membership of the Royal College of Surgeons examination in the United Kingdom and Ireland is associated positively with likelihood of selection onto surgical training programs, further predictive validity evidence is lacking.²⁵ Data-driven approaches to trainee selection are therefore increasingly being explored.^{15,26,27,28,29}

The objective of this study was to determine the potential association between measured candidate factors at the time of a multimodal national selection process for surgical trainees in Ireland and future success in early surgical training, examined via rate of progression to higher specialist training, performance as assessed through workplace-based and simulation-based assessments, and attrition rates.

Methods

This study is reported according to the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines³⁰ and was approved by the ethics committee of the University of Medicine and Health Sciences at the Royal College of Surgeons in Ireland. Informed consent for the collection and processing of training data were obtained from trainees at the time of recruitment.

Setting and Participants

Selection into Core Surgical Training (CST) (the first 2 years of dedicated, undifferentiated postgraduate surgical training) is a centralized process conducted by the Royal College of Surgeons in Ireland (RCSI). Candidates undergo psychomotor aptitude testing (Table 1) and a 4-station multiple mini-interview evaluating clinical judgment, interpersonal skills, professional development, and suitability for specialist training. Scores from these structured interview stations and aptitude assessments are combined with a score awarded for the candidate’s undergraduate medical degree centile to produce a total score (Table 1).

Table 1. Analyzed Selection Metrics and Performance Outcomes.

Domain	Description	Method/measures
Selection domain (relative scoring %)
Centile	Centile score awarded in a candidate’s undergraduate medical degree	Prescored, scaled to 15-point measure
Surgical aptitude (15)	Psychomotor skills, visuospatial ability, perception	3 Equally weighted validated assessments of baseline aptitude, PicSOr, visuospatial aptitude, grooved pegboard
Clinical judgment (15)	2 Structured sequential clinical scenarios	Structured interview
Interpersonal skills (15)	Communication, teamwork and leadership; crisis management, negotiation, and conflict resolution	Structured interview
Professional development (15)	Clinical research projects, attendance at meetings and courses, audit projects, and teaching activities	Structured interview
Suitability for specialist training (25)	References, motivation and drive; knowledge of specialty, time and stress management, work ethic, and professionalism	Structured interview
Assessment domain
Attrition	Withdrawal from the CST program	Binary outcome measure
Progression	Successful progression to HST	Binary outcome measure
Operative performance	SSAOP assessment scores	Mean scores awarded across all submitted SSAOP assessments over 2 y of training, scored using global (not procedure specific) checklists on a scale from 0-150
Simulated operative performance	OSS assessment scores	Mean scores awarded across 8 technical skill OSCE stations conducted in annual sessions (4 stations at each assessment), scored using predefined procedure or task-specific checklists
Nontechnical skill performance	Communication skill assessment scores	Mean scores awarded across 8 nontechnical skill OSCE stations conducted in annual sessions, scored using predefined scenario-specific checklists and the Calgary Cambridge assessment tool31
Clinical performance	SCA assessment scores	Mean scores awarded across all submitted SCA assessments, scored using generic (not diagnosis-specific) checklists from 0-100
Operative experience	eLogbook scores	Scores calculated algorithmically, incorporating total procedural numbers, case complexity, and level of autonomy. Mean standardized scores were calculated, awarded across 2 rotations in year 1 (scored 0-25) and 1 rotation in year 2 (scored 0-50).
Trainee assessment reports	Trainer evaluations of performance in the workplace	Mean standardized scores awarded across 3 rotations over 2 y, scored from 0-50 across 2 rotations in year 1 and 0-100 in year 2

Abbreviations: CST, Core Surgical Training; eLogbook, electronic logbook; HST, Higher Specialist Training; OSCE, Objective Structured Clinical Examination; OSS, operative surgical skill; PicSOr, Pictorial Surface Orientation testing; SCA, structured clinical assessment; SSAOP, Supervised Structured Assessment of Operative Performance.

Successful candidates undergo 2 years of CST. Trainees complete 2 rotations of different specialties in their first year of training, followed by a further year in their intended specialty, usually in a different training location. Performance during this time is assessed through workplace-based assessments of operative and clinical performance, simulated assessments of technical and nontechnical skills, trainer evaluation reports, and operative experience—recorded and stored using an electronic logbook. On completion of CST, candidates are eligible to apply for a Higher Specialist Training (HST) position in their surgical specialty of interest. Selection at this stage is again a competitive process^32,33 and is informed by the following: (1) CST Trainer Assessment Reports, (2) electronic logbook scores, (3) simulation-based communication skill and objective surgical skill assessments, (4) performance in part B of the Intercollegiate Membership of the Royal College of Surgeons examination, and (5) performance in a further standardized multi-station interview. Information on the progression pathway is available.³⁴

A retrospective analysis of prospectively collected application data from successful applicants to RCSI’s CST program from 2016 through 2020 inclusive was performed. Assessment data, attrition rates, and progression to HST were recorded until July 2022 inclusive.

Variables, Data Sources, and Measurement

Potential explanatory variables explored are outlined in Table 1. Psychomotor aptitude testing was conducted using 3 validated assessments: Pictorial Surface Orientation testing,^{35,36,37,38,39,40,41,42} digitized visuospatial assessments,^43,44 and dexterity testing using a grooved pegboard.⁴⁵ Pictorial Surface Orientation testing requires candidates to orientate a spinning arrowhead so that it rests perpendicular to a displayed cube face; scores are calculated based on speed and accuracy with Pearson product moment correlation coefficients used to compare the orientation chosen by the candidate with the theoretically correct orientation.³⁵ Four visuospatial aptitude tests are used at selection, derived from a Kit of Factor Referenced Cognitive Tests developed by Ekstrom et al⁴⁴: surface development, map planning, card rotation, and cube comparison.^43,44 Manual dexterity is assessed using a grooved pegboard, requiring candidates to orientate pegs until they slot into keyhole-shaped openings; scores are awarded for speed with deductions accrued for dropped pegs.⁴⁶

Outcome variables are summarized in Table 1. Workplace-based assessments of operative performance are scored using generic checklists (supervised structured assessment of operative skill). Workplace-based assessments of clinical performance—for example, history and physical examination in the outpatient setting—are also conducted (structured clinical assessments). Global assessments of overall performance in a training post are completed at 6-monthly intervals (trainee assessment reports). A 4-station simulated skills assessment is conducted at the end of each year of training (operative surgical skill assessment). Communication skills are also assessed at these time points across 4 structured stations. Due to the COVID-19 pandemic, trainees recruited in 2019 and 2020 (n = 124) using predefined scenario-specific checklists and the Calgary Cambridge assessment tool³¹ had disruptions to their annual assessments and undertook 1 sitting of the operative surgical skill and communication skill assessments only. Example assessment forms are available in the eAppendix in Supplement 1. Scoring of electronic logbooks as a proxy measure of operative experience has been previously described,⁴⁷ but briefly is calculated based on the number of procedures performed, the degree of trainee involvement, and the complexity of the logged procedures.

Bias and Sample Size

Potential explanatory variables were assessed prior to outcome variables. Assessors of simulation-based assessments and workplace-based performance were blinded as to the scores achieved by trainees at selection. Workplace-based assessors are registered trainers with the Royal College of Surgeons in Ireland, registered on the relevant specialist division of the Irish Medical Council, registered on a professional competence (continuous professional development) program, and practicing at a consultant (attending) surgeon level. Assessors for simulation-based assessments undergo rater training with standard setting conducted prior to each assessment session. Based on a minimum of 193 participants, this study has adequate power to detect a weak correlation (0.2 or more) between independent variables and measured outcome variables.⁴⁸

Quantitative Variables and Statistical Analysis

Total scores were summarized using means and SDs. Potential associations between explanatory variables and outcome measures were explored using Pearson correlation coefficients, logistic, and linear regression analyses.⁴⁹ Multicollinearity was assessed using variance inflation factor. Results are presented as β coefficients or odds ratios (ORs), 95% CIs, and P values. Variance in assessment scores across surgical specialties and training locations was explored using analysis of variance. A correlation coefficient of 0 to 0.19 was regarded as very weak, 0.20 to 0.39 as weak, 0.40 to 0.59 as moderate, 0.60 to 0.79 as strong, and 0.8 to 1 as very strong.⁵⁰ All statistical analysis was performed using SPSS version 23 (SPSS).

Results

Selection data from 303 successful applicants to RCSI’s CST program from 2016 to 2020 were analyzed. Mean scores achieved during the recruitment process are outlined in Table 2.

Table 2. Mean Scores Awarded to Candidates (N = 303) Successful in Obtaining a Core Surgical Training Post.

Measure	Mean (SD) [range]
Selection metric
Surgical aptitudea (0-15)	9.75 (2.48) [3.00-14.67]
Perception (PicSOr) (0-5)	3.39 (1.73) [0.00-5.00]
Visuospatial aptitude (0-5)	2.31 (1.07) [0.00-4.82]
Manual dexterity (pegboard) (0-5)	4.04 (0.94) [0.00-5.00]
Centile (0-15)	9.67 (3.98) [0.15-15.00]
Clinical judgment (0-15)	12.09 (1.73) [5.75-15.00]
Interpersonal skills (0-15)	11.49 (1.63) [5.25-15.00]
Professional development (0-15)	11.09 (1.91) [4.50-15.00]
Suitability for specialty training (0-25)	19.65 (2.27) [11.90-25.00]
Total score (0-100)	73.43 (6.79) [56.93-89.54]
Assessment method
Operative experience (eLogbook) (0-25)	18.62 (4.25) [2.88-25.00]
Workplace-based operative performance assessments (SSAOP) (0-50)	37.68 (3.64) [21.40-44.50]
Workplace-based clinical assessments (SCA) (0-30)	26.32 (2.53) [15.00-30.00]
Standardized global trainer evaluations (TAR)	45.70 (3.49) [34.75-61.65]
Simulation-based assessments of technical skill (OSS) (0-50)	38.71 (5.24) [17.55-47.36]
Simulation-based assessments of Communication skills (0-50)	38.89 (4.37) [23.80-48.15]

Abbreviations: eLogbook, electronic logbook; PicSOr, Pictorial Surface Orientation testing; OSS, Operative Surgical Skill; SCA, Structured Clinical Assessment; SSAOP, Supervised Structured Assessment of Operative Performance; TAR, Trainee Assessment Report.

^a Surgical aptitude scores are the sum of perception, visuospatial aptitude, and manual dexterity assessment scores. The total score is the sum of surgical aptitude, centile, clinical judgment, interpersonal skills, professional development, and suitability for specialist training scores.

Relationships With Other Variables

Attrition and Progression

Of 303 trainees, 276 completed the CST program (91.09%). Of the 276 trainees who completed CST, 215 subsequently applied to HST (77.89%) and 172 progressed to HST (62.32%). Of these, 121 progressed to HST directly from CST (70.34%), while a further 51 candidates did not progress directly from CST, but did achieve an HST place in subsequent years. The total progression rate from CST to HST was 56.76%. The withdrawal rate was 29.04%. Univariable logistic regression showed no evidence of an association between total score at application and completion of the CST program (OR, 1.02; 95% CI, 0.96-1.08; P = .48) (Table 3). There was evidence of an association with progression to HST (OR, 1.09; 95% CI, 1.05-1.13; P < .001) and the same for direct progression from CST to HST on first attempt (OR, 1.09; 95% CI, 1.05-1.13; P < .001) (Table 3). When the intended specialty of HST application was included in a multiple logistic regression model, significant associations were again observed between total score and progression to HST (OR, 1.13; 95% CI, 1.06-1.20; P < .001) and direct HST progression on first attempt (OR, 1.10; 95% CI, 1.04-1.15; P < .001).

Table 3. Multiple Logistic Regression Analyses Demonstrating the Association Between Application Centile, Psychomotor, and Structured Interview Scores and Outcome Measures.

Measure	OR (95% CI)	P value
Completion of CST
Centile	0.93 (0.84-1.04)	.22
Surgical aptitude	1.07 (0.91-1.25)	.43
Clinical judgment	1.21 (0.95-1.53)	.12
Interpersonal skills	0.90 (0.69-1.18)	.45
Professional development	1.06 (0.85-1.32)	.61
Suitabilitya	1.22 (1.02-1.45)	.03
Progression to HST
Centile	1.05 (0.98-1.12)	.16
Surgical aptitude	0.96 (0.87-1.07)	.49
Clinical judgmenta	1.48 (1.26-1.74)	<.001
Interpersonal skills	1.06 (0.89-1.25)	.52
Professional developmenta	1.15 (1.01-1.32)	.04
Suitabilitya	1.13 (1.01-1.27)	.04
Direct progression to HST b
Centile	1.05 (0.98-1.12)	.12
Surgical aptitudea	1.19 (1.07-1.32)	<.001
Clinical judgmenta	1.39 (1.18-1.64)	<.001
Interpersonal skills	1.18 (0.99-1.36)	.07
Professional development	1.06 (0.92-1.21)	.43
Suitabilitya	1.01 (0.89-1.13)	<.001

Abbreviations: CST, core surgical training; HST, Higher Specialist Training; OR, odds ratio.

^a Statistically significant associations.

^b Direct progression to HST indicates the likelihood of progression to higher specialist training on first application from core surgical training.

On multiple logistic regression analyses, scores in the suitability for specialist training\interview were the only factor at application positively associated with CST completion (Table 3), while scores in clinical judgment, professional development, and suitability for specialist training interviews were positively associated with HST progression. Scores in surgical aptitude (baseline psychomotor assessments), along with scores in 4 of the 5 interview stations, were positively associated direct progression to HST (Table 3).

Associations With In-Training Assessment Scores

Mean scores awarded in CST assessments are outlined in Table 2. Total score at application was significantly and positively correlated with performance in all future CST assessments (r = 0.22-0.34) (eTable 1 in Supplement 1). Correlations between scores awarded at application and scores in future performance assessments are further outlined in eTable 1 in Supplement 1.

Multivariable linear regression again demonstrated a number of significant associations between assessed factors at application and future performance (Table 4). When examining the standardized coefficients, surgical aptitude was the largest contributor of the independent variables to scores in both workplace-based and simulation-based assessments of operative skill. Scores awarded in interpersonal skill interview stations were the largest contributor to performance in subsequent communication skill assessments. There was observable and significant variance in assessment scores across training locations and surgical specialties (eTables 2 and 3 in Supplement 1).

Table 4. Measured Candidate Factors at Selection and Associations With Future Performance on Multivariable Linear-Regression Analysis.

Selection domain	Core surgical training assessment
	Mean standardized eLogbook	Mean SSAOP	Mean SCA	Mean standardized TAR	Mean OSS	Mean communication skills
Centile
β (95% CI)	0.19 (0.07-0.31)	0.14 (0.04-0.25)	0.09 (0.02-0.17)	0.12 (0.03-0.22)	0.17 (0.02-0.32)	0.17 (0.05-0.30)
P value	.002	.01	.01	.01	.02	.01
Surgical aptitude
β (95% CI)	0.22 (0.02-0.41)	0.31 (0.14-0.48)	0.40 (0.25-0.56)	0.57 (0.33-0.81)	0.57 (0.33-0.81)	0.26 (0.06-0.46)
P value	.03	<.001	.03	<.001	<.001	.01
Clinical judgment
β (95% CI)	0.22 (−0.06 to 0.51)	−0.26 (−0.51 to 0.02)	−0.14 (−0.31 to 0.04)	−0.22 (−0.45 to 0.01)	0.24 (−0.11 to 0.59)	−0.38 (−0.68 to −0.09)
P value	.12	.04	.13	.06	.18	.01
Interpersonal skills
β (95% CI)	0.62 (0.31-0.93)	0.34 (0.07-0.61)	0.21 (0.02-0.39)	0.54 (0.29-0.79)	0.60 (0.22-0.99)	0.55 (0.22-0.87)
P value	<.001	.01	.03	<.001	.002	.001
Professional development
β (95% CI)	0.12 (−0.14 to 0.38)	0.18 (−0.05 to 0.40)	0.18 (0.03-0.34)	0.22 (0.01-0.42)	0.07 (−0.25 to 0.39)	−0.06 (−0.33 to 0.21)
P value	.36	.12	.02	.04	.67	.69
Suitability
β (95% CI)	0.18 (−0.04 to 0.39)	0.14 (−0.05 to 0.33)	0.06 (−0.07 to 0.19)	0.14 (−0.04 to 0.31)	0.28 (0.00-0.55)	0.37 (0.14-0.60)
P value	0.11	.16	.38	.12	.05	.002

Abbreviations: eLogbook, electronic logbook; SCA, Structured Clinical Assessment; SSAOP, Supervised Structured Assessment of Operative Performance; OSS, Objective Surgical Skills; TAR, Trainee Assessment Report.

Discussion

Performance in a multifaceted selection process is not associated with completion of CST, but is positively associated with subsequent progression to HST in the trainee’s chosen subspecialty. Scores awarded at selection are significantly associated with the future performance across a range of workplace-based and simulation-based assessments. Surgical aptitude, as measured using psychomotor assessments, was the largest contributor of measured factors to future operative performance assessment scores. Candidate performance in the interpersonal skills station of a multistation interview was the most strongly associated selection variable with performance in future simulated communication skill assessments.

Candidate performance at the time of selection was not associated with withdrawal from the CST program. A low attrition rate may act as a useful measure of the efficacy of a selection process.³ However, prior work from our institution suggests that trainees do not leave surgical training due to self-perceived skill-program mismatch, but rather due to negative perceptions regarding program delivery, training atmosphere, lifestyle considerations, peer influence, and concerns regarding career progression.⁵¹ Therefore, it is appropriate to view attrition as influenced by both candidate and program factors. It is also important to consider consequential effects of selection on future performance when using progression as a measure of success. In Ireland, rank achieved at selection informs the location and specialties that trainees rotate through; a higher ranking leads to greater choice. Candidates who perform less well at selection are less likely to undertake their chosen rotations. This may lead to demotivation with effects on performance, and ultimately, a lower likelihood of progression. The candidate experience of selection, ranking, and the perceived impact on future training should be explored.

A number of significant associations are noted in this study between measured attributes at the time of selection and performance in various workplace-based and simulation-based assessments. Individual correlations between selection and future performance measures are weak. The CST selection process sits within a complex system of recruitment, training, assessment, and progression. Experiences of training can vary greatly for trainees. At our institution, we endeavor to measure operative volume and autonomy through surgical logbooks⁴⁷ and have previously demonstrated the association of baseline technical aptitude assessments with operative performance when controlling for experience.²⁹ It would be naïve to assume, however, that operative volume is the sole measure of a positive training experience. The effects of individual trainers,⁵² the institutional training environment,⁵³ engagement with deliberate practice,⁵⁴ and innumerable further variables contribute to trainee performance. The inherent variability in trainee experience postselection may explain in part why studies to date have struggled to relate selection metrics with performance outcomes.²³

Variance in assessment scores was observable across training locations and specialties (eTable 1 in Supplement 1). There are several factors that could account for observed differences, including inter-rater variability, differences in operative exposure or autonomy granting, or differences in case mix across hospital sites. Contributing factors to assessment score variance can be explored statistically.⁵⁵ Variance across training experiences could subsequently contribute to the likelihood of progression to HST. It is reasonable to anticipate that candidates who have undertaken certain posts during their training may be more likely to obtain a place in HST; whether this relates to the experiences gained in certain posts leading to better assessment outcomes or the perception of said posts by interviewers at the time of HST recruitment should be studied. Trainees rotate across specialties and locations during CST. As trainees rank their own rotation preferences, it is likely that trainees who perform well at selection seek out positive training experiences. Candidate self-selection could account for a significant proportion of observed variance in assessment outcomes and likelihood of progression. The experience of trainees as they progress through the process of selection and training is worthy of further qualitative study, and can be informed by social validity principles.⁵⁶ The relative contributions of trainee aptitude and training experiences to assessment outcomes and progression are challenging, but can and should be explored by future studies.

The association between measured factors and future performance is just one source of validity evidence for the selection program. The content validity (ie, the extent to which measured factors accurately reflect the construct of interest)⁵⁷ and response process validity (ie, the fit between the construct of interest and the process of selection as it is actually engaged in) can be further analyzed both quantitatively and qualitatively. For example, in this study, the suitability for specialty training mini-interview station scores demonstrated a relatively wide range and SD. Scores in this station correlate with likelihood of progression. The response process validity of this station could be evaluated by qualitatively assessing rater’s thoughts and actions during candidate interview to determine how raters come to decisions on ranking.⁵⁸ This would be important to ensure that stations representing as broad a construct as suitability do not represent a source of bias, and subsequently, contribute to a lack of candidate diversity.

Beyond validity, there are further considerations regarding the utility of RCSI’s selection program across reliability, educational impact, acceptability, feasibility and cost-effectiveness domains.⁵⁹ Further research will be required to analyze the additional potential benefit of assessing other skills or attributes at the time of selection, through situational judgment testing,⁶⁰ assessment of grit,⁶¹ or a more nuanced quantification of academic record. Lastly, differential attainment across demographic groups⁶² should be explored to ensure that the selection process meets high standards of diversity, equity, and inclusion. While RCSI publishes annual reports on the gender, country of origin, and further demographic data on CST appointees,⁶³ the impact of such attributes on candidate selection and progression was not explored by this study.

Approaches to the selection of surgical trainees vary across jurisdictions,³ ranging from highly standardized and centralized processes, to completely decentralized local institutional recruitment.^3,10 Technical skill or aptitude assessments are rarely used to inform recruitment. Some studies have explored the use of simulated surgical task assessments to stratify candidates.^64,65,66 Lund et al⁶⁷ report that simulation-based assessment scores were associated with higher future milestone assessment scores (β = 0.45; P = .03). Assessment of performance in surgical tasks or procedures, however, may risk selecting candidates based on experience rather than aptitude. Using psychomotor tests of aptitude, far removed from the representation of real-world tasks, may represent a more valid method for assessing a candidate’s technical learning ability. Such aptitude tests are associated with the rate of skill acquisition.^19,68,69 The findings of this study support the use of these inexpensive tests to inform selection decisions.

Limitations

Performance data are available only for candidates who were successful in obtaining a CST number at the Royal College of Surgeons in Ireland. Selected candidates are those who possess the same sought-after characteristics, and common method variance may therefore account for some of the correlation observed between selection and in-training assessments.¹³ The reliability of the findings above relies on the accuracy and validity of trainer- and trainee-submitted data. A broad range of knowledge, skill, and attitude domains likely contribute to the complex concept of future competence; it is difficult, if not impossible, to account for all of these known and unknown contributors at the time of selection.

Conclusions

In this study, performance at the time of selection for surgical training, measured across technical and nontechnical domains, was associated with future performance. Performance at the time of selection is further positively associated with progression to higher specialist training in the candidate’s preferred surgical specialty. Of measured factors at selection, surgical aptitude was the most strongly associated with future operative performance, while the interpersonal skills station of a multi-station interview was the most strongly associated with communication skill assessment scores. Future studies will explore the consequential validity of selection, including potential unintended downstream effects of selection and ranking on candidate performance.

Supplement 1.

eAppendix. Assessment tools

eTable 1. Correlation between applicant factors evaluated at the time of selection, and future performance in assessments throughout Core Surgical Training, presented as Pearson correlation coefficients (r) and P-value

eTable 2. Analysis of Variance (ANOVA) demonstrating variance in assessment scores across training locations

eTable 3. Analysis of Variance (ANOVA) demonstrating variance in assessment scores across sub-specialties

Supplement 2.

Data sharing statement