Abstract
Background:
An oversupply of qualified applicants leads to intense competition for the limited number of first year orthopedic residency positions. Therefore, program directors can be more selective in choosing their future residents. However, it is unclear if there are resident characteristics that correspond with trainee performance.
Methods:
We asked (1) what resident characteristics are associated with subjective residency performance score? and (2) what resident characteristics are associated with Orthopedic In-Training Examination (OITE) score?
A total of 119 orthopedic residents accepted at the Harvard Combined Orthopedic Residency Program from 1999 – 2009 were included in this study. The current program director together with two former program directors in the selected time period defined the subjective residency performance score based on the clinical skills of the residents during training.
Results:
Former Olympic or varsity athlete (P=0.018) and Alpha Omega Alpha (AOA) status (P=0.014) were associated with a better subjective residency performance score. Higher USMLE step 1 score (P=0.0038), known person within faculty prior to the residency (did a research rotation, or local medical student) (P=0.041), and AOA (P=0.015) status were associated with a higher OITE score.
Conclusion:
AOA status of the applicant for orthopedic residency is associated with both a higher OITE score and a better subjective residency performance score.
Key Words: Interview, Orthopedic surgery, Residency, Selection process
Introduction
The number of applicants for orthopedic residency programs consistently exceeds the available number of positions each year (1). An oversupply of qualified applicants leads to intense competition for the limited number of first year orthopedic residency positions. This allows program directors to be more selective in choosing their future residents (1, 2).
A few studies address the factors involved in selecting residency applicants, but the results are inconsistent (3-8). For example, Carmichael et al. looked at age at the start of residency and Orthopedic In-Training Examination (OITE) scores and found no association (8). On the other hand, Clark et al. (2) found that successful orthopedic residency applicants were significantly younger. The mean age of successful candidates was 26 years and the mean age of unsuccessful candidates was 28 years (2).
It is not clear whether factors used to select residents correlate with performance during residency or later as orthopedic surgeons. We studied resident characteristics associated with their performance as a resident. Specifically, we studied the following markers for resident performance: (1) subjective residency performance score and (2) OITE score. Our primary null hypothesis was that there were no resident characteristics associated with subjective residency performance score and our secondary hypothesis was that there were no resident characteristics associated with OITE score.
Materials and Methods
Our institutional review board approved this retrospective study. We included all accepted orthopedic residents at the Harvard Combined Orthopedic Residency Program from 1999 – 2009 [Table 1]. One resident was excluded, because the application form was missing. A total of 119 orthopedic residents were included.
Table 1.
Baseline characteristics (n = 119)
| Parameter | |
|---|---|
| Age, mean (sd) | 29 (2.7) |
| Sex, n (%) | |
| Male | 96 (81) |
| Female | 23 (18) |
| American nationality, n (%) | 112 (94) |
| Married, n (%) | 64 (54) |
| Advanced research degree, n (%) | 3 (3) |
| Number of publications, n (%) | |
| 0 | 30 (35) |
| <5 | 66 (55) |
| 5-10 | 15 (13) |
| >10 | 8 (7) |
| USMLE step 1 score, n (%) | |
| <192 | 5 (4) |
| 192-228 | 26 (22) |
| 229-264 | 83 (70) |
| >264 | 5 (4) |
| Match ranking, median (ICQ) | 12 (6-20) |
| Former Olympic or varsity athlete, n (%) | 27 (23) |
| Musical instrument in orchestra/band, n (%) | 4 (3) |
| Involvement in charity, n (%) | 101 (85) |
| Management experience, n (%) | 39 (33) |
| Known person within faculty, n (%) | 46 (39) |
| Entrepreneur background, n (%) | 7 (6) |
| Alpha Omega Alpha, n (%) | 46 (39) |
| USMLE = United States Medical Licensing Examination | |
Our primary outcome measure was a subjective residency performance score as outcome measure. The current program director together with two former program directors in the selected time period defined the subjective residency performance score based on the clinical skills and interim evaluation forms of the residents during training. This resulted in a subjective resident performance from 1 (best score) to 3 (lowest score). We took the mean score and rounded up when there was a discrepancy in resident performance scores.
Our secondary outcome measure was the OITE score. This examination is a measure of orthopedic medical knowledge of residents and is used by all American Orthopedic residency programs on a yearly basis. Based on the first OITE score from the training records the resident was categorized in the following quartiles: <25%; 25-50%; 51-75%; >75%.
We included the following explanatory variables: age, sex, nationality, marital status, advanced research degree (e.g. masters, PhD), number of publications (0, <5, 5-10, > 10), United States Medical Licensing Examination (USMLE) step one (<192, 192-228, 229-264, > 264), match ranking (the number the applicant was ranked during the interviews), former Olympic or varsity athlete, musical instrument (in orchestra or band), involvement in charity, management experience (e.g. organizing a congress, conference or event), known person within faculty (performed previous internship/rotation/or elective with faculty), entrepreneur background, and Alpha Omega Alpha (AOA) status. Data was gathered from letters of recommendation, resumes and application forms of the resident.
Two residents quit residency during the program.
Statistical analysis
Continuous variables were described using means with standard deviations, and with medians and interquartile ranges if inspection of the data showed non-normality. Categorical variables were presented with frequency and percentage statistics.
In bivariate analyses, the association of the outcome measure resident performance score and explanatory variables was assessed using a Fishers exact test for dichotomous variables, and a Kruskal-Wallis test for continuous and categorical variables.
The association of the outcome measure OITE score and explanatory variables was assessed using a Mann-Whitney U test for dichotomous variables, and a Spearman’s rank correlation coefficient for continuous and categorical variables.
All analyses were performed with Stata 13 (StataCorp LP). A two-tailed p value smaller than 0.05 was considered significant.
Results
Former Olympic or varsity athlete (P=0.018) and AOA status (P=0.014) were associated with a better subjective residency performance score [Table 2].
Table 2.
Bivariate analysis: factors associated with subjective residency performance score (based on a 1 to 3 score of the program director at that time) (n = 119)
| Parameter | Group 1 | Group 2 | Group 3 | p value |
|---|---|---|---|---|
| Age, median (ICQ) | 30 (27-31) | 28 (27-29) | 28 (27-30) | 0.066 |
| Match ranking, median (ICQ) | 10 (7-18) | 11 (6 (20) | 15 (9-25) | 0.28 |
| Number of publications, n(%) | 0.75 | |||
| 0 | 9 (7.6) | 15 (13) | 6 (5.0) | |
| <5 | 18 (15) | 26 (22) | 22 (18) | |
| 5 - 10 | 6 (5.0) | 5 (4.2) | 4 (3.4) | |
| >10 | 1 (0.8) | 6 (5.0) | 1 (0.8) | |
| USMLE step 1 score, n(%) | 0.75 | |||
| <192 | 1 (0.8) | 2 (1.7) | 2 (1.7) | |
| 192 - 228 | 10 (8.4) | 9 (7.6) | 7 (5.9) | |
| 229 - 264 | 22 (18) | 37 (31) | 24 (20) | |
| >264 | 1 (0.8) | 4 (3.4) | 0 (0) | |
| Sex, n(%) | 0.61 | |||
| Male | 29 (24) | 42 (35) | 25 (21) | |
| Female | 5 (4.2) | 10 (8.4) | 8 (6.7) | |
| Nationality, n(%) | 0.63 | |||
| American | 31 (26) | 50 (42) | 31 (26) | |
| Non American | 3 (2.5) | 2 (1.6) | 2 (1.7) | |
| Marital status, n(%) | 0.32 | |||
| Married | 20 (17) | 30 (25) | 14 (12) | |
| Single | 14 (12) | 22 (18) | 19 (16) | |
| Advanced research degree, n(%) | 0.11 | |||
| Yes | 1 (0.8) | 0 (0) | 2 (1.7) | |
| No | 33 (28) | 52 (44) | 31 (26) | |
| Former Olympic or varsity athlete, n(%) | 0.018 | |||
| Yes | 13 (11) | 11 (9.2) | 3 (2.5) | |
| No | 21 (18) | 41 (34) | 30 (25) | |
| Musical instrument in orchestra/band, n(%) | 0.29 | |||
| Yes | 0 (0) | 2 (1.7) | 2 (1.7) | |
| No | 34 (29) | 50 (42) | 31 (26) | |
| Involvement in charity, n(%) | 0.35 | |||
| Yes | 27 (23) | 47 (39) | 27 (23) | |
| No | 7 (5.9) | 5 (4.2) | 6 (5.0) | |
| Management experience, n(%) | 0.11 | |||
| Yes | 15 (13) | 12 (10) | 12 (10) | |
| No | 19 (16) | 40 (34) | 21 (18) | |
| Known person within faculty, n(%) | 0.77 | |||
| Yes | 15 (13) | 19 (16) | 12 (10) | |
| No | 19 (16) | 33 (28) | 21 (18) | |
| Entrepreneur background, n(%) | 0.13 | |||
| Yes | 2 (1.7) | 1 (0.8) | 4 (3.4) | |
| No | 32 (27) | 51 43) | 29 (24) | |
| Alpha Omega Alpha, n(%) | 0.014 | |||
| Yes | 17 (14) | 23 (19) | 6 (5.0) | |
| No | 17 (14) | 29 (24) | 27 (23) | |
| USMLE = United States Medical Licensing Examination | ||||
| OITE = Orthopaedics In-Training Exam |
Higher USMLE step 1 score (P=0.0038), known person within faculty prior to the residency (did a research rotation, or local medical student) (P=0.041), and AOA (P=0.015) status were associated with a higher OITE score [Table 3].
Table 3.
Bivariate analysis: factors associated with OITE score (n = 119)
| Parameter | Coefficient | p value |
|---|---|---|
| Age | -0.080 | 0.39 |
| Match ranking, median | -0.080 | 0.41 |
| Number of publications | -0.088 | 0.34 |
| USMLE step 1 score | 0.26 | 0.0038 |
| Mean (sd) | p value | |
| Sex | 0.29 | |
| Male | 3.2 (0.90) | |
| Female | 3.0 (0.88) | |
| Nationality | 0.29 | |
| American | 3.2 (0.88) | |
| Non american | 3.4 (1.1) | |
| Marital status | 0.83 | |
| Married | 3.2 (0.91) | |
| Single | 3.2 (0.88) | |
| Advanced research degree | 0.72 | |
| Yes | 3.3 (1.2) | |
| No | 3.2 (0.89) | |
| Former Olympic or varsity athlete | 0.45 | |
| Yes | 3.1 (0.85) | |
| No | 3.2 (0.91) | |
| Musical instrument in orchestra/band | 0.56 | |
| Yes | 2.8 (1.5) | |
| No | 3.2 (0.87) | |
| Involvement in charity | 0.42 | |
| Yes | 3.1 (0.90) | |
| No | 3.3 (0.91) | |
| Management expierence | 0.42 | |
| Yes | 3.3 (0.89) | |
| No | 3.2 (0.90) | |
| Known person within faculty | 0.041 | |
| Yes | 3.0 (0.95) | |
| No | 3.3 (0.83) | |
| Entrepreneur background | 0.91 | |
| Yes | 3.3 (0.76) | |
| No | 3.2 (0.91) | |
| Alpha Omega Alpha | ||
| Yes | 3.4 (0.81) | 0.015 |
| No | 3.0 (0.92) |
USMLE = United States Medical Licensing Examination
OITE = Orthopaedics In-Training Exam
Discussion
There is strong competition for the limited number of first year orthopedic residency positions (1, 2). Therefore program directors can be more selective in choosing their future residents. However, it is unclear if there are resident characteristics that correspond with trainee performance. The primary null hypothesis of this study was that there were no resident characteristics associated with a subjective residency performance score.
We found that former Olympic or varsity athlete and AOA status were associated with a better subjective residency performance score.
This study should be interpreted with its limitations in mind. First, the data registry is drawn from the Harvard Combined Orthopedic Residency Program and might not be representative of other residency programs. Second, the retrospective study design is somewhat susceptible to data loss, bias, and confounding, although we felt this was likely uncommon. Third, our measures (e.g. a higher OITE score) are surrogates for actual performance as a surgeon. Fourth, there might be more factors that can influence residence acceptance, such as for example likability. However, it is not possible to analyze this. Lastly, the residency performance score is a subjective score given by the program directors.
Our study showed that former Olympic or varsity athlete status was associated with a better subjective residency performance score. This might be due to perseverance to reach a goal. However, it is shown that extracurricular activities were ranked relatively less important in the selection process for residents (9). Our results showed that AOA status was associated with a better subjective residency performance score. Several studies showed that medical school honors grades were associated with more career potential (2, 10-13). Dirschl et al. attempted to correlate residency selection characteristics with subsequent faculty evaluations (14). The faculty members were asked to evaluate the residents in psychomotor, affective, cognitive, and overall performance. Clinical clerkship performances as indicated by honors grades was found to have the highest correlation to overall resident performance, and AOA status was second (14).
We found that a higher USMLE step 1 score was associated with higher OITE scores. This is consistent with the study of Carmichael et al. that showed that residents with an USMLE step 1 score of 220 or lower had a lower OITE score as compared to the residents with a score above 220 (8). It makes some sense that test scores are consistent, because applicants that had high scores at USMLE step 1 are probably good test takers and will do well at the OITE test too. Our study showed that a known person within the faculty was associated with higher OITE scores. An explanation might be that only the best applicants known within faculty will get accepted to the program. Previous studies reported that taking an elective in the program director’s hospital and specialty seems to be of great importance in the selection procedure of orthopedic residents (1, 4, 15-17). Bernstein et al. found that performing a rotation at the director’s institution was ranked as most important (1). Another study found that the most important academic criterion in the selection of residents among orthopedic program directors was the candidates grade in his or her orthopedic senior elective (4). In 1986, Wagoner et al. found that 86% of program directors across multiple medical and surgical specialties gave preference to students who performed well in an elective in the program director’s specialty and hospital (16). We found that AOA status was associated with a higher OITE score. An explanation might be that the OITE score and AOA status are both based on knowledge. This is consistent with the study of Carmichael et al. (8). Several studies reported that AOA membership was one of the strongest predictors of a successful application for orthopedic residency (1, 2, 4).
In conclusion, AOA status of the applicant is associated with a higher OITE score and a better subjective residency performance score. Future studies should focus on objective clinical measurements of the resident’s performance.
Conflicts of Interest:
The authors declare they have no competing interests.
Disclosure:
there were no conflicts of interest.
Acknowledgements
one.
References
- 1.Bernstein AD, Jazrawi LM, Elbeshbeshy B, Della Valle CJ, Zuckerman JD. An analysis of orthopaedic residency selection criteria. Bull Hosp Jt Dis. 2002;61(1-2):49–57. [PubMed] [Google Scholar]
- 2.Clark R, Evans EB, Ivey FM, Calhoun JH, Hokanson JA. Characteristics of successful and unsuccessful applicants to orthopedic residency training programs. Clin Orthop Relat Res. 1989;241(1):257–64. [PubMed] [Google Scholar]
- 3.Scherl SA, Lively N, Simon MA. Initial review of electronic residency application service charts by orthopaedic residency faculty members Does applicant gender matter? J Bone Joint Surg Am. 2001;83(1):65–70. doi: 10.2106/00004623-200101000-00009. [DOI] [PubMed] [Google Scholar]
- 4.Wagoner NE, Suriano JR. Program directors’ responses to a survey on variables used to select residents in a time of change. Acad Med. 1999;74(1):51–8. [PubMed] [Google Scholar]
- 5.Zagumny MJ, Rudolph J. Comparing medical students’ and residency directors’ ratings of criteria used to select residents. Acad Med. 1992;67(9) doi: 10.1097/00001888-199209000-00019. [DOI] [PubMed] [Google Scholar]
- 6.Taylor CA, Mayhew HE, Weinstein L. Residency directors’ responses to the concept of a proposed electronic residency application service. Acad Med. 1994;69(2):138–42. doi: 10.1097/00001888-199402000-00020. [DOI] [PubMed] [Google Scholar]
- 7.Ross CA, Leichner P. Criteria for selecting residents: a reassessment. Can J Psychiatry. 1984;29(8):681–6. doi: 10.1177/070674378402900809. [DOI] [PubMed] [Google Scholar]
- 8.Carmichael KD, Westmoreland JB, Thomas JA, Patterson RM. Relation of residency selection factors to subsequent orthopaedic in-training examination performance. South Med J. 2005;98(5):528–32. doi: 10.1097/01.SMJ.0000157560.75496.CB. [DOI] [PubMed] [Google Scholar]
- 9.Crane JT, Ferraro CM. Selection criteria for emergency medicine residency applicants. Acad Emerg Med. 2000;7(1):54–60. doi: 10.1111/j.1553-2712.2000.tb01892.x. [DOI] [PubMed] [Google Scholar]
- 10.Dorsey ER, Raphael BA, Balcer LJ, Galetta SL. Predictors of future publication record and academic rank in a cohort of neurology residents. Neurology. 2006;67(8):1335–7. doi: 10.1212/01.wnl.0000240058.26740.22. [DOI] [PubMed] [Google Scholar]
- 11.Lawton MT, Narvid J, Quinones-Hinojosa A. Predictors of neurosurgical career choice among residents and residency applicants. Neurosurgery. 2007;60(5):934–9. doi: 10.1227/01.NEU.0000255445.51989.33. [DOI] [PubMed] [Google Scholar]
- 12.Bilbey JH, Fache JS, Burhenne HJ. Are there predictors for future academic radiologists? A Canadian survey. Can Assoc Radiol J. 1992;43(5):369–73. [PubMed] [Google Scholar]
- 13.McCaffrey JC. Medical student selection of otolaryngology-head and neck surgery as a specialty: influences and attitudes. Otolaryngol Head Neck Surg. 2005;133(6):825–30. doi: 10.1016/j.otohns.2005.07.042. [DOI] [PubMed] [Google Scholar]
- 14.Dirschl DR, Dahners LE, Adams GL, Crouch JH, Wilson FC. Correlating selection criteria with subsequent performance as residents. Clin Orthop Relat Res. 2002;399(1):265–71. doi: 10.1097/00003086-200206000-00034. [DOI] [PubMed] [Google Scholar]
- 15.Garden FH, Smith BS. Criteria for selection of physical medicine and rehabilitation residents A survey of current practices and suggested changes. Am J Phys Med Rehabil. 1989;68(3):123–7. doi: 10.1097/00002060-198906000-00005. [DOI] [PubMed] [Google Scholar]
- 16.Wagoner NE, Suriano JR, Stoner JA. Factors used by program directors to select residents. J Med Educ. 1986;61(1):10–21. doi: 10.1097/00001888-198601000-00002. [DOI] [PubMed] [Google Scholar]
- 17.Sherry E, Mobbs R, Henderson A. Becoming an orthopaedic surgeon: background of trainees and their opinions of selection criteria for orthopaedic training. Aust N Z J Surg. 1996;66(7):473–7. doi: 10.1111/j.1445-2197.1996.tb00785.x. [DOI] [PubMed] [Google Scholar]