Abstract
BACKGROUND
Program directors for Family Medicine residencies must navigate an increasingly complex recruitment landscape. With increasing United States allopathic and osteopathic graduates and continued high volumes of international graduates, the ability to identify application characteristics that predict quality residents both for filtering applications for interview offers and ranking is vital. Our study concentrates on the predictive value of reported life experiences including volunteerism, work experiences, prior career, research experience, and participation in medical student organizations including student leadership.
METHODS
Through a retrospective cohort study, we extracted the described life experiences from resident application materials. We then obtained initial clinical performance data on the Family Medicine inpatient service during the first six months of residency to determine readiness for residency. This analysis occurred in 2020 and included all matriculants in the graduating classes of 2013 through 2020 for a single residency. Of 110 matriculating residents, data were available for 97(88%).
RESULTS
Applicants with a history of a prior career demonstrated improved overall readiness for residency with competency domain-specific advantages in Interpersonal and Communication Skills and Systems-Based Practice. In contrast, applicants reporting participation in research performed below peers in all competency domains. Applicant reports on volunteerism, work experience, academic productivity and student involvement did not correlate with initial clinical performance.
CONCLUSIONS
Residency directors should recognize applicants with prior careers as likely having strong communications and systems-based practice skills. All other examined experiences should be evaluated within the context of broader applicant assessments including research experience which overall has a potential negative correlation to clinical readiness.
Keywords: Volunteerism, prior career, applicant research, residency recruitment
Introduction
Family Medicine residency programs distill a significant volume of applicant data during the recruitment process. What criteria is useful during initial sorting and the predictive value of corresponding applicant data remains unclear.1,2 Based on the premise that past accomplishments predict successful residents, 3 we investigated whether volunteerism, work experience, prior career, research experience, or student organizational engagement were predictive of resident performance.
The importance of the biopsychosocial model of care in the specialty of Family Medicine seems to naturally support the humanism associated with volunteerism as a predictive attribute in applicants. Literature supports self-reported growth in multiple areas of the Accreditation Council for Graduate Medical Education (ACGME) competency domains after meaningful participation in service activities. One study in a developed Asian country demonstrated personal self-reported growth in empathy, communication skills, organization, decision-making, and interpersonal and leadership skills after participation in a community service experience in a cohort of medical students. 4 Unfortunately, literature is lacking in determining a correlation between applicant self-reported volunteerism and clinical performance as a resident.
Applicant participation in research and in associated publication activity has been reported for several medical specialties. One study of 1020 matriculants at the Uniformed Services University of the Health Sciences determined no relationship between self-reported research and success in medical school or graduate medical education. 5 An Internal Medicine evaluation of applicant reported research also showed no correlation to clinical performance, 6 while an investigation into the selection process for surgical residents found that prior research experience and research publications had a negative correlation with clinical performance ratings during residency. 7 Research remains a central focus in residency applications, interviews and ranking conversations for many specialties despite specialty literature questioning its value. It is worth noting the distinction between self-reported applicant research and scholarship that occurs within the context of residency training; there is a positive correlation between participation in research during residency and clinical performance. 8 Literature in these areas remains lacking in Family Medicine.
We further examined two areas in which little literature is available for comparison: work experience and engagement in medical school organizations. An applicant history of work experience both during education and as a separate, prior career is a self-reported component in an Electronic Residency Application System (ERAS) application. Work experience likely provides opportunities for growth in communication skills, professionalism, teamwork, organizational skills and work ethic. These attributes correlate well with multiple ACGME competencies providing a theoretical framework for applicant advantage. However, a multicenter study on the predictors of top performers in Emergency Medicine found no correlation between full-time work experience and resident performance. 9 Similarly, investment of personal time during medical school in organizational activities and leadership such as student government would create opportunities for anticipated growth across multiple ACGME competency domains. Although extracurricular activities such as these are looked upon favorably by program directors, 10 literature evaluating whether these characteristics were predictive of clinical performance in residency for Family Medicine or other specialties is limited.
Methods
This research occurred in an urban, medical school-based residency. Research included all matriculating residents for the graduating cohorts from 2013 through 2020. Research occurred in 2020.
Data from self-reported information in ERAS applications were extracted utilizing a standard form. This included volunteer and work experiences subdivided by type, duration and cumulative hours. A prior career defined as two or more years of established full-time employment while not engaged in education was separated from work experience for separate analysis. Research experience was divided into basic science research, non-basic science research and academic productivity through peer-reviewed publications, posters, abstracts and presentations. Engagement in medical school student organizations including leadership was gathered. Evidence of residency readiness was determined utilizing a composite of resident clinical assessments during Family Medicine inpatient rotations occurring during the first six months of training. Assessment reports were generated for each resident in composite form with utilization of quantitative data with a range of three to eight assigned evaluators. Assessments included representative milestones for the inpatient setting in each of the ACGME competency domains utilizing developmental behaviors and level of expectations as defined by the Family Medicine Residency Review Committee. The assessment tool length and value of data gathered were evaluated through a three-month pilot prior to implementation. The assessment scale mirrored the ACGME Family Medicine milestones with a one representing a novice, a two an advanced beginner, a three competent, a four proficient (graduation target) and a five an expert. The tool was 25 questions in length. Assessments were completed for all residents on the inpatient service at the conclusion of each week by the assigned rounding faculty. Delivery of assessments to faculty and results to residents occurred electronically through the residency management system.
All matriculating residents through the National Resident Matching Program (NRMP) were included in research analysis. Five residents separated from the residency due to transfer, academic or personal reasons prior to graduation. Three residents matriculated outside of the NRMP process. Data was included as available.
We selected Welch's two-sample t-test as the statistical procedure since dealing with disparate sample sizes between comparison groups.
This study was determined to be exempt by the Indiana University School of Medicine Institutional Review Board. The need for informed consent is waived by the Indiana University School of Medicine Institutional Review Board as it is secondary research. All methods were carried out in accordance with relevant guidelines and regulations.
Results
Of 110 residents who entered the program during the study period, complete assessment information was available for 97 (88%). Applicant data on the 97 applicants varied. Table 1 describes resident characteristics.
Table 1.
Demographic and life experience characteristics among matriculating residents.
| N | |
|---|---|
| Female | 43 |
| Participation in scholarly activity | 20 |
| US allopathic medical school graduate | 24 |
| International medical school graduate | 73 |
| Work experience prior to medical school | 60 |
| Engagement in medical school organizations | 38 |
| Engaged in a career prior to residency Medical professional (N = 20)
|
27 |
Formal research experience
|
32 |
Publications and/or posters
|
33 |
Volunteer experiences
|
82 |
Using the presence or absence of six life experiences, we performed comparisons between applicant groups to determine if their academic performance or clinical ratings differed. Results of the statistical procedure were: (1) applicants with less than 30 months of volunteer experience scored significantly higher in the domain of professionalism (Table 2); (2) applicants with no work experience scored significantly higher in the domain of system-based practice (Table 3); (3) applicants with a history of a prior career (distinct from work experience) scored significantly higher in the domains of Interpersonal and Communication Skills Systems-Based Practice, and on their composite score (Table 4); (4) applicants with no research experience scored significantly higher in every competency domain (Table 5); and (5) there were no significant differences for any of the domains between groups in the areas of academic productivity (publications and posters) and organizational engagement in medical school.
Table 2.
Welch's t-test for matriculating residents volunteer experience.
| Assessment Domain |
Mean scores for residents having volunteer experience prior to residency (SD) | P value | |
|---|---|---|---|
| <30 months (n = 13) | ≥30 months (n = 31) | ||
| Patient Care | 3.07 (.90) | 2.52 (.87) | .195 |
| Medical Knowledge | 2.91 (.89) | 2.43 (.83) | .118 |
| Interpersonal and Communication Skills | 2.93 (.92) | 2.42 (.93) | .108 |
| Professionalism | 3.08 (1.0) | 2.35 (1.02) | .040* |
| Systems-Based Practice | 2.51 (1.0) | 2.02 (1.07) | .161 |
| Practice-Based Learning and Improvement | 2.96 (.82) | 2.59 (.86) | .195 |
| Composite | 2.91 (.85) | 2.39 (.91) | .085 |
*Difference is statistically significant @ P < .05.
Table 3.
Welch's t-test for matriculating residents work experience.
| Assessment domain | Mean scores for residents having work experience prior to residency (SD) | P value | |
|---|---|---|---|
| Yes (n = 56) | No (n = 41) | ||
| Patient Care | 2.68 (0.83) | 2.85 (0.91) | .337 |
| Medical Knowledge | 2.61 (0.80) | 2.81 (0.90) | .250 |
| Interpersonal and Communication Skills | 2.63 (0.85) | 2.86 (1.05) | .258 |
| Professionalism | 2.59 (.94) | 2.74 (1.05) | .438 |
| Systems-Based Practice | 2.24 (1.03) | 2.68 (1.04) | .045* |
| Practice-Based Learning and Improvement | 2.61 (0.80) | 2.85 (0.83) | .456 |
| Composite | 2.58 (0.84) | 2.78 (0.95) | .305 |
*Difference is statistically significant @ P < .05.
Table 4.
Welch's t-test for matriculating residents prior career.
| Assessment Domain |
Mean scores for residents having a career prior to residency (SD) | P value | |
|---|---|---|---|
| Yes (n = 24) | No (n = 46) | ||
| Patient Care | 2.91 (0.73) | 2.60 (0.88) | .123 |
| Medical Knowledge | 2.89 (0.69) | 2.54 (0.86) | .072 |
| Interpersonal and Communication Skills | 3.04 (0.82) | 2.51 (0.96) | .019* |
| Professionalism | 2.89 (0.80) | 2.47 (1.06) | .066 |
| Systems-Based Practice | 2.76 (0.88) | 2.14 (1.11) | .015* |
| Practice-Based Learning and Improvement | 2.92 (0.67) | 2.65 (0.81) | .143 |
| Composite | 2.91 (0.74) | 2.49 (0.91) | .042* |
*Difference is statistically significant @ P < .05.
Table 5.
Welch's t-test for matriculating residents research experience.
| Assessment domain | Mean scores for residents having research experience prior to residency (SD) | P value | |
|---|---|---|---|
| Yes (n = 20) | No (n = 77) | ||
| Patient Care | 2.19 (0.709) | 2.90 (0.842) | .001* |
| Medical Knowledge | 2.23 (0.700) | 2.82 (0.841) | .003* |
| Interpersonal and Communication Skills | 2.14 (0.766) | 2.88 (0.928) | .001* |
| Professionalism | 2.05 (0.851) | 2.81 (0.960) | .001* |
| Systems-Based Practice | 1.71 (0.786) | 2.62 (1.03) | .001* |
| Practice-Based Learning and Improvement | 2.35 (0.758) | 2.88 (0.769) | .008* |
| Composite | 2.11 (0.734) | 2.81 (0.870) | .001* |
*Difference is statistically significant @ P < .01.
Higher professionalism scores in residents without a history of volunteerism reached statistical significance. However, at the calculated effect size of d = .73, statistical power was limited to .57 with our available sample size of 44. Power analysis revealed that in order for an effect of this size to be detected reliably as significant at α = .05, a sample of 112 participants would be required. Similarly, the higher scores in systems-based practice for residents without work experience (prior careers excluded) reached statistical significance but a power analysis yielded a coefficient of only .24. Generating an effect size of this magnitude with a more suitable power coefficient of .80 would require a sample size of 318.
A power analysis for results pertaining the characteristics of having a prior career and research experience demonstrated higher predictive values. PGY1 residents who began graduate training after having a previous career scored significantly higher in the domain of interpersonal communication skills compared to PGY1 residents who had not had a prior career (x̅ = 3.04 vs 2.51, P = .019; d = .62, achieved power = .68). Previous career interns also scored significantly higher in system-based practice when contrasted with their non-career peers (x̅ = 2.76 vs 2.14, P = .015; d = .62, achieved power = .67). Power analysis revealed that in order for an effect of this size to be detected reliably as significant at α = .05, a sample of 152 participants would be required. In examining the influence of research experience, PGY1 residents with no research experience prior to residency scored higher in every domain when compared to PGY1 residents who participated in one or more such experiences. All of those differences were statistically significant at a level of P = .008 or lower. Effect sizes among the six domains ranged from d = .69 to d = 1.01. In all comparisons for this independent variable achieved power was >.77.
Discussion
Our study revealed two unique findings. The first is that residents with a prior career performed better than peers in the competency domains of Interpersonal and Communications Skills and Systems-Based Practice along with an overall higher clinical performance composite. This finding links identifiable applicant data to areas considered valuable during recruitment.2,9 The second was applicants reporting any participation in research were rated lower across all ACGME competency domains compared to peers. Prior studies in various specialties have demonstrated a lack of correlation between research and clinical performance with at least one study demonstrating a negative correlation in General Surgery. 10
Residents in our study who had a prior career scored significantly higher in the ACGME competency domains for Interpersonal and Communication Skills and the Professionalism. Although variable in nature, a prior career likely provides a very direct opportunity to develop skills across these domains through a structured environment that is not experienced by other applicants prior to residency. Previous studies demonstrate that program directors place Interpersonal and Communication skills only behind Clinical Skills and Medical Knowledge when ranking the importance of resident characteristics, 2 and the importance of team-based functioning abilities is critical. 11 Although a larger sample size with an improved power analysis is desirable, the significance of these findings in the study cohort warrant heightened consideration of prior careers during the application process for program directors seeking candidates with more advanced readiness skills in the areas of communication, interpersonal skills and systems-based practice.
In contrast, residents in our study with research experience prior to residency scored significantly lower in all six domains. The literature contains many studies demonstrating research as non-predictive for resident performance, and at least one study demonstrating a negative correlation between research experience and resident performance in General Surgery. 7 The cause or causes for the observed differences may include: applicants for both medical school and residency who are unsuccessful often obtain research experience to increase competitiveness; personal characteristics or personality types attracted to research differ from those that foster clinical excellence in Family Medicine; applicants with research history begin residency with less clinical exposure; or a number of other possibilities. Although reflecting on potential relationships between personal characteristics and subsequent readiness for residency, it is important to acknowledge we were unable to demonstrate causation through this study. It is also important to remember these differences between groups are within the context of applications for residency. The benefit of research participation and scholarship inquiries during residency are well documented to improve resident development and clinical performance.
We also explored several other life experience categories such as volunteering, work experience, academic productivity and student organizational engagement during medical school, but these failed to yield any significance in predicting resident performance. Although another study showed that Emergency Medicine residents with five or more presentations and publications outperformed their peers, 9 our study on Family Medicine residents did not yield the same result. Our results were more consistent with the conclusion that publication history is not a strong predictor of performance among residents. 6
Our study is predominantly limited by the nature of self-reported data in a highly competitive application process. ERAS data has been documented in the literature as being potentially inflated or even deceptive. 12 This likely contributes to a lack of significance in areas you might expect more predictive value. Our study is further limited by utilizing data from a single academic institution that may not be as predictive for community-based residencies. However, the mix of international and allopathic residents is representative of the national norms. This study focused on applicant readiness for starting a residency; further studies including data from additional institutions with a focus on residency outcomes and specialty satisfaction after graduation from residency would be valuable extensions to this analysis.
Although this study provides valuable insights into evaluating students with established prior careers and with a history of research experience, it validates the continued need to explore better ways of assessing for characteristics of a successful resident through the application and interview process. The application materials alone are unable to provide enough predictive value. As the trends toward virtual recruitment continue to evolve, more studies are needed to determine predictive values of various applicant assessment tools during the process.
Conclusion
This research study from a single academic-based United States Family Medicine residency demonstrates the history of a prior career among residency applicants is a predictor of stronger communications skills, systems-based practice skills and overall clinical competency assessments compared to peers. This study also concludes a history of research, while often looked upon favorably during application process, does not correlate with higher clinical assessments and as a group actually correlates with lower overall clinical assessment scores.
Footnotes
Author Contributions: M.B., B.M., and J.G. contributed to conception and design. B.M. contributed data acquisition. J.G. contributed statistical analysis. M.B., B.M., J.G., C.A., G.K. and Z.P. contributed to literature review, interpretation of data and drafting of manuscript. All authors approved the final manuscript for publication. All authors are in agreement to be accountable for all aspect of the work in ensuring that questions related to accuracy or integrity of any part of the work are appropriately investigated and resolved.
Availability of Data and Materials: The datasets sued and/or analyzed for this study are available from the corresponding author on reasonable request.
Ethics Approval and Consent to Participate: This study was determined to be exempt by the Indiana University School of Medicine Institutional Review Board. The need for informed consent is waived by Indiana University School of Medicine Institutional Review Board as it is secondary research. All methods were carried out in accordance with relevant guidelines and regulations.
DECLARATION OF CONFLICTING INTERESTS: The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
FUNDING: The author(s) received no financial support for the research, authorship, and/or publication of this article.
REFERENCES
- 1.Metro D., Talarico J., Patel R., Wetmore A. The resident application process and its correlation to future performance as a resident. Anesth Analg. 2005;100(2), pp.502‐505. [DOI] [PubMed] [Google Scholar]
- 2.Martin M., Salzberg L. Resident characteristics to evaluate during recruitment and interview: a Delphi study. Educ Prim Care. 2016;28(2), pp.81‐85. [DOI] [PubMed] [Google Scholar]
- 3.Porter S., Graves M. Resident selection beyond the United States medical licensing examination. J Am Acad Orthop Surg. 2017:25(6), pp.411‐415. [DOI] [PubMed] [Google Scholar]
- 4.Loh A., Tan J., Lee J., Koh G. Voluntary community service in medical school: a qualitative study on student leaders’ motivations, experiences, and outcomes. Med Teach. 2016;38(7), pp.683‐690. [DOI] [PubMed] [Google Scholar]
- 5.Paolino N., Artino A., Saguil A., Dong T., Durning S., DeZee K. Predicting medical school and internship success: does the quality of the research and clinical experience matter? Mil Med. 2015;180(suppl_4), pp.12‐17. [DOI] [PubMed] [Google Scholar]
- 6.Cavalcanti R., Detsky A. Publishing history does not correlate with clinical performance among internal medicine residents. Med Educ. 2010;44(5), pp.468‐474. [DOI] [PubMed] [Google Scholar]
- 7.Bhat R., Takenaka K., Levine B., et al. Predictors of a top performer during emergency medicine residency. J Emerg Med. 2015;49(4), pp.505‐512. [DOI] [PubMed] [Google Scholar]
- 8.Katzung K., Ankel F., Clark M., et al. What do program directors look for in an applicant? J Emerg Med. 2019;56(5), pp.e95‐e101. [DOI] [PubMed] [Google Scholar]
- 9.Lu W., Goolsarran N., Hamo C., Frawley S., Rowe C., Lane S. Teaching patient safety using an interprofessional team-based learning simulation model in residency training. MedEdPORTAL. 2016;12(1), pp.1-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Brothers T., Wetherholt S. Importance of the faculty interview during the resident application process. J Surg Educ. 2007;64(6), pp.378‐385. [DOI] [PubMed] [Google Scholar]
- 11.Seaburg L., Wang A., West C., et al. Associations between resident physicians’ publications and clinical performance during residency training. BMC Med Educ. 2016;16(1), pp.1-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Grover M., Dharamshi F., Goveia C. Deception by applicants to family practice residencies. Fam Med. 2001;33(6), pp.441‐446. [PubMed] [Google Scholar]