This cross-sectional study assesses if the rise in National Resident Matching Program–reported data is associated with a rise in verifiable, indexed publications from matched allopathic dermatology applicants from 2007 to 2018.
Key Points
Question
How does the number of indexed publications of matched dermatology residency applicants compare with the self-reported research numbers published by the National Resident Matching Program (NRMP)?
Findings
Between 2007 and 2018, the mean number of indexed publications per matched applicant rose from 1.6 to 4.7, while the NRMP mean number of research items rose from 5.7 to 14.7.
Meaning
Indexed publications compose a minority of self-reported NRMP research items; the percentage of research items attributed to indexed publications has remained consistent from 2007 to 2018.
Abstract
Importance
According to the National Residency Matching Program’s biennial Charting Outcomes in the Match (NRMP ChOM) reports, the mean number of research items of matched allopathic dermatology applicants has nearly tripled since 2007, rising from 5.7 to 14.7. Research items are self-reported by applicants and serve as an approximation of research output. Because the NRMP research items field is unverified and reported as an aggregate of several different research pursuits, it may not be an accurate representation of applicant research output.
Objective
To determine if the rise in NRMP-reported data is associated with a rise in verifiable, indexed publications from matched allopathic dermatology applicants from 2007 to 2018.
Design, Setting, and Participants
Cross-sectional study including a bibliometric analysis on accepted applicant research output among 2234 matched allopathic dermatology applicants, with a total of 6229 publications, in dermatology residency programs for the years 2007, 2009, 2011, 2014, 2016, and 2018.
Main Outcome and Measures
The primary outcomes were the mean number of peer-reviewed indexed publications and mean number of NRMP ChOM research items. Secondary outcomes assessed the quality of indexed publications by analyzing article type and journal of publication.
Results
From 2007 to 2018, the mean number of indexed publications per matched dermatology applicant increased from 1.6 to 4.7 (203% increase). Indexed publications consistently compose a minority of NRMP ChOM research items (28.8% across the 6 years of the study). Nonindexed research items increased at more than double the rate of indexed publications. Bibliometric analysis showed that all other types of publications are increasing at a rate of 6 to 9 times that of basic science publications, dermatology-related publications increased at 5 times the rate of non-dermatology publications, and publications in lower–impact factor dermatology journals increased at 4 times the rate of publications in higher–impact factor dermatology journals.
Conclusions and Relevance
This cross-sectional study provides data on the research output of matched dermatology applicants. Indexed publications compose a minority of NRMP research items. Medical student self-reports of research output may emphasize research quantity over quality.
Introduction
The National Residency Matching Program (NRMP) has published Charting Outcomes in the Match (ChOM), analyzing nationwide metrics of matched applicants, since 2007.1 In the NRMP ChOM, research output is approximated by the number of research items—an aggregate of abstracts, presentations, and publications. The mean number of research items per matched dermatology applicant nearly tripled from 2007 to 2018, rising from 5.7 to 14.7.1 Because this field is self-reported and unverified and combines several types of research activity into a single category, it is difficult to compare individual research experiences with nationwide metrics.2 We aimed to determine how the mean number of peer-reviewed indexed publications (IPs) per matched dermatology applicant compares with NRMP ChOM data from 2007 to 2018. We hypothesized that IPs constitute a minority of NRMP research items and that the growth in applicant research output could be disproportionately attributed to dermatology publications in lower–impact factor journals.
Methods
Allopathic medical students matching into Accreditation Council for Graduate Medical Education–accredited dermatology residencies during the academic years 2007, 2009, 2011, 2014, 2016, and 2018 were eligible for inclusion. These years were selected to align with the release of NRMP ChOM data. Applicant characteristics were obtained from individual dermatology residency program websites, social networking sites, and the American Society for Dermatologic Surgery mailing list.3,4,5 Medical school rankings were obtained from the Blue Ridge Institute for Medical Research and based on National Institutes of Health (NIH) funding awarded to a specific institution during a given year.6 The University Hospitals Cleveland Medical Center institutional review board exempted the study from review because it was determined to be not human subjects research.
Characteristics of matched applicant IPs were obtained from the Scopus database of peer-reviewed literature (Elsevier). Searches for IPs were filtered with no lower limit of publication date and an upper limit of the end of the calendar year corresponding to the matched applicant’s medical school graduation year. Higher–impact factor dermatology publications were those published in a dermatology journal with the 4 highest impact factors during the study period: Archives of Dermatology/JAMA Dermatology, Journal of the American Academy of Dermatology, Journal of Investigative Dermatology, or British Journal of Dermatology.7,8
Indexed publications were attributed to individual applicants based on Scopus author profiles. For matched applicants with multiple potential author profiles, all suspected profiles were searched for inclusion into the study. If a matched applicant appeared to have multiple surnames, all known names were investigated. If it was unclear whether a publication should have been attributed to a given individual, ResearchGate and Google Scholar profiles were cross-referenced. Author profiles that could not be attributed to matched applicants were excluded. Conference proceedings, abstracts, book chapters, errata, responses to the editor, correspondences, commentaries, and non-English publications were excluded.
Multivariable linear regression models explored the associations between each publication characteristic and the average number of publications per applicant. Trends were investigated with the inclusion of interaction terms using cross products. Interaction terms allow us to determine whether publication trends differ among the categories we investigated (IPs vs nonindexed research items [NIRIs]; case reports/series vs clinical original vs clinical review vs basic science; dermatology related vs non–dermatology related; and lower–impact factor dermatology journal vs higher–impact factor dermatology journal). Trend line comparisons were conducted with parameter estimate contrast statements. Statistical significance was set at 5%, and all analyses were performed with SAS, version 9.4 (SAS Institute).
Results
A total of 2234 matched applicants from 140 Accreditation Council for Graduate Medical Education–accredited dermatology residencies with a combined 6229 IPs met inclusion criteria (eFigure in the Supplement). This 618 program-year sample represents 100% of residency programs and 91% of total dermatology residency program-years (eTables 1 and 2 in the Supplement).9
The majority of matched applicants were female (1419 of 2234 [63.5% across all cohort years]), did not have a PhD degree (2102 of 2234 [94.1%]), attended medical schools ranked 51 and greater in NIH funding (1106 of 2234 [49.5%]), and graduated from medical school programs in the South (837 of 2234 [37.5%]). There was an increase in the percentage of matched applicants from medical schools ranked 51 and greater in NIH funding and a decrease in the percentage of students from programs ranked 26 to 50. There were no additional significant differences between cohort years (eTable 3 in the Supplement).
From 2007 to 2018, the mean number of NRMP ChOM research items (5.7 vs 14.7; 158% increase), IPs (1.6 vs 4.7; 203% increase), and NIRIs (4.2 vs 10.0; 141% increase) all increased. The mean number of NIRIs per applicant has increased at double the rate of mean IPs per applicant (IPs: increase of 0.24 per year; NIRIs: increase of 0.52 per year; P = .005). The percentage of NRMP ChOM research items attributed to IPs remained consistent (Figure 1).
Figure 1. Comparison of Research Output Measures in Matched Dermatology Residency Applicants.
The publication trends significantly differed among the nonindexed research items (NIRIs) and indexed publications (IPs) (P = .005). NIRIs increased at 0.52 items per year (P = .005), and IPs increased at 0.24 publications per year (P = .01). NRMP indicates National Residency Matching Program.
The mean numbers of case reports/series (206% increase), clinical original (279% increase), and clinical review (425% increase) IPs per matched applicant have increased. All other types of publications increased at a faster rate compared with basic science publications (P = .01) (Figure 2A). The proportion of basic science publications relative to other types of publications decreased over the study period, falling from the most common type of publication in 2007 (38.8%) to the least common type of publication in 2018 (20.4%) (P < .001). Clinical-review publications were the least common type of publication in 2007 (13.4%), but the second most common in 2018 (23.7%) (Table).
Figure 2. Trends in Mean Publication Characteristics per Matched Dermatology Residency Applicant.
A, The publication trends significantly differed according to type of publication (P = .01) and impact of dermatology journal publication (P = .005). Case report/case series increased at 0.06 publications per year (P = .001), clinical original increased at 0.09 publications per year (P < .001), and clinical reviews increased at 0.07 publications per year (P <.001); however, basic science remained relatively unchanged (slope, 0.01; P = .46). B, Higher–impact factor dermatology journals increased at 0.04 publications per year (P = .002), while lower–impact factor dermatology journals increased at 0.17 publications per year (P = .008). Higher–impact factor dermatology journals were defined as a related specific publication in either Archives of Dermatology/JAMA Dermatology, Journal of the American Academy of Dermatology, Journal of Investigative Dermatology, or British Journal of Dermatology.
Table. Characteristics of Peer-Reviewed Indexed Publications of Matched Dermatology Residency Applicants by Graduation Year.
| Characteristic | No. (%) | P valuea | |||||
|---|---|---|---|---|---|---|---|
| 2007 | 2009 | 2011 | 2014 | 2016 | 2018 | ||
| Total No. of publications | 474 | 769 | 744 | 1102 | 1361 | 1779 | NA |
| Typeb | |||||||
| Basic science | 183 (38.8) | 326 (43.5) | 263 (35.6) | 288 (26.5) | 310 (23.2) | 353 (20.4) | <.001 |
| Case reports | 105 (22.3) | 138 (18.4) | 182 (24.6) | 269 (24.8) | 334 (25.0) | 399 (23.0) | |
| Clinical | |||||||
| Original | 121 (25.6) | 144 (19.2) | 156 (21.1) | 272 (25.1) | 409 (30.6) | 572 (32.9) | |
| Review | 63 (13.4) | 142 (18.9) | 138 (18.7) | 257 (23.7) | 284 (21.2) | 411 (23.7) | |
| Dermatology, yesc | 246 (51.9) | 407 (52.9) | 457 (61.4) | 715 (64.9) | 908 (66.7) | 1264 (71.1) | <.001 |
| High–impact factor dermatology journal, yesd | 70 (28.5) | 98 (24.1) | 126 (27.6) | 177 (23.8) | 199 (21.9) | 249 (19.7) | .002 |
Abbreviation: NA, not applicable.
Values obtained from χ2 tests.
Publications categorized as Other: 2007 (2), 2009 (19), 2011 (5), 2014 (16), 2016 (24), 2018 (46).
Publications related to dermatology.
Defined as a dermatology-related publication in Archives of Dermatology/JAMA Dermatology, Journal of the American Academy of Dermatology, Journal of Investigative Dermatology, or British Journal of Dermatology.
The mean number of dermatology-related publications per applicant has increased at 5 times the rate of non–dermatology-related publications (increase per year of 0.20 vs 0.04; P = .005). Dermatology-related publications have increased at 0.20 publications per year (P < .001), while non–dermatology-related publications remained relatively unchanged (slope = 0.04; P = .20). Among dermatology-related publications, the rate of lower–impact factor dermatology publications has increased at 4 times the rate of higher–impact factor dermatology publications (P = .005). The percentage of dermatology-related publications attributed to higher–impact factor dermatology journals has decreased by 29% (P = .03) (Figure 2B).
Discussion
From 2007 to 2018, the mean number of IPs for accepted allopathic dermatology residency applicants increased from 1.6 to 4.7, while NRMP ChOM mean research items increased from 5.7 to 14.7. We found that IPs constitute approximately 30% of NRMP ChOM research items, supporting our hypothesis that IPs compose a minority of NRMP research items.
The increase in applicant research output is disproportionately attributable to dermatology-specific publications and publications in lower–impact factor dermatology journals, supporting our secondary hypothesis. We speculate, based on these findings combined with the lack of growth in basic-science publications, that the rise in publication quantity may be driven by lower-quality publications. This is important in light of previous findings suggesting that both publication quantity and quality as a medical student are associated with future academic pursuits.10,11 However, it is important to note that research quality is subjective, and we define quality in this study as a surrogate for a sustained research experience resulting in publication.
Limitations
Study limitations include relying on names provided by dermatology residency program websites and American Society for Dermatologic Surgery mailing lists. We may have missed publications owing to spelling errors or name changes. Our selection of a publication cutoff date corresponding to the end of the calendar year of the matched applicant’s medical school graduation year may introduce bias against studies that take longer to publish. This overall rise in research output that we show may be partially attributed to applicants who completed research fellowships.12 Additionally, authorship on a scholarly work may not always accurately reflect the magnitude of academic pursuits, and the eventual influence of a given work may be attributed more to the mentor than the student.
It is important to consider that the increase in applicant research items is not limited to dermatology, but rather reflected across all specialties in the NRMP ChOM.1 How the rate of increase in IPs among matched dermatology applicants compares with that of other specialties is unknown. Furthermore, delineating the increase in applicant NIRIs could be a topic of future investigation.13
Conclusions
In this cross-sectional study, IPs composed a minority of NRMP research items. The large increase in applicant research quantity from 2007 to 2018 can be attributed to non–basic science publications and publications in lower–impact factor dermatology journals.
eFigure. Study Consort Diagram
eTable 1. Summary of Residency Program Sampling Success
eTable 2. Detailed Residency Program Sampling Success
eTable 3. Characteristics of Matched Allopathic Dermatology Applicants by Graduation Year
References
- 1.National Resident Matching Program . Main residency match data and reports. Accessed May 11, 2021. https://www.nrmp.org/main-residency-match-data/
- 2.Zhao J, Tegtmeyer K, Lio PA. Reducing academic misrepresentation in the dermatology residency application process: a need for better reporting of research output. J Am Acad Dermatol. 2020;83(5):e367-e368. doi: 10.1016/j.jaad.2020.06.1017 [DOI] [PubMed] [Google Scholar]
- 3.American Society for Dermatological Surgery . Find a dermatologic surgeon. Accessed September 29, 2020. https://www.asds.net/find-a-dermatologic-surgeon/advanced/true
- 4.US Census Bureau . Geographic Areas Reference Manual. US Department of Commerce; 1994:8-7. Accessed May 10, 2021. https://www.census.gov/geo/reference/garm.html [Google Scholar]
- 5.Internet Archive Project . Internet Archive Wayback Machine. Published 2020. Accessed May 10, 2021. https://archive.org/web/
- 6.Blue Ridge Institute for Medical Research . NIH Ranking of Each School of Medicine. Published 2019. Accessed May 10, 2021. http://www.brimr.org/NIH_Awards/2018/NIH_Awards_2018.htm
- 7.Clarivate Analytics . InCites Journal Citation Reports. Published 2020. Accessed May 10, 2021. https://jcr.clarivate.com
- 8.SCIMAGO Institutions . Scimago Journal Rank: Dermatology. 2019. Accessed May 10, 2021. https://www.scimagojr.com/journalrank.php?category=2708
- 9.Accreditation Council for Graduate Medical Education . ACGME Accreditation Data System. Accessed May 10, 2021. https://apps.acgme.org/ads/Public
- 10.Shi CR, Tung JK, Nambudiri VE. Demographic, academic, and publication factors associated with academic dermatology career selection. JAMA Dermatol. 2018;154(7):844-846. doi: 10.1001/jamadermatol.2018.0743 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Stephens MR, Barbieri JS, Lipoff JB. Predicting future dermatology academic productivity from medical school publications. J Am Acad Dermatol. 2020;83(2):624-626. doi: 10.1016/j.jaad.2019.08.076 [DOI] [PubMed] [Google Scholar]
- 12.Akhiyat S, Cardwell L, Sokumbi O. Why dermatology is the second least diverse specialty in medicine: how did we get here? Clin Dermatol. 2020;38(3):310-315. doi: 10.1016/j.clindermatol.2020.02.005 [DOI] [PubMed] [Google Scholar]
- 13.Maverakis E, Li C-S, Alikhan A, Lin TC, Idriss N, Armstrong AW. The effect of academic “misrepresentation” on residency match outcomes. Dermatol Online J. 2012;18(1):1. [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
eFigure. Study Consort Diagram
eTable 1. Summary of Residency Program Sampling Success
eTable 2. Detailed Residency Program Sampling Success
eTable 3. Characteristics of Matched Allopathic Dermatology Applicants by Graduation Year