What Are the Trends in Research Publication Misrepresentation Among Orthopaedic Residency and Fellowship Applicants From 1996 to 2019? A Systematic Review

Abstract

Background

Because research experience is increasingly important in ranking orthopaedic residency and fellowship applicants, determining the accuracy of candidates reporting their scholarly activity is essential. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies.

Questions/purposes

In this systematic review, we asked: (1) What percentage of research publications are misrepresented among orthopaedic residency and fellowship applicants? (2) What percentage of applications contain one or more example of academic misrepresentation? (3) Is research misrepresentation associated with any individual applicant characteristics? (4) What is the publication status of articles listed by applicants as having been submitted to journals?

Methods

A systematic review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines. PubMed, EBSCOhost, Medline, and Google Scholar electronic databases were searched on March 10, 2022, to identify all studies that evaluated research misrepresentation in orthopaedic residency and fellowship applications between January 1, 1995, and March 1, 2022. Articles were included if full-text articles in English were available and the study reported on research misrepresentation among orthopaedic residency or fellowship applicants. Studies investigating nonorthopaedic publications, systematic reviews, case studies, duplicate studies among databases, and gray literature were excluded. Two reviewers independently evaluated the quality of included studies using the Methodological Index for Nonrandomized Studies (MINORS) tool. This is a validated assessment tool that grades noncomparative studies from 0 to 16 and studies with control groups from 0 to 24, based on eight criteria related to study design, outcomes assessed, and follow-up. All included articles were noncomparative studies, so the maximum score here was 16, with higher scores indicating better study quality. The mean MINORS score was 13 ± 1 in the studies we included. The final analysis included 10 studies with 5119 applicants. Eight studies evaluated orthopaedic residency applicants and two evaluated fellowship applicants. The applicant classes ranged from 1996 to 2019. Research misrepresentation was defined among studies as nonauthorship of an existing article, claimed authorship of a nonexistent article, or incorrect listing of authorship order for an existing article. Each study’s findings and definition of research misrepresentation were considered to allow for a discussion of overall trends. The percentage of misrepresentation was further broken down by the misrepresentation type. Applicant characteristics and destination of submitted articles were also evaluated. Given the potential overlap between applicants among the studies, no pooled analysis was conducted, and results are presented as a narrative summary.

Results

The percentage of overall publication misrepresentation was estimated to range between 1% (13 of 1100) and 21% (27 of 131), with more-recent studies reporting a lower proportion of overall articles misrepresented. Most studies we found claimed that authorship of a nonexistent article was the most common type of misrepresentation. Nonauthorship of an existing article and incorrect authorship order were less common. The percentage of applications with at least one misrepresentation was approximately 20% between 1998 and 2017. Most studies found no applicant characteristics, such as match outcomes, demographic markers, or academic records, that were consistently associated with a higher odds of the candidate misrepresenting his or her research credentials. Finally, approximately half of the articles listed as submitted to journals went on to publication, with one-third going to a different journal with a lower Impact Factor.

Conclusions

Our systematic review found that the percentage of overall publication misrepresentations among orthopaedic residency and fellowship applicants has generally been low over the past 20 years. However, approximately one-fifth of applications had at least one research misrepresentation, with 2% having multiple misrepresentations on reported publications. There were no consistent applicant characteristics associated with higher odds of research misrepresentation. Additionally, most of the articles listed as submitted to journals for publication were ultimately published.

Clinical Relevance

Although the decrease in overall publication misrepresentation is encouraging, our finding that one-fifth of applicants have research misrepresentation is a cause for concern. In light of a continually evolving application process, orthopaedic residency and fellowship programs must ensure there is integrity related to information that is self-reported by applicants. These findings also serve to encourage faculty members involved in the application screening and decision process to limit biases related to applicant demographics perceived to be associated with a high odds of misrepresentation. Furthermore, governing agencies and program leadership should evaluate methods of verifying unpublished work and provide opportunities for applicants to give publication updates throughout the application cycle.

Introduction

With increasing competition for orthopaedic surgery residency and fellowship programs, a major domain for applicants to distinguish themselves is research experience [6, 18, 19, 28]. Program directors of orthopaedic residency and fellowship programs value research experience highly as a marker of an applicant’s knowledge of the scientific process, capacity to manage a project through completion, and the potential to continue pursuing research in their program [4, 30].

Corresponding to the emphasis orthopaedic program directors place on scholarly activities, the mean number of publications and presentations of orthopaedic surgery residency applicants doubled between 2007 and 2014 [11], and this trend is likely to continue as the United States Medical Licensing Examination Step 1 converted to pass/fail scoring [15, 29]. Because the timeline to publication of research projects may be lengthy, many projects may not be published before application deadlines. In turn, programs used by orthopaedic residency applicants such as the Electronic Residency Application Service have options to list research projects as published, in-press, accepted, provisionally accepted, or submitted to journals [2]. Given the importance of research experience in ranking orthopaedic residency and fellowship applicants, program selection committees must receive accurate information about applicants’ research involvement. Whether intentional or by accident, some candidates may exaggerate their scholarly involvement and achievements, which is a form of academic misrepresentation. Common examples of misrepresentation include nonauthorship of an existing article, claimed authorship of a nonexistent article, and erroneously elevating one’s rank in the authorship order of an existing article.

Research misrepresentation in the field of orthopaedics has been a concern for some time. Dale et al. [10] published the first investigation of research misrepresentation among orthopaedic residency applicants that we are aware of in 1999 and found that 18% (14 of 76) of publications were misrepresented. Over the past 20 years, at least nine more studies evaluating research misrepresentation have been published by orthopaedic residency and fellowship programs, with mixed results [3, 8, 9, 12, 13, 22, 24, 26, 33]. The inconsistent evidence makes it difficult to draw meaningful conclusions from these individual studies. Furthermore, all were performed at a single academic institution that may represent a biased study population and limit the generalizability of the findings. A systematic review across multiple institutions and applicant classes is therefore warranted to evaluate the quality of evidence and provide a comprehensive summary. This will allow further characterization of research misrepresentation and clarify possible associations between research misrepresentation and certain applicant characteristics.

We therefore asked: (1) What percentage of research publications are misrepresented among orthopaedic residency and fellowship applicants? (2) What percentage of applications contain one or more example of academic misrepresentation? (3) Is research misrepresentation associated with any individual applicant characteristics? (4) What is the publication status of articles listed by applicants as having been submitted to journals?

Materials and Methods

Search Strategy and Information Sources

We conducted a comprehensive review of the PubMed, EBSCOhost, Medline, and Google Scholar electronic databases on March 10, 2022, to identify all studies that evaluated research misrepresentation in orthopaedic residency and fellowship applications between January 1, 1995, and March 1, 2022. The following keywords and Medical Subject Heading terms were used in combination with the “AND” or “OR” Boolean operators: “Orthopedics [MeSH],” “Internship and Residency [MeSH],” “fellowship,” “application,” “misrepresentation,” “unpublished,” “ghost publication,” and “self-reported publication.”

Eligibility Criteria

Articles were included if full-text articles in English were available and the study described the publication status or misrepresentation of research among orthopaedic residency applicants. Fellowship applicants were also included, given the paucity of orthopaedic research concerning publication misrepresentation. Research misrepresentation was defined as nonauthorship of an existing article, claimed authorship of a nonexistent article, and incorrect authorship order of an existing article. Articles were considered available if the full-text article was accessible through the journal website or through our institution’s interlibrary loan network. The following were excluded from our analysis: studies investigating research misrepresentation among nonorthopaedic applicants, systematic reviews, duplicate studies among databases, case studies, and gray literature such as abstracts and articles on preprint servers.

Study Selection

In accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis guidelines, two reviewers (RJB and AHK) independently assessed the eligibility of each article to be included in our review. Disagreements were discussed, and a third independent reviewer (CJH) was consulted to achieve consensus. The initial query yielded 528 publications, which were then screened for appropriate studies that aligned with the purpose of our review. After removal of duplicates and reading each abstract, 35 studies were selected for further consideration. The full text of each article was then reviewed; 10 fulfilled our inclusion criteria. A thorough review of each study’s reference list did not yield any additional articles (Fig. 1).

Fig. 1.

This PRISMA flow diagram demonstrates the search strategy and inclusion criteria. Criteria included English-language studies, publication data from 1995 to 2022, and studies that described the publication status or misrepresentation rates of research of orthopaedic residency or fellowship applicants. All results that were not full-text articles or gray literature were excluded.

Data Collection

The following data were extracted from each study: study design, data source, sample size, gender, training level and years of applicant classes, medical degree (Doctor of Medicine, Doctor of Osteopathic Medicine, or international medical graduate), research misrepresentation criteria, percentage of publications and applicants with research misrepresentation, and follow-up time until analysis.

Study Characteristics

Our analysis included 10 studies with 5119 applicants (Table 1). Eight studies evaluated orthopaedic residency applicants and two evaluated fellowship applicants. The applicant classes ranged from 1996 to 2019, with most studies evaluating applicant classes in the past 10 years. The follow-up time was defined as the time from application cycle year to investigator verification of publications. This ranged from 2 to 5 years among included studies. All were retrospective single-institution studies and evaluated articles listed as “published,” “accepted,” “in-press,” or “submitted.” Given the potential overlap between candidates among the studies because individuals applied to numerous programs, no pooled analysis was conducted, and the results are presented as a narrative summary.

Table 1.

Characteristics of articles included in the final analysis

Article	Study design	Data source	Training level	Applicant classes	Sample size, n	% men, n	% applicant degrees, n	Time until follow-up analysisa	Research misrepresentation criteria	MINORS scoreb
Barnes [3]	Retrospective	Single institution	Residency	2016	573	n/a	n/a	30 months	Nonauthorship on an existing article	12
Collier and Janney [8]	Retrospective	Single institution	Residency	2019	519	83 (430)	93 (484) MD 6.7 (35) DO	24 months	Nonauthorship on an existing article Incorrect authorship order	13
Cortez et al. [9]	Retrospective	Single institution	Fellowship	2013-2017	435	89 (389)	78 (338) MD 8 (35) DO 14 (62) IMG	24 months	Nonauthorship on an existing article Claimed authorship of a nonexisting article Incorrect authorship order Incorrect journal listed	13
Dale et al. [10]	Retrospective	Single institution	Residency	1998	213	n/a	n/a	n/a	Nonauthorship on an existing article Claimed authorship of a nonexisting article	11
el Beaino et al. [12]	Retrospective	Single institution	Residency	2017	264	80 (211)	91 (241) USMG 9 (23) IMG	n/a	Nonauthorship of an existing article Claimed authorship of a nonexisting article Incorrect authorship order	11
Freshman et al. [13]	Retrospective	Single institution	Residency	2013-2015	1303	84 (1090)	99 (1288) USMG 1 (15) IMG	24 months	n/a	13
Konstantakos et al. [22]	Retrospective	Single institution	Residency	2005-2006	396	n/a	n/a	24 months	Nonauthorship of an existing article Claimed authorship on a nonexisting article	13
Lemme et al. [24]	Retrospective	Single institution	Residency	2014	563	83 (468)	n/a	5 years	n/a	13
Meeks et al. [26]	Retrospective	Single institution	Residency	2017	573	n/a	92 (526) USMG 8 (47) IMG	24 months	Nonauthorship of an existing article Claimed authorship on a nonexisting article Incorrect authorship order	13
Patel et al. [33]	Retrospective	Single institution	Fellowship	1996-1998	280	90 (253)	n/a	24 months	Claimed authorship on nonexisting article Incorrect authorship order	13

^aFollow-up time was defined as the time from application cycle year to investigator verification of publications.

^bThe MINORS scale is from 0 to 16 for noncomparative studies and 0 to 24 for studies with control groups. This study consisted of only noncomparative studies, so the maximum score is 16. Higher scores represent better study quality. USMG = United States medical graduate; DO = Doctor of Osteopathic Medicine; IMG = international medical graduate; MD = Doctor of Medicine; MINORS = Methodological Index for Nonrandomized Studies

Risk of Bias in Individual Studies

Two reviewers (RJB and CJH) independently evaluated the quality of included studies using the Methodological Index for Nonrandomized Studies (MINORS) tool [36]. This is a validated assessment tool that grades noncomparative studies from 0 to 16 and studies with control groups from 0 to 24 based on eight criteria related to study design, outcomes assessed, and follow-up. Across these domains, each item is scored 0 if not reported, 1 when reported but inadequate, and 2 when reported and adequate. All studies we evaluated were noncomparative studies, and the maximum MINORS score here was 16, with higher scores indicating better study quality. Any discrepancies in grading were resolved by discussion with a third reviewer (AJA). The main reason for lost points among the included studies was a lack of prospective data collection. The mean MINORS score was 13 ± 1.

Primary and Secondary Study Outcomes

Our primary study goal was to determine what percentage of research publications was misrepresented among residency and fellowship applications. To achieve this, we considered each study’s findings and definition of research misrepresentation to allow for a discussion of overall trends. Six studies reported the percentage of research publications that were misrepresented [3, 8, 9, 10, 22, 26]. Publication misrepresentation was further broken down by the type of misrepresentation, when provided. Types of misrepresentation among studies included nonauthorship of a published article in which authorship was claimed, listing an article that could not be located in the reported journal or elsewhere, and incorrectly listing authorship order.

Our secondary study goal was to determine what percentage of applications contain one or more research misrepresentation. Additionally, we evaluated whether certain applicant characteristics were consistently associated with research misrepresentation and the publication status of articles listed as submitted to journals. To achieve this, we considered each study’s findings and extracted data to allow for a discussion of overall trends. Six articles reported the percentage of applicants who had one or more misrepresentations of their research [9, 10, 12, 22, 26, 33]. Five studies evaluated whether applicant characteristics were associated with a higher percentage of research misrepresentation [8, 9, 12, 13, 26]. Applicant characteristics included age, gender, United States medical graduate or international medical graduate, United States Medical Licensing Examination Step 1 and Step 2 scores, Alpha Omega Alpha membership, medical school attendance at a National Institutes of Health Top 40 research institution, research years, additional advanced degree, and residency match outcome. Four studies reported on the destinations of articles listed as submitted to journals [3, 9, 13, 24]. The percentage of submitted articles that were published at the time of follow-up analysis was extracted.

Results

Percentage of Publications With Research Misrepresentation

The percentage of overall publication misrepresentation was estimated to range between 1% (13 of 1100) and 21% (27 of 131) [3, 8-10, 22, 26]. This has generally decreased over time, with more-recent studies reporting a lower proportion of overall articles misrepresented (Fig. 2). Three of the four studies reporting proportions for each type of misrepresentation found claimed authorship of a nonexistent article to be the most common [3, 8, 22] (Table 2). Nonauthorship of an existing article was less common across studies, with an estimated range of 0.1% (one of 1100) to 12% (12 of 104). Three large studies [8, 9, 26] found incorrect authorship order was relatively rare, with an estimated range of 1% (10 of 1100) to 3% (41 of 1504). However, one study [26] noted that incorrect authorship order was responsible for 77% (10 of 13) of recorded cases.

Fig. 2.

This graph demonstrates the percentage of research articles identified as misrepresented over time.

Table 2.

Percentage of publications with research misrepresentation

Study (applicant year)	Nonauthorship on an existing article, % (n)	Claimed authorship of a nonexisting article, % (n)	Incorrect authorship order, % (n)	Overall, % (n)
Collier and Janney [8] (2019)	0.9 (13 of 1315)	5 (63 of 1315)	3 (35 of 1315)	8 (111 of 1315)
Cortez et al. [9] (2013-2017)	n/a	3 (44 of 1504)	3 (41 of 1504)	6 (85 of 1504)
Meeks et al. [26] (2017)	0.1 (1 of 1100)	0.1 (2 of 1100)	1 (10 of 1100)	1 (13 of 1100)
Barnes [3] (2016)	3 (8 of 309)	8 (26 of 309)	n/a	11 (34 of 309)
Konstantakos et al. [22] (2007)	12 (12 of 104)	14 (15 of 104)	n/a	21 (27 of 131)
Dale et al. [10] (1999)	n/a	n/a	n/a	18 (14 of 76)

Percentage of Applicants With Research Misrepresentation

Although one study [26] found that only 4% (12 of 323) of applicants misrepresented a scholarly article, most studies reported the percentage of applications with one or more misrepresentations to be near 20% between 1998 and 2017 (Fig. 3). Most of these studies did not report proportions for each type of misrepresentation for applicants. However, two studies [12, 26] with similar sample sizes differed in the proportion of applicants with incorrect authorship order. One study [26] evaluating applicants from the 2016 to 2017 academic year reported that 3% (10 of 313) of applicants incorrectly listed authorship order. In contrast, a similar study [12] evaluating applicants from 2017 found incorrect authorship among 15% of applicants (40 of 264). The remaining types of misrepresentation were not consistently reported. Four studies found examples of an applicant with more than one publication misrepresentation [9, 12, 26, 33]. However, three of the four studies [12, 26, 33] found this to be relatively rare, with an estimated range of 0.3% (one of 323) to 2% (six of 264).

Fig. 3.

This graph demonstrates the percentage of applicants who had at least one misrepresentation of their research over time.

Association Between Research Misrepresentation and Applicant Characteristics

Most studies found no applicant characteristics, such as match outcomes, demographic markers, or academic records, that were consistently associated with a higher odds of the candidate misrepresenting his or her research credentials (Table 3) [8, 9, 12, 13, 26]. Two studies [13, 26] found a higher percentage of misrepresentation among international medical graduates. Meeks et al. [26] reported misrepresentation among 11% (five of 47) of international medical graduates compared with 1% (seven of 526) of United States medical graduates (p = 0.03). Similarly, Freshman et al. [13] found a greater number of unverified publications among international graduates (2.3 ± 3.9) than among United States medical graduates (0.9 ± 1.2 [95% CI -0.405 to -2.397]; p < 0.01). However, other studies found no association between international medical graduate applicants and research misrepresentation [9, 12]. One study found applicants who had taken a year off from their medical training for research were associated with more unverified publications (1.15 ± 1.385) than applicants with uninterrupted training (0.9 ± 1.8 [95% CI 2.053 to 2.756]; p < 0.001) [13].

Table 3.

Studies evaluating the association between applicant characteristics and misrepresentation

Study	Key findings	Characteristics assessed
Collier and Janney [8]	No difference in the percentage of misrepresentation between matched and unmatched orthopaedic residency applicants	Match outcome
Freshman et al. [13]	IMGs had a higher number of unverified publications (2.3 ± 3.9 versus 0.9 ± 1.2 [95% CI -0.405 to -2.397]; p < 0.01) Research year applicants had a higher number of unverified publications (0.9 ± 1.8 [95% CI 2.053 to 2.756]; p < 0.001) The remaining factors were not associated with a residency applicant having one or more misrepresentation	Age, gender, USMLE Step 1 score, USMLE Step 2 score, AOA membership, USMG or IMG, attended an NIH Top 40 research institution, additional advanced degree, and research year
Cortez et al. [9]	None of the explored factors were associated with a fellowship applicant having one or more misrepresentation	Age, gender, USMLE Step 1 score, AOA membership, type of medical degree, USMG or IMG, attended an NIH Top 40 research institution, and additional advanced degree
el Beaino et al. [12]	None of the explored factors were associated with a residency applicant having one or more misrepresentation	Gender, USMLE Step 1 score, USMLE Step 2 score, AOA membership, USMG or IMG, and additional advanced degree
Meeks et al. [26]	Holding an advanced degree was not associated with misrepresentation. However, IMGs had a higher percentage of misrepresentation than USMGs (11% [5 of 47] versus 1% [7 of 526]; p = 0.03)	USMG or IMG and additional advanced degree

IMG = international medical graduate; USMLE = United States Medical Licensing Examination; AOA = Alpha Omega Alpha; USMG = United States medical graduate; NIH = National Institutes of Health.

Publication Status of Articles Listed as Submitted to Journals

Between 40% (280 of 693) and 66% (1071 of 1636) of articles were published by the time of each study’s follow-up analysis (Table 4) [3, 9, 13, 24]. Two studies found that of eventually published “submitted” articles, approximately two-thirds were published in the same journal the applicant listed and one-third in a different journal [9, 13]. Of note, compared with the listed journal of submission upon application, articles that were published by a different journal were more likely to have a lower Impact Factor (Clarivate), with Freshman et al. [13] reporting Impact Factors of 3.0 ± 2.5 compared with 1.5 ± 2.7 (p < 0.001) and Cortez et al. [9] reporting Impact Factors of 3.91 ± 0.79 compared with 0.97 ± 0.13 (p < 0.01).

Table 4.

Studies evaluating the destinations of articles listed as submitted to journals

Study	Submitted and eventually published, % (n)	Submitted and published in the original journal listed, % (n)	Submitted and published in a different journal than listed, % (n)	Submitted and remain unpublished, % (n)
Barnes [3]	40 (280 of 693)	n/a	n/a	n/a
Cortez et al. [9]	64 (446 of 696)	42 (294 of 696)	22 (152 of 696)	28 (197 of 696)
Freshman et al. [13]	66 (1071 of 1636)	43 (709 of 1636)	22 (362 of 1636)	35 (565 of 1636)
Lemme et al. [24]	63 (318 of 501)	n/a	n/a	n/a

Data are presented as % (number of submitted articles with the trait mentioned, divided by the total number of papers listed as submitted).

Discussion

With orthopaedic residency and fellowship applications becoming increasingly competitive, some applicants may embellish their scholarly activities to distinguish themselves. Because program selection committees must receive accurate information to properly stratify applicants, we performed a systematic review to assess the frequency of academic misrepresentation and determine whether certain applicant characteristics were more associated with misrepresentation. Our analysis found that most orthopaedic surgery publications were accurately reported. Studies over the past 10 years estimated the proportion of overall publication misrepresentation to be between 1% (13 of 1100) and 11% (34 of 309). Claimed authorship of a nonexistent article was the most commonly reported type of misrepresentation, followed by nonauthorship on an existing article. Incorrect authorship order was relatively infrequent. Although it is encouraging to see a decrease in overall publication misrepresentation, our finding that approximately 20% of applicants have a misrepresentation is a cause for concern. Although studies consistently found candidates were unlikely to have greater than one misrepresentation in their applications, any case of misrepresentation should be carefully scrutinized when evaluating prospective candidates. No applicant characteristics were consistently associated with a higher odds of misrepresentation. Finally, approximately half of the articles listed as submitted to journals went on to publication, with one-third going to a different journal with a lower Impact Factor. Based on our findings, faculty involved in the selection process should be aware that most publications are accurately listed, and the proportion of overall publication misrepresentation has generally decreased over time. However, one-fifth of applicants will have a publication misrepresentation, and when misrepresentation does occur, it is most often claimed authorship of a nonexistent article. Other types of misrepresentation, such as incorrectly listing authorship order, are generally less common. Programs should also avoid using demographic characteristics to determine an applicant’s odds of research misrepresentation, given the inconsistent findings across studies. Lastly, faculty should evaluate articles listed as submitted to journals with caution, because these frequently remain unpublished or published in less impactful journals than originally intended.

Limitations

First, the present study did not factor in potential heterogeneity among orthopaedic residency and fellowship programs, in which certain institutions may have designated research tracks or a reputation for academic scholarship, leading to selection bias for research-oriented applicants compared with the overall applicant pool. Our analysis included data across multiple institutions and applicant classes to minimize selection bias and improve external validity. Another limitation relates to the inclusion of fellowship applicants in this review. Fellowship applicants generally have more publications than residency applicants and, consequently, more chances for misrepresentation [6, 20]. However, a meta-analysis evaluating residency and fellowship applicants across multiple specialties found no difference in the percentages of research misrepresentation between the two groups [38]. Because the number of misrepresentation studies evaluating orthopaedic applicants was limited, it was reasonable to evaluate them as a combined group.

The inclusion of older studies in our analysis should also be considered. Many of these studies were conducted before substantial changes were made to the Electronic Residency Application Service section process and may therefore provide less insight into the current state of research misrepresentation [10, 22, 33]. Still, these studies provide valuable information on publication misrepresentation across multiple institutions and timepoints for comparison across studies. The criteria to define misrepresentation also varied across studies, potentially leading to inconsistent reporting of the percentage of research misrepresentation. Most studies considered nonauthorship of an existing article and claimed authorship of a nonexistent article as examples of misrepresentation. A handful of studies also included incorrect authorship order, which may lead to inconsistent results [8, 9, 26]. In 2010, a meta-analysis of 18 studies found that variance in research misrepresentation decreases when uniform inclusion criteria are applied [38].

Another important consideration is the timeframe of follow-up analysis. Two studies did not list the time period until follow-up analysis [10, 12]. A short follow-up period may not account for articles that were eventually published after lengthy revisions and incorrectly classify these articles as misrepresentation. Additionally, some articles classified as misrepresentations may have been honestly recorded by the candidates, because changes to the author listing may have occurred between submitting and publishing the scholarly work. However, this situation may be less likely because no studies reported examples of an applicant incorrectly claiming a lower authorship position than recorded. The lack of a comparison group among reported studies must also be noted. Future studies should evaluate candidates across multiple specialties to determine whether these results are consistent across disciplines.

Finally, the present study did not include research posters and podium presentations. These are important measures of scholarly productivity that most orthopaedic applicants will include on their applications [5, 6]. The Electronic Residency Application Service requires conference details including name, location, and date to be listed for each presentation. However, no additional verification is required, and future research is needed to determine whether these entries are another source of research misrepresentation.

Percentage of Publications With Research Misrepresentation

Most studies found that claimed authorship of a nonexistent article was the most common type of misrepresentation. This was still relatively infrequent, with four recent studies reporting a range of 0.1% (two of 1100) to 8% (26 of 309) among all listed publications [3, 8, 10, 26]. However, any amount of fraudulent information is important to consider when evaluating prospective candidates, and the fabrication of a study raises serious concerns about an applicant’s professionalism and integrity. Several studies across surgical and medical specialties have also found that claimed authorship of a nonexistent article was the most common type of misrepresentation [16, 27, 32, 34, 35, 39, 40]. A recent analysis of applicants to child neurology conducted over six application cycles (2014 to 2020) found that 6% (33 of 514) of articles listed as “published,” “accepted,” or “in-press” were actually nonexistent [34]. Of concern, the authors noted that among these fabricated articles, 33% (11 of 33) were linked to a nonexistent PubMed Identifier. Other types of misrepresentation include nonauthorship of an existing article and incorrect authorship order. On the other hand, four recent studies found these were less common among orthopaedic applicants, occurring in approximately 1% to 3% of publications [3, 8, 9, 26].

Another important finding is the general decrease in publication misrepresentation over time. One recent study [26] directly compared publication misrepresentation at their institution across multiple time periods (1999, 2007, and 2017) and found a substantial reduction in misrepresented articles over time. Similarly, three other single-institution studies evaluating orthopaedic applicants found a reduction in publication misrepresentation in the past 10 years [3, 8, 9]. This is consistently reported across medical and surgical specialties. The likely explanation for this reduction is the requirement for applicants to list the PubMed Identifier of articles in the peer-reviewed publication section of the Electronic Residency Application Service application, starting in 2014 [31]. The incorporation of the PubMed Identifier streamlines the process to verify a publication and may discourage misrepresentation on residency applications. Although studies generally agree that progress has been made over the past several years, misrepresentation still occurs to some extent, and further intervention is warranted.

One possible solution is to require verification of articles that are listed as in-press, accepted, or provisionally accepted. Given that each institution receives thousands of reported publications each year, it would be nearly impossible to verify these publications at the program level. However, a central publication verification process through the Electronic Residency Application Service in which applicants could attach electronic copies of published articles for verification may be possible. Requiring a digital object identifier for research that is not PubMed-indexed would allow for easy verification of articles and serve as a possible deterrent. Lastly, intervention at the medical school level should also be considered. Applicants should be well-educated on research ethics and definitions of publication subtypes. Furthermore, faculty should encourage applicants to reach out to advisors or consult medical librarians for guidance on how to classify publications as needed.

Percentage of Applicants With Research Misrepresentation

Most studies found that the percentage of applicants with one or more misrepresentation was approximately 20% during the past 20 years. However, two studies evaluating recent applicant classes reported noticeably different results. One study [26] found that only 4% (13 of 323) of applicants had at least one misrepresentation, while the other study [12] reported a proportion of 21% (54 of 264). The reason for this difference is unclear because both studies applied similar misrepresentation criteria and evaluated a recent cohort of applicants. However, one of these studies [26] included applicants applying to two 6-year research tracks and noted this may have skewed the cohort to more research-oriented applicants. It is also unclear why the proportion of applicants with at least one misrepresentation has remained approximately 20% while the proportion of overall publication misrepresentation has decreased over time. A similar finding has been reported among applicants to neurosurgery residency [25]. A 2-year analysis found that the proportion of neurosurgical candidates with one or more misrepresentation was 19% (49 of 253) and 18% (56 of 302) in 2015 and 2020, respectively. This difference may be because applicants to competitive surgical subspecialties generally have multiple listed publications [1, 7, 18, 37]. Our findings suggest that most applicants correctly list their publications, but roughly one-fifth of applicants have a misrepresentation among their listed publications. Of note, the number of applicants with more than one misrepresentation was relatively infrequent, with most studies estimating this occurs in 2% of applicants. The high proportion of applicants with a publication misrepresentation raises serious ethical concerns, and governing agencies and residency selection committees must ensure there is integrity related to self-reported information by applicants.

Association Between Research Misrepresentation and Applicant Characteristics

Most studies found no applicant characteristics, such as gender, United States Medical Licensing Examination Step 1 and 2 scores, and Alpha Omega Alpha membership, that were associated with a higher odds of the candidate misrepresenting his or her research credentials. These results align with those of studies evaluating research misrepresentation among surgical subspecialties [7, 23, 39]. However, our findings were mixed regarding the association between research misrepresentation and international medical graduate candidates. Conflicting results have also been reported on this topic across medical specialties [7, 14, 21, 23, 39]. A number of factors may contribute to these results. Several studies have found that international medical graduate applicants list more research entries on their application than United States medical graduate applicants [13, 25, 35]. Similarly, applicants who have taken a year off from their medical training for research were found to have more research entries and more unverified publications [13]. The volume of research may therefore be a confounding variable. Likewise, the sample size of international medical graduates should also be considered. The percentage of international medical graduates ranged from 1% (six of 594) to 14% (62 of 435) among included studies; thus, further investigation with a larger sample size is needed. No other objective demographic data were associated with research misrepresentation. Given these findings, program directors and members of the selection committee should be encouraged to continue a holistic review of applications. We do not recommend using demographic information when assessing an applicant’s risk of research misrepresentation.

Publication Status of Articles Listed as Submitted to Journals

Research is a time-intensive process, and approximately 50% of orthopaedic applicants include articles that are not yet published, such as in-press articles, accepted articles, and articles submitted for publication, on their application. Our analysis found that between 40% and 65% of articles listed as submitted are subsequently published, with one-third published in journals with a lower Impact Factor than that of the originally intended journal. This is comparable to the percentage reported in other specialties [14, 16, 17]. An analysis of applicants to radiology at a single academic institution found that 44% (128 of 293) of submitted manuscripts went on to be published, with a substantial portion in journals with a lower Impact Factor [17]. Because most submitted articles will go on to publication, these articles should still be considered when evaluating a candidate’s scholarly productivity. However, given our findings, they should be interpreted with some caution and weighed less than fully published works. Furthermore, effort should be made to improve the process for reporting submitted articles in order to decrease publication misrepresentation. Application services could allow applicants to provide updates on submitted research throughout the application cycle. Institutions could also encourage applicants to provide significant research updates throughout the application cycle or verify submitted articles during interviews.

Conclusion

Our systematic review found that the percentage of publication misrepresentations among orthopaedic residency and fellowship applicants has generally been low over the past 20 years. However, approximately one-fifth of applicants had at least one research misrepresentation, with 2% having multiple misrepresentations on reported publications. Additionally, no applicant characteristics were associated with higher odds of research misrepresentation. These findings serve to encourage faculty members involved in the application screening and decision process to remain vigilant when evaluating a candidate’s listed publications as well as limit biases related to applicant demographics that are perceived to be associated with high proportions of misrepresentation. Furthermore, in light of a continually evolving application process, governing agencies and program leadership should evaluate methods of verifying unpublished work and opportunities for applicants to provide updates throughout the application cycle. Future studies should continue to evaluate the contemporary proportion of misrepresentation in the setting of increased publication volume among applicants, and shift toward a more-holistic application review in the setting of a pass/fail United States Medical Licensing Examination Step 1 result. Similarly, survey information regarding why applicants might misrepresent publication information is crucial to help reduce its future incidence.