Factors Associated With the Highest and Lowest Cited Research Articles in General Surgery Journals

Abstract

Introduction:

Citation count is a common bibliometric tool used to determine long-term impact and performance of journal articles. Many of the other potential factors associated with highly and lowly cited articles in the general surgery literature, however, remain unknown. The purpose of this study was to attempt to identify characteristics of articles that may predict or correlate with article citation counts and, consequently, article impact.

Methods:

We identified articles from Annals of Surgery, British Journal of Surgery, and Journal of the American College of Surgeons between 1998 and 2008 that had 0–5 total citations. We then matched these articles to an identical number of the highest cited articles from these same journals for comparison. Student’s t tests, Wilcoxon rank-sum tests, chi-squared tests, and Fisher’s exact tests were used to determine significance of difference between data sets at a predetermined level of significance set to p <0.05.

Results:

Significant differences of article characteristics between the two cohorts included higher prevalence of clinical studies (p = 0.3919), multi-institutional (p = 0.0007) and multi-national (p = 0.0023) studies, surgical oncology (p <0.0001) or hepatobiliary focus (p <0.0001), and published in Annals of Surgery (p <0.0001)for the highly cited cohort. Highly cited articles were also more likely to have larger sample sizes (p = 0.0009), more authors (p <0.0001), presence of statistically significant results (p <0.0001), more references (p <0.0001), more tables (p <0.0001), more figures (p = 0.0001), and higher word counts for manuscript (p <0.0001), abstract (p <0.0001), and title (p <0.0001).

Conclusion:

There are a relatively small number of articles with 0–5 citations after 10 years for these major general surgery journals. This indicates that journals are consistently able to select articles that will be impactful in aiding future research. Certain factors, however, are associated with being highly cited as opposed to lowly cited, and an understanding of these factors can aid researchers and journals in designing and reporting future studies.

Introduction:

Research in medicine, from basic science to translational application, is an essential component of the academic mission of physicians, institutions, and journals alike^1,2. This same research, however, is also too often plagued with inefficiencies, waste, and high costs that consume substantial time and funding^3,4. While these expenditures are often necessary to produce impactful research, the problem is amplified when high quality articles are obscured by an ever-increasing pool of poorly designed research published in the name of profits or academic output^2,5.

For journals, the differentiation of high impact from low impact has historically been made with bibliometric profiling through “impact factors,” often calculated using gross journal citation counts per number of manuscripts published over the past two years. In spite of certain flaws, this method is generally well accepted as a way to measure journal impact and has been used in prior studies.^2,5,6 For individual articles, however, the flaws inherent to journal impact factors make correlations between journal and article impact difficult. Namely, journals may rely heavily on a relatively small number of articles to drive impact factor ratings higher, with a prior study identifying as many 90% of total citations coming from only 30% of journal articles.⁷

While citation counts are an easily quantifiable metric to retrospectively identify highly impactful articles, the identification of potentially predictive factors have proved more elusive. Efforts within the urology, cardiology, and radiology literature have been attempted, but no similar effort has yet to be applied to the general surgery literature^2,5,6. In addition, the prior study of cardiology literature attempted to analyze a wide-array of journals and larger sample of articles, with 222 journals and over 150,000 articles identified, but this methodology limited the granularity of their results.² As such, examination of the extreme end points of citation data may allow for a more detailed analysis of factors to then later be included in larger studies. If such factors could be identified, their application could assist in the reduction of wasted effort and the promotion of highly cited articles for the surgical community and its audience.

Methods:

We searched SCImago Journal and Country Rank, a public database utilizing Scopus to organize journals by scientific domain.⁸ A list of journals under the category “surgery” was generated and journals that only addressed subspecialty topics were excluded. Subsequently, three journals, the Journal of the American College of Surgeons (JACS), British Journal of Surgery (BJS), and Annals of Surgery (AS) were identified as having the highest citations per document during the study period of 1998 to 2008. These journals were then searched for original research articles published between 1998 and 2008 using PubMed’s advanced search tool.⁹ This time frame was chosen based on previously published research indicating a ten-year window as the time required for accumulation of an articles citation to maturity.¹⁰ Articles were eligible for inclusion if they contained a full abstract and were excluded if they were classified as a review article, editorial, case report, guideline statement, or errata.^6,11 All articles published in the indicated journals were included, regardless of open-access status. This did, however, involve exclusion of articles published ahead-of-print, as articles were only included if they were in print and published as indexed on Pubmed. Based on the search, Pubmed’s search tool generates a list of article identification numbers. These numbers were analyzed by iCite from the National Institute of Health to determine the total citation count for each article.¹²

Articles with 0–5 total citations from the search were included in the “low citation” cohort for analysis. The number of articles included within this cohort was used to determine the number of articles in the “high citation” cohort. Subsequently, the articles with the highest citation count within that range were selected for evaluation. For example, if the search were to yield 10 articles with 0–5 citations, then the “high citation” cohort would be selected by identifying the top 10 highest cited articles from the search. Cohorts were designed in this manner to allow for matched data set analysis.

Two investigators independently analyzed each cohort for pre-identified characteristics. Discrepancies between evaluator analysis were resolved by discussion and mutual decision between the two investigators. The characteristics evaluated were adapted from prior studies in the radiology and urology literature.^5,6 Author information, departmental affiliation, and institutional information was based on data presented in the article as published.

Each article was evaluated based on authorship factors such as USA or non-USA first author affiliation, presence of a general surgeon author, presence of a non-general surgeon author, quantity of non-surgeon authors, single or multi-institutional representation, and national or international collaboration. Author demographic analysis also included the presence or absence of funding and conflicts of interest reports and the journal of publication. In addition, each article was labeled as being as most associated with a particular surgical fellowship and surgical research topic based on the discretion of the reviewers.

Article structure analysis was based on the number of words in each article title, the presence or absence of any punctuation in the title, total abstract word count, total article word count, numbers of independent tables and figures, number of references, and total page count of the final publication. Between the abstract and manuscript body, articles were evaluated for the presence or absence of statistically significant results for both independently. Study methodology was also included in the analysis and stratified by clinical or non-clinical and prospective vs. retrospective.

Data analysis was performed using bivariate comparison with student’s t test and Wilcoxon rank-sum test for normal and non-normal variable distributions respectively. Chi- squared test or Fischer’s exact test were used for categorical variable analysis (SAS, version 9.4). A p value of less than 0.05 determined whether or not data was considered significant.

Results:

For the 10-year search period of the three identified journals (JACS, BJS, AS), we classified 58 articles as having 0–5 citations. These 58 articles were then matched to the 58 highest cited articles from the same journals within the same time period. The mean for citation count per article between both cohorts was 318.1±874.1. Annals of Surgery had a significantly higher number of articles present in the highly cited cohort relative to lowly cited cohort (84% vs 17%, p <0.0001) Between the two groups, we identified a number of significant differences (Table 1). The highly cited articles were more likely to be clinical studies (81% vs 53%, p = 0.0015) with multinational (20% vs 2%, p = 0.0023) and multi-institutional (47% vs 17%, p = 0.0007) efforts. For the clinical studies between both groups, the highly cited articles were more likely to have a larger mean sample size as well (505 vs 153, = 0.0009). Higher cited articles were more likely to be related to the field of Hepatobiliary research (33% vs 1%, p <0.0001) or Surgical Oncology (45% vs 9%, p <0.0001) and were more likely to have statistically significant results in the abstract (83% vs 45%, p <0.0001) and manuscript (98% vs 69% p <0.0001).

Table 1.

Comparison of articles with the lowest and highest citations.

	Articles with the Lowest Number of Citations n=58	Articles with the Highest Number of Citations n=58	p-value
Journal			<.0001
J. Am. Coll. Surg.	25 (43.10)	3 (5.17)
Br. J. Surg.	23 (39.66)	6 (10.3)
Ann. Surg.	10 (17.24)	49 (84.48)
Mean Number of Authors (SD)	4.81 (2.24)	8.00 (3.88)	<.0001
Country			0.2509
USA	25 (43.10)	19 (32.76)
Non-USA	33 (56.90)	39 (67.24)
Presence of Non-Gen Surg			0.0087
Gen Surg Only	32 (55.17)	18 (31.03)
Non-Gen Surg Present	26 (44.83)	40 (68.97)
Any Presence of Gen Surg			0.4957
Gen Surg Present	58 (100.0)	56 (96.55)
No Gen Surg Present	0 (0.0)	2 (3.45)
Mean Number of Other Disciplines (SD)	0.86 (1.21)	1.10 (1.21)	0.2841
Number of Institutions			0.0007
One Institution	48 (82.76)	31 (53.45)
Multi-Institution	10 (17.24)	27 (46.55)
Number of Countries			0.0023
One Country	57 (98.28)	47 (81.03)
Multinational	1 (1.72)	11 (18.97)
Funding			0.0594
Funding Reported	19 (32.76)	29 (50.0)
No Funding Reported	39 (67.24)	29 (50.0)
Conflicts of Interest			0.1185
None declared	54 (93.10)	58 (100.0)
Declared	4 (6.90)	0 (0.0)
Number of Words in Title (SD)	5.72 (6.20)	15.97 (4.56)	<.0001
Punctuation in Title			0.0002
No Title Punctuation	40 (68.97)	20 (34.48)
Title Punctuation	18 (31.03)	38 (65.52)
Abstract Word Count (SD)	171.78 (118.32)	303.21 (77.39)	<.0001
Manuscript Word Count (SD)	1932.05 (1070.54)	3350.07 (948.38)	<.0001
Abstract Results			<.0001
No Stat Sig Result	32 (55.17)	10 (17.24)
Stat Sig Result	26 (44.83)	48 (82.76)
Paper Results			<.0001
No Stat Sig Result	18 (31.03)	1 (1.72)
Stat Sig Result	40 (68.97)	57 (98.28)
Number of References (SD)	18.41 (12.64)	33.93 (14.57)	<.0001
Number of Tables (SD)	1.98 (1.57)	4.19 (2.06)	<.0001
Number of Figures (SD)	2.03 (2.18)	3.62 (2.13)	0.0001
Number of Manuscript Pages (SD)	5.29 (2.55)	8.88 (2.12)	<.0001
Fellowship Classification			<.0001
MIS	9 (15.52)	1 (18.97)
Vascular	11 (18.97)	0 (0.00)
Surg Onc	5 (8.62)	26 (44.83)
Trauma	4 (6.9)	2 (3.45)
Colorectal	7 (12.07)	12 (20.69)
Transplant	9 (15.52)	2 (3.45)
Cardiothoracic	2 (3.45)	1 (1.72)
Pediatric	2 (3.45)	0 (0.0)
Breast	1 (1.72)	1 (1.72)
Endocrine	1 (1.72)	1 (1.72)
Other	7 (12.07)	2 (3.45)
Paper Classification			<.0001
Basic Science	7 (12.07)	2 (3.45)
Oncologic Pathology	5 (8.62)	11 (18.97)
Perioperative medicine	5 (8.62)	1 (1.72)
Benign Pathology	11 (18.97)	2 (3.45)
QI	2 (3.45)	3 (5.17)
Critical Care	2 (3.45)	1 (1.72)
Laparoscopy	7 (12.07)	1 (1.72)
Education	2 (3.45)	4 (6.90)
Bariatric	2 (3.45)	4 (6.90)
HPB	1 (1.72)	19 (32.76)
Other	14 (24.14)	10 (17.24)
Clinical vs Non-Clinical			0.0015
Clinical	31 (53.45)	47 (81.03)
Non-clinical	27 (46.55)	11 (18.97)
Study Design (Clinical)			0.3919
Retrospective	7 (22.58)	6 (12.77)
Prospective	24 (77.42)	40 (85.11)
Both	0 (0.00)	1 (2.13)
Mean Sample Size (SD) (Clinical)	153.13 (213.59)	505.55 (537.36)	0.0009

Values are n (%) unless otherwise stated

In addition, the highly cited cohort was more likely to have more authors (8 vs 4.8, p <0.0001) and have a non-general surgeon as an author present (69% vs 45%, p = 0.0087). Articles in the highly cited cohort were also significantly more likely to have more words in the title (15.97 vs 5.72, p <0.0001), higher prevalence of punctuation in the title (38% vs 18%, p =0.0002), have a higher abstract word count (303.21 vs 171.78, p <0.0001) and manuscript word count (3350.07 vs 1932.05, p <0.0001). Comparatively, highly cited articles also had more references (33.93 vs 18.41, p <0.0001), more tables (4.19 vs 1.98, p <0.0001), more figures (3.62 vs 2.03, p = 0.0001), and more manuscript pages (8.88 vs 5.29, p <0.0001).

There was no significant difference (p >0.05) between the cohorts in country affiliation of first author, presence of a general surgeon author, quantity of disciplinary representation among authors, reported funding or conflicts of interest, or study design for clinical studies (retrospective vs prospective vs both).

Discussion:

With the use of PubMed and iCite we identified and compared the highest and lowest cited articles within a 10-year period for the top general surgery journals. Only 58 articles were identified as having 0–5 citations between all three journals, and these were matched for comparison to the 58 highest cited between the same three journals. The fact that only 58 articles have 0–5 citations between these 3 journals is testament to the journals’ rigorous commitment to methodologic integrity, clinical relevance, and potential impact of published articles. Compared to prior, comparable studies in the urologic and radiologic literature, there were 50 and 47 articles respectively with 0 to 1 citation identified within the same time period,^5,6 suggesting a relatively high performance of even the lowest cited articles among these general surgery journals.

Citation counts, while an imperfect measure of article performance, remains a traditional and validated method for comparing article impact and exposure.^10,13 With the rise of newer bibliometric tools and measurement, such as Altmetrics, a number of studies have evaluated article impact and correlations between traditional and newer models.¹⁴ For this studies two cohorts, however, the timeframe of interest (1998–2008) predates these newer bibliometric tools and were, therefore, excluded from the analysis.

While our results highlight many significant variables relative to article performance, many of these differences may be obvious to readers without the need for formal analysis. Larger sample sizes, statistically significant results, and clinical studies would be expected in highly cited articles and concur with prior studies in the urologic and radiology literature.^5,6 These results also highlight, however, characteristic and significant differences between the cohorts that may not be as outwardly evident, including abstract and manuscript word count, title word counts, and title punctuation. Some of these, notably longer abstract and manuscript word counts, were also found to be predictors of higher cited articles in the urology and radiologic literature.^5,6 Understandably, these predictors may represent confounding from study complexity, but they may also underscore a preferable or familiar linguistic style linking reader with expected level of article impact and subsequent attention.

Interestingly, this study discovered a heavy prevalence for articles with a focus in hepatobiliary research or surgical oncology research associated with higher cited articles. In addition, articles with a focus on vascular surgery or in benign pathology were more likely to be in the lowly cited cohort. While this may represent current trends in funding and journal focus, it is possible that this represents an underlying bias towards or against certain topics independent of methodologic rigor or impact to the field of general surgery as a whole. In addition, we found that articles in the highly cited cohort almost exclusively had a result reported as “statistically significant.” In light of recent discussion on the topic of significant p-value thresholds and their role in bias among research reporting and interpretation, this finding further highlights the prevalence of statistical significance reporting among highly cited manuscripts.¹⁵

For researchers, journals, and institutions, research activity and publication is a long- standing central aspect of the mission of academic medicine, not only for surgery, but for all specialties.^13,16 As researchers conduct research and report their results, an understanding of characteristics such as these can help maximize publication potential and future article performance. In addition, as journal efficiency and quality of published research continue to strive for a compatible arrangement, an understanding of baseline indicators of citation rates can impact the process of research from design to publication.

The authors are aware of the limitations of this study. The study only included 3 of the journals that cover the general surgery literature, covering a wide range of subspecialty topics as well. It is unknown if the results of this study are applicable to articles from journals not included here. The decision here to only include 58 articles within each cohort was balanced with the analysis of articles only on the extreme two ends of citation counts. This decision, along with the decision to include only the highest impact journals in General Surgery, limits generalization to other journals and exposes these results to selection bias. In order to perform such an in-depth analysis of each article within a reasonable time frame, however, our aim was to look only at the extremes present in the literature. While evaluation of all general surgery journals with this level of detail would have been impractical given our resources, our hope for this study is to establish a baseline for factors associated with high and low cited articles. From this data, a subset of these factors can be used in a follow-up study that encompasses a wider array of the surgery literature.

Correlation studies including articles within the excluded range of citation counts could further increase the understanding of the degree to which each of these variables impacts citation counts. The methodology of included studies was not assessed beyond the sample size and study design, and the authors cannot guarantee that included articles represent high quality methodologic practice. Also, the authors were limited to the data provided by PubMed and iCite, and as such, any error in the collection or reporting of data from these sites cannot be confirmed. Notably, it has previously been found that difference exist between article citation count metrics between database (iCite, Scopus Web of Science, Google Scholar).¹⁷ While iCite and PubMed should demonstrate internal consistency through data sharing between the National Library of Medicine and the National institute of Health, iCite was initially chosen due to investigator preference as it is widely publicly available. We also did not exclude citations that represented self-citation by authors and the presence of such may have obscured or biased our results from the highly cited cohort.

Overall, there is a limited number of articles with 0–5 citations in the top general surgery journals that have been in publication for 10 years. This encouraging finding highlights the strength of articles chosen for publication by these journals and the journals editors’ ability to predict future article performance and impact. Between two highly and lowly cited article cohorts, there were a number of recognizable differences that can be used to guide further research and publishing practices.