Abstract
Purpose
Each year, the American Society for Surgery of the Hand (ASSH) selects several abstracts for podium presentations during a “Best Papers” session. We examined these papers to better understand their characteristics and impact on the field of hand surgery.
Methods
“Best Papers” from the 2010 to 2020 ASSH Annual Meetings were reviewed. Online databases were searched to find matching publications. Descriptive data were collected from the publications. The Hirsch index value for each corresponding author and the number of citations for each publication were recorded. Descriptive statistics were used to analyze the data.
Results
Fifty-nine “Best Papers” were awarded during the study period. Forty-nine (83%) were clinical and 10 were basic science studies. A total of 39 observational studies, 11 human trials, 8 experimental studies, and 1 case series were present. Fifty-four (91.5%) were published at the time of our review. Twenty-six of those (48%) were multicenter studies, and the remaining 28 were from a single institution. The average time from presentation to publication was 16 months. The top three journals of publication were the Journal of Hand Surgery (33%), the Journal of Bone and Joint Surgery (9%), and the Journal of Hand Surgery, European (7%). The median level of evidence for all “Best Papers” was 3, with a trend toward a higher level of evidence during the study period. The average h-index value of all corresponding authors was 27.3. The average number of citations per publication was 37.
Conclusions
The ASSH “Best Papers” were primarily clinical studies with an increasingly strong level of evidence and were likely led by an author with a history of research productivity. Selection as a “Best Paper” at ASSH Annual Meetings is a strong predictor of future publication and impact.
Clinical relevance
This study evaluates the “value” of the best paper designation at the ASSH annual meeting.
Key words: ASSH annual meeting, Best papers, Hand society, Impact factor
The American Society for Surgery of the Hand (ASSH) Annual Meeting includes a mix of symposia, instructional course lectures, posters, and podium presentations of research. In advance of each meeting and after review of a submitted abstract, the ASSH Scientific Committee, under the leadership of the Annual Meeting Chair(s), designates a number of these abstracts as the “Best Papers.” These podium presentations occur at the beginning of the meeting with an expected large audience. Although previous studies have examined the publication patterns, levels of evidence, and study characteristics of all presented abstracts, limited information on these awarded papers exists.1, 2, 3, 4, 5
The rationale behind this study was to examine these presentations closely to define their study characteristics and subsequent publication patterns. We sought to understand their ultimate impact by assessing whether the presentation was published, where it was published, and the details of the impact of the publication. We hypothesized that more than 90% of these selected presentations would be published within 2 years. As a secondary outcome, we examined the impact these studies have had on the practice of hand surgery based on the journal of publication and subsequent citations.
Methods
The online abstract books from the 2010 to 2020 ASSH Annual Meetings were queried to identify the “Best Papers.” We collected data on the month and year of the meeting, abstract title, author(s), level of evidence, type of study, and research category (basic science or clinical). The types of studies were defined as follows: experimental (animal or histochemical studies), case series, human trial (clinical or randomized), observational (case-control, cohort, and cross-sectional studies), or systematic review. We further categorized bench research and animal studies as basic science, and clinical papers were defined as both studies that directly involved human subjects and other studies that involved the practice of hand surgery, such as cost analyses and systematic reviews.
A search of PubMed and Embase was performed in July of 2023 to identify publications associated with the presented abstracts. Two of the authors independently searched for publications first by title, then by author(s), and finally by key words and phrases from the abstract. An abstract was deemed not yet published if neither author was able to find the matching publication.
Data collected from the associated publications included manuscript title, author(s), corresponding author, month and year of publication, performing institution(s), level of evidence, and journal of publication. The month and year of publication were first obtained from the PubMed or Embase webpage for each publication and were confirmed on the PDF version of each article. In many of the publications, the level of evidence was specifically stated in the manuscript. For the publications that did not specifically state the level of evidence, the level of evidence was assigned by the authors using the Oxford Center for Evidence-Based Medicine 2011 Levels of Evidence Tool. A Wilcoxon signed-rank test was used to evaluate any statistical difference in the compared level of evidence between the presented abstracts and subsequent publication.
In an effort to understand leadership of the research teams for these chosen presentations, we used the h-index as a tool to best understand the corresponding author. The h-index, proposed by J. E. Hirsch in 2005, was designed to estimate the significance and impact of a scientist’s cumulative research contributions.6 It has been used as a tool to evaluate the performance of researchers to aid in decisions for resource allocation and academic promotion.7 Based on the work presented by Walker et al8 in 2016 that highlighted the interrater reliability and simple user interface, we used Scopus for the calculation of the h-index.
Multiple biometric indices were used to understand the quality of journals in which the Best Papers were published, including the Journal Impact Factor, Eigenfactor Score, and Article Influence Score. These metrics were obtained from the 2023 Journal Citation Reports by Clarivate Analytics. The current number of publication citations for each article was obtained from Google Scholar.
Simple descriptive statistics were then analyzed for each year’s group of “Best Papers.” Additionally, an independent two-sample proportions z-test was used to compare the publication rate found in this study with that of other studies that have reported on publication patterns of papers presented at national hand surgery meetings.1, 2, 3, 4
Results
A total of 59 abstracts were identified as the “Best Papers” from the ASSH Annual Meetings between 2010 and 2020. The number of papers chosen each year for this designation ranged from 4 to 8, with 5 being the mode. Forty-nine of the 59 papers (83%) were clinical studies, and the remaining 10 were basic science. A total of 39 observational studies, 11 human trials, 8 experimental studies, and 1 case series were noted (Table 1). Fifty-four abstracts had been published as full-length manuscripts at the time of this investigation, with a publication rate of 91.5%. Twenty-six of the 54 publications (48%) were multicenter studies, and the remaining 28 (52%) were performed at a single institution. The average time from presentation to publication was 16 months (ranging from 8 months before presentation to 65 months after). Forty-three of the 54 publications (80%) were published within 2 years of presentation. Of note, at the time of this review, more than two and a half years after the 2020 Annual Meeting, four of the five presentations have been published. A total of 23 different journals published these studies (Fig. 1). The most frequent journals for publication were the Journal of Hand Surgery (JHS) (33.3%), the Journal of Bone and Joint Surgery (JBJS) (9.3%), and the Journal of Hand Surgery, European (JHS(E)) (7.4%), which accounted for 50% of the total.
Table 1.
Study Designs of the Published ASSH Best Papers
| Type of Study | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 | 2018 | 2019 | 2020 | Total |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Case series | 1 | 1 | ||||||||||
| Experimental | 1 | 1 | 1 | 1 | 2 | 1 | 1 | 8 | ||||
| Human trial | 1 | 1 | 1 | 2 | 1 | 1 | 3 | 1 | 11 | |||
| Observational | 4 | 5 | 4 | 2 | 5 | 7 | 1 | 3 | 4 | 1 | 3 | 39 |
| Grand total | 4 | 5 | 5 | 4 | 8 | 8 | 4 | 6 | 5 | 5 | 5 | 59 |
Figure 1.
The number of “Best Paper” publications for each journal.
The publication percentage by group of papers per year varied and ranged from 100% to 75% with a decreasing trend during the period (Fig. 2). The median level of evidence for all clinical papers over the time period was 3 (range 1–4), and the level of evidence increased over the time of this investigation (Fig. 3). The level of evidence of the presentations and subsequent publications were compared, and the results showed no significant difference, P = .506. The average h-index of the corresponding author was 27.3 (range 1–103). The mean values of Journal Impact Factor, Eigenfactor Score, and Article Influence Score of the publishing journals were 4.1, 0.02158, and 1.4, respectively. The average number of citations for all publications was 37 (range 0–212). As might be expected, the number of citations was fewer for the more recently published articles (Fig. 4).
Figure 2.
The publication percentage of each year’s group of “Best Papers.”
Figure 3.
The median level of evidence for each year’s group of “Best Papers” and a trend of an increasing level of evidence over time.
Figure 4.
The average number of citations per paper for each year’s group of “Best Papers.”
Discussion
The “Best Papers” from the 2010 to 2020 ASSH Annual Meetings were most often clinical studies with increasing quality of evidence over the time period of this investigation. These studies were likely led by an author with high research productivity and were just as likely to be performed at a single center or multiple centers. Typically, but not entirely, these abstracts were published in high-quality journals at an average of just over a year after presentation and have since been cited frequently since publication.
The average h-index for the corresponding authors of the “Best Papers” in our study was 27.3. This roughly translates to the authors, on average, having 27.3 publications with each having at least 27.3 citations. Hirsch surmised as an example that “after 20 years of research, an h-index of 20 is good, 40 is outstanding, and 60 is exceptional.”6 Thus, an h-index of 27.3 would be considered good for potential impact, leaving room for improvement with future studies.
To further examine the journals most often chosen for publication of the “Best Papers,” we evaluated their Journal Impact Factor, Eigenfactor Score, and Article Influence Score. These metrics were obtained from the 2023 Journal Citation Reports by Clarivate Analytics. The impact factor of a journal is one measure of a journal’s role in its field, as assessed by the citation frequency with which a typical article in a journal has in a particular time period.9 Specifically, the impact factor of a journal corresponds to the average number of citations received per article published in that journal during the preceding two years. The current impact factors for the top journals of publication found in our study (JHS, JBJS, and JHS(E)) are 1.9, 5.3, and 1.8, respectively. For JHS, this roughly translates to the average article published in that journal in 2021 and 2022 having an average of 1.9 citations in 2023. When ranking these journals by impact factor among the 86 other journals listed in the category of orthopedic surgery, JHS ranked number 50, JBJS ranked number 5, and JHS(E) ranked number 53.
Recently, it has been suggested that the Journal Impact Factor has certain shortcomings when assessing the true impact of medical journals.10, 11, 12 A few specific concerns regarding the Journal Impact Factor are that it does not consider differences in the prestige of citing journals, and it does not take into account differences in citation patterns across disciplines. The risk of potential inflation of the impact factor by means of self-citation is also observed.13 These limitations led to the development of the Eigenfactor metrics by Carl Bergstrom and Jevin West in 2007.14
The Eigenfactor metrics comprise two main scores, the Eigenfactor Score and the Article Influence Score. Both are a reflection of the network of citations around the journal using the preceding 5 years of cited content. In addition to the number of citations a journal has, they also consider the source of those citations. Highly cited sources will influence the network more than less cited sources. The Eigenfactor Score calculation eliminates journal self-citations. The Article Influence Score normalizes the Eigenfactor Score according to the cumulative size of the cited journal across the previous 5 years. The mean Article Influence Score for each article is 1.00. A score greater than 1.00 indicates that each article in the journal has above-average influence. In general, the Eigenfactor Score measures the journal’s total importance and the Article Influence Score reflects the journal’s prestige. The mean Eigenfactor Score of the journals that published these Best Papers was 0.02158, and the corresponding mean Article Influence Score was 1.419, which indicates that the Best Papers are typically published in journals with above-average impact in the field of hand surgery.
Stepan et al recently reported on the increasing level of evidence and rate of publication of general abstracts presented at ASSH Annual Meetings over a 23-year period (1992–2014).4 Their study included both poster and podium presentations, and they reported an overall publication rate of 53.6%. After stratifying their total study period into three equal time periods, they noted a significantly greater publication rate in the most recent 8-year time period compared with the earlier time periods (61% in 2007–2014 vs 52% in 1999–2006 and 47% in 1992–1998). Kuczmarski et al5 evaluated the publication patterns of the podium presentations at the ASSH and the American Association for Hand Surgery national meetings from 2007 to 2012, reporting publication rates of 72% and 58%, respectively. Abzug et al2 evaluated both podium and poster presentations from 2000 to 2005 ASSH meetings and reported a 46% total publication rate (54% for podium and 41% for poster), and an average time of 2 years from presentation to publication.
Our hypothesis that more than 90% of these abstracts would be published within 2 years of presentation proved untrue. Forty-three of the 59 Best Papers (73%) were published within 2 years of presentation. Interestingly, a notable difference was observed in the overall publication rate when contrasting these awarded papers (91.5%) to the groups of papers evaluated in other studies on hand surgeon meetings (Table 2). A decreasing publication rate over the examined time period (Fig. 2) was observed, but abstracts presented in the past few years may still lead to publication in the future.
Table 2.
Comparison of the Publication Rate Found in This Study (54/59 or 91%) to Other Studies That Have Reported on Publication Patterns of Papers Presented at National Hand Surgery Meetings
|
Publication Percent in Referenced Studies |
95% CI for Difference of Proportions |
||
|---|---|---|---|
| P | Lower Bound | Upper Bound | |
| 206/397 (52%) [1] | < .001 | 29.2% | 43.1% |
| 518/1127 (46%) [2] | < .001 | 35.2% | 49.1% |
| 393/798 (49%) [3] | < .001 | 26.5% | 40.4% |
| 942/1757 (53.6%) [4] | < .001 | 27.6% | 41.5% |
Theman et al3 evaluated differences between abstracts delivered as podium presentations at national hand meetings and their subsequent publications. They reported a 49% publication rate with an average of 18 months from presentation to publication. Interestingly, they also found that 14% of publications had different results/conclusions than their respective abstracts and advised readers to use caution when applying results from presented abstracts to practice.
The average time from presentation to publication in our study was 16 months. This compares favorably with the 18-month and 25-month average time to publication demonstrated by Abzug et al2 and Theman et al,3 respectively. Stepan et al4 identified a median time to publication of 24 months but reported that 5% of the abstracts in their study were published after 6 or more years. The longest time between presentation and publication in our study was just over 5 years (65 months). We found the top journals of publication to be JHS and JBJS, consistent with previous studies.1, 2, 3, 4, 5
Our study demonstrated that clinical studies were more likely to be awarded as a “Best Paper” than were basic science studies (83% vs 17%). Given that the number of submissions was certainly greater for clinical papers, this is not surprising. Regarding publication patterns, 3 of the 49 clinical studies and 2 of the 10 basic science studies remain unpublished.
The average number of citations per publication in our study group was 37. As would be expected, the publications from more recent years have fewer citations compared with the earlier years. Over time, we would expect these papers to be increasingly cited in the literature. With an increasing reliance on evidence-based medicine, we surmise that publications that have more citations are more relevant to the practice of hand surgery, given the greater focus of research.
We recognize the limitations of our study. We finalized the study in July of 2023 and examined the “Best Papers” from 2010 to 2020. This allowed a time period of over 2.5 years from presentation to publication for the 2020 group, and this is likely reflected in a lower publication rate for 2019 and 2020. Other studies have suggested allowing for at least 3 to 5 years for publication before analysis.1, 2, 3, 4, 5 An additional limitation is our use of the number of citations as a surrogate measure for the impact a publication has had on the field of hand surgery. Although we believe that publications in quality journals with a high number of citations are a reasonable subjective measure of impact, we understand that no truly objective way exists to measure the impact of research publications on the practice of medicine.
Acknowledgments
The authors would like to thank Rosey Zackula, M.A. for her help with statistical analysis.
Footnotes
Declaration of interests: No benefits in any form have been received or will be received related directly to this article.
References
- 1.Gavazza J.B., Foulkes G.D., Meals R.A. Publication pattern of papers presented at the American Society for Surgery of the Hand annual meeting. J Hand Surg Am. 1996;21(5):742–745. doi: 10.1016/S0363-5023(96)80185-7. [DOI] [PubMed] [Google Scholar]
- 2.Abzug J.M., Osterman M., Rivlin M., Paryavi E., Osterman A.L. Current rates of publication for podium and poster presentations at the American Society for Surgery of the Hand annual meetings. Arch Bone Jt Surg. 2014;2(3):199–202. [PMC free article] [PubMed] [Google Scholar]
- 3.Theman T.A., Labow B.I., Taghinia A. Discrepancies between meeting abstracts and subsequent full text publications in hand surgery. J Hand Surg Am. 2014;39(8):1585–1590. doi: 10.1016/j.jhsa.2014.04.041. [DOI] [PubMed] [Google Scholar]
- 4.Stepan J.G., Chen F.R., Prabhakar P., et al. Level of evidence and publication rates of abstracts presented at the American Society for Surgery of the Hand Annual Meetings: comparison over 23 years. J Hand Surg Am. 2020:981–988. doi: 10.1016/j.jhsa.2020.04.017. [DOI] [PubMed] [Google Scholar]
- 5.Kuczmarski A.S., Lemme N.J., Biron D., Sobel A.D., Weiss A.C. Characteristics and publication rates for podium presentations at National Hand Surgery Meetings from 2007 to 2012. J Hand Microsurg. 2020;12(2):107–110. doi: 10.1055/s-0039-1695663. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Hirsch J.E. An index to quantify an individual's scientific research output. Proc Natl Acad Sci USA. 2005;102(46):16569–16572. doi: 10.1073/pnas.0507655102. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Patel V.M., Ashrafian H., Almoudaris A., et al. Measuring academic performance for healthcare researchers with the H index: which search tool should be used? Med Princ Pract. 2013;22(2):178–183. doi: 10.1159/000341756. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Walker B., Alavifard S., Roberts S., Lanes A., Ramsay T., Boet S. Inter-rater reliability of h-index scores calculated by Web of Science and Scopus for clinical epidemiology scientists. Health Info Libr J. 2016;33(2):140–149. doi: 10.1111/hir.12140. [DOI] [PubMed] [Google Scholar]
- 9.Sharma M., Sarin A., Gupta P., Sachdeva S., Desai A.V. Journal impact factor: its use, significance and limitations. World J Nucl Med. 2014;13(2):146. doi: 10.4103/1450-1147.139151. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Huggett S. Journal bibliometrics indicators and citation ethics: a discussion of current issues. Atherosclerosis. 2013;230(2):275–277. doi: 10.1016/j.atherosclerosis.2013.07.051. [DOI] [PubMed] [Google Scholar]
- 11.Kodumuri P., Ollivere B., Holley J., Moran C.G. The impact factor of a journal is a poor measure of the clinical relevance of its papers. Bone Joint J. 2014;96-B(3):414–419. doi: 10.1302/0301-620X.96B3.32279. [DOI] [PubMed] [Google Scholar]
- 12.Yuen J. Comparison of Impact Factor, Eigenfactor Metrics, and SCImago Journal Rank Indicator and h-index for Neurosurgical and Spinal Surgical Journals. World Neurosurg. 2018;119:e328–e337. doi: 10.1016/j.wneu.2018.07.144. [DOI] [PubMed] [Google Scholar]
- 13.Miyamoto S. Self-citation rate and impact factor in the field of plastic and reconstructive surgery. J Plast Surg Hand Surg. 2018;52(1):40–46. doi: 10.1080/2000656X.2017.1319847. [DOI] [PubMed] [Google Scholar]
- 14.Ali M.J. Understanding the eigenfactor. Semin Ophthalmol. 2021;36(3):65–66. doi: 10.1080/08820538.2021.1913313. [DOI] [PubMed] [Google Scholar]