Abstract
Background:
Increasing use of “hype” language (eg, language overstating research impact) has been documented in the scientific community. Evaluating language in abstracts is important because readers may use abstracts to extrapolate findings to entire publications. Our purpose was to assess the frequency of hype language within orthopaedic surgery.
Methods:
One hundred thirty-nine hype adjectives were previously identified using a linguistics approach. All publicly available abstracts from 18 orthopaedic surgery journals between 1985 and 2020 were obtained, and hype adjectives were tabulated. Change in frequency of these adjectives was calculated.
Results:
A total of 112,916 abstracts were identified. 67.0% (948/1414) of abstracts in 1985 contained hype adjectives, compared with 92.5% (5287/5714) in 2020. The average number of hype adjectives per abstract increased by 136% (1.1 to 2.6). Of the 139 adjectives, 87 (62.5%) increased in frequency and 40 (28.7%) decreased in frequency while 12 (9%) were not used. The hype adjectives with the largest absolute increases in frequency were quality (+324wpm), significant (+320wpm), systematic (+246wpm), top (+239wpm), and international (+201wpm). The five hype adjectives with the largest relative increases in frequency were novel (+10500%), international (+2850%), urgent (+2600%), robust (+2300%), and emerging (+1400%).
Conclusion:
Promotional language is increasing in orthopaedic surgery abstracts. Authors, editors, and reviewers should seek to minimize the usage of nonobjective language.
Effective communication of new discoveries is critical to the advancement and dissemination of scientific knowledge. The challenge in effective communication is accurately summarizing and portraying findings while minimizing bias. Regardless, researchers have an ethical responsibility to accurately and objectively report their findings to the greater scientific community.1,2 Because there is flexibility in what data are reported and how the data are reported, researchers may intentionally or unintentionally affect the impression the results produce in readers through the reporting and interpretation of their findings—a phenomenon labeled as ‘spin’.1,3 This is especially critical.
Recent attention has focused on how hyperbolic and subjective promotional language is used to implement spin and exaggerate aspects of research—a phenomenon labeled as ‘hype’.4,5 Linguistics experts have identified 139 words occupying eight broad categories that portray hype language (i.e., words that readers will connect with good science and advancement).6,7 Use of these subjective hype adjectives to overstate the effect of research undermines the responsibility for objectivity and diminishes trust in the scientific process by other researchers and members of the public.1,2 This is especially true in scientific abstracts because busy practitioners may erroneously rely on abstracts as a substitute for reading entire manuscripts.
Increasing usage and prevalence of hype has been documented in the broader scientific community, specifically within National Institutes of Health (NIH) grant applications,4 NIH funding announcements,8 and PubMed abstracts resulting from NIH grants.9 Evaluating trends about the use of hype language in specific medical specialties has not been done, and there is a paucity about its use within orthopaedic surgery research.
Assessing the language written in abstracts is particularly important given that abstracts to not necessarily fully report the results of the article. In addition, it is unclear whether readers use abstracts to extrapolate findings and takeaways of scientific reports rather than evaluating the entire publication.1,2 The purpose of this study was to assess the frequency of hype language in higher impact orthopaedic surgery journal abstracts. We hypothesized that use of hype language would increase, in parallel with trends observed in the broader scientific community.
Methods
No institutional review board approval was required for this retrospective bibliometric analysis, and no funding was required for this research.
Abstract Selection
Eighteen journals were selected to represent higher impact publications within each orthopaedic subspecialty and general audience orthopaedic publications. These journals included the Journal of Bone and Joint Surgery, Clinical Orthopaedics and Related Research, Journal of the American Academy of Orthopaedic Surgeons, Journal of Orthopaedic Trauma, Journal of Pediatric Orthopaedics, Journal of Pediatric Orthopaedics—B, Journal of Children's Orthopaedics, Journal of Hand Surgery, Journal of Shoulder and Elbow Surgery, American Journal of Sports Medicine, British Journal of Sports Medicine, Arthroscopy, Orthopaedic Journal of Sports Medicine, Journal of Arthroplasty, Arthroplasty Today, Spine, Global Spine Journal, and Foot and Ankle International.
All articles published in the 18 orthopaedic surgery journals from 1985 to 2020 (35 years of abstracts) were identified on PubMed (https://pubmed.ncbi.nlm.nih.gov/) using the rEntrez package and the National Center for Biotechnology Information (NCBI) application programming interface.10 Journal information, dates of publication, and abstract text were downloaded from each article. The rEntrez package directly interfaces with the PubMed database to systematically obtain abstracts and article metadata for a given set of criteria.10 Articles without publicly available abstracts were excluded.
Hype Adjectives and Categorization
Millar et al8 systematically generated a list of 139 “hype” adjectives from a set of over 900,000 NIH abstracts using a linguistic approach known as keyword analysis. Adjectives were grouped into eight “semantic categories” based on their usage: importance, novelty, rigor, scale, utility, quality, attitude, and problem. These hype adjectives and their respective semantic categories were tabulated in each abstract automatically using R. The full list of adjectives is reproduced in Table 1.
Table 1.
Table of Hype Adjectives and Semantic Categories
| Category | Adjectives |
| Importance | Compelling, critical, crucial, essential, foundational, fundamental, imperative, important, indispensable, invaluable, key, major, paramount, pivotal, significant, strategic, timely, ultimate, urgent, vital |
| Novelty | Creative, emerging, first, groundbreaking, innovative, latest, novel, revolutionary, unique, unparalleled, unprecedented |
| Rigor | Accurate, advanced, careful, cohesive, detailed, nuanced, powerful, quality, reproducible, rigorous, robust, scientific, sophisticated, strong, systematic |
| Scale | Ample, biggest, broad, comprehensive, considerable, deeper, diverse, enormous, expansive, extensive, fastest, greatest, huge, immediate, immense, interdisciplinary, international, interprofessional, largest, massive, multidisciplinary, myriad, overwhelming, substantial, top, transdisciplinary, tremendous, vast |
| Utility | Accessible, actionable, deployable, durable, easy, effective, efficacious, efficient, generalizable, ideal, impactful, intuitive, meaningful, productive, ready, relevant, rich, safer, scalable, seamless, sustainable, synergistic, tailored, tangible, transformative, user-friendly |
| Quality | Ambitious, collegial, dedicated, exceptional, experienced, intellectual, long-standing, motivated, premier, prestigious, promising, qualified, renowned, senior, skilled, stellar, successful, talented, vibrant |
| Attitude | Attractive, confident, exciting, incredible, interesting, intriguing, notable, outstanding, remarkable, surprising |
| Problem | Alarming, daunting, desperate, devastating, dire, dismal, elusive, stark, unanswered, unmet |
Statistical Analysis
Consistent with prior analyses, the frequency of hype language was normalized according to words per million (wpm).4,8,9 The frequency of each hype adjective and the frequency of each semantic category were calculated for each individual year from 1985 to 2020 (35 separate years). The average number of hype adjectives per abstract was calculated for each year from 1985 to 2020. The absolute change in frequency of individual hype adjectives was compared between abstracts published in 1985 and those published in 2020. The relative change in frequency of hype adjectives was calculated as the percentage change between 1985 and 2020. For adjectives not used in 1985, the relative change was calculated as the percentage increase from a theoretical frequency of one wpm.
All analyses were conducted using R Studio Version 2023.06.2 + 561 and R Version 4.1.2 (R Foundation).10 All statistical tests were two-sided, and the threshold for statistical significance was set at 0.05.
Results
In total, 150,174 articles were identified from the 18 journals between 1985 and 2020, of which 112,916 articles (75.2%) had an abstract publicly available in the PubMed database and were included in the analysis. The number of abstracts containing hype adjectives increased from 67.0% (948/1414) in 1985 to 92.5% (5287/5714) in 2020 (Figure 1). The average number of hype adjectives per orthopaedic surgery abstract increased from 1.1 (1407/1265) in 1985 to 2.6 (14529/5565) in 2020 (Figure 2).
Figure 1.
Graph showing number of orthopaedic abstracts containing hype adjectives from 1985 to 2020.
Figure 2.
Graph showing average number of hype adjectives per orthopaedic abstracts from 1985 to 2020.
In total, there was an absolute increase in the frequency of hype adjectives in orthopaedic abstracts from 1985 to 2020 of 913 wpm. Of the 139 hype adjectives, 87 (62.5%) increased in frequency and 40 (28.7%) decreased in frequency, while 12 (9%) were never used in either year.
The five hype adjectives with the largest absolute increases in frequency between 1985 and 2020 were quality (+324 wpm), significant (+320 wpm), systematic (+246 wpm), top (+239 wpm), and international (+201 wpm, Figure 3A). The five hype adjectives with the largest relative increases in frequency were novel (+10500%), international (+2850%), urgent (+2600%), robust (+2300%), and emerging (+1400%, Figure 3B).
Figure 3.
Graphs showing yearly frequencies of the top five hype words by largest absolute increase and largest relative increase.
Fifty hype adjectives used in abstracts published in 2020 were never used in 1985. Of these, novel (+105 wpm), urgent (+26 wpm), robust (+23 wpm), emerging (+14 wpm), and innovative (+14 wpm) had the largest absolute increases. The five adjectives with the greatest decreases in absolute frequency were successful (−348 wpm), first (−237 wpm), dire (−152 wpm), accurate (−146 wpm), and immediate (−140 wpm).
The frequency of hype adjectives within each specific category are shown in Figure 4. Between 1985 and 2020, scale adjectives increased by 374 wpm (18%), importance adjectives by 363 wpm (13%), rigor adjectives by 327 wpm (34%), and utility by 208 wpm (27%). Attitude decreased by 19 wpm (18%), novelty by 27 wpm (4%), problem by 119 wpm (28%), and quality by 193 wpm (27%).
Figure 4.
Graph showing comparison of hype language frequency in semantic categories in 1985 and 2020 in words per million.
Discussion
Our bibliometric analysis identified that the prevalence of hype language in orthopaedic surgery PubMed abstracts increased from 67% in 1985 to 93% in 2020. From 1985 to 2020, the average number of hype adjectives per orthopaedic abstract more than doubled from 1.1 to 2.6. The largest increase in absolute frequency of hype language usage promoted rigor (top three: robust, systematic, nuanced) and utility (top three: accessible, meaningful, sustainable) while the largest decrease in frequency was seen in diction endorsing a problem (eg, dire). Similar trends have been established within the NIH grant applications,4 NIH funding announcements,8 and PubMed abstracts resulting from NIH grants demonstrating connotations for increasing salesmanship in scientific literature.9 The increased utilization of hype adjectives in orthopaedic surgical abstracts threatens to embellish the rigor and utility of reported findings.
In orthopaedic surgery abstracts, 87 of 139 words increased in frequency for an average of 31.9 wpm (485%). In comparison, hype language increased in 130 of 139 adjectives for successful applications for NIH funding (mean increase of 1378%), 138 of 139 in NIH funding opportunity announcements, and 133 of 139 in abstracts describing the outcomes of NIH-funded research (mean increase of 1404%). In addition, the largest absolute increases in hype language in orthopaedics were seen in semantic categories for rigor and utility while the series of publications for NIH use of hype language found increases in the categories for novelty (eg, novel, innovative) and importance (eg, crucial, key). NIH publications have increased emphasis on originality and significance likely to help earn grants or demonstrate the value of the grant-funded research.4,8,9 Orthopaedic surgery research has also seen increased usage of this language with substantially more utilization of words emphasizing importance (3100 wpm) and scale (2,470 wpm) than other semantic categories (<1,300 wpm; Figure 4). However, the greatest absolute increase in words highlighting efficacy, usefulness, and methodological quality may reflect a focus for proving surgical techniques and technological advancements in orthopaedics.
In accordance with the publications on hype language in NIH literature, our study found a large percentage increase in novel and important adjectives.4,8,9 Among the hype words never used in orthopaedic surgery abstracts in 1985, novelty adjectives such as novel, emerging, and innovative saw the steepest rise. Several studies have highlighted that researchers and readers commonly correlate these emphasis words with good science and advancement.6,7 Although some of these hype adjectives are considered appropriate when describing results such as reporting statistically “significant” results, many are superfluous. As orthopaedic surgery becomes more competitive at various stages (eg, residency match, fellowship match, promotions) and with increasing difficulty to have articles published in higher quality journals, some may be using this salesmanship to gain advantage in the scientific process.11-13
Several other studies have identified concern with hype language and its influence on readers.3,14-16 Our study found that the average number of hype adjectives in each orthopaedic surgery abstract more than doubled from 1985 to 2020. Evaluating the language in abstracts is important when it is unclear whether readers have evaluated the entirety of the publication or use abstracts to extrapolate the findings and takeaways of scientific reports.1,2 This evaluation of a short abstract can be challenging when hype language is used to exaggerate findings. One study found that one in three psychiatry studies overstate the effectiveness of the studied intervention in comparison with the full-text results.17 Another study found that 56% (159/286) of primary care physicians accurately assessed the validity of studies based on abstracts with and without overstatements.15 The increasing use of promotional language in abstracts makes it challenging for readers to draw accurate conclusions and may also contribute to press releases and news articles which perpetuate exaggerated results.18
Care should be taken to maintain the integrity of scientific reporting and limit the focus of manuscripts and abstracts to the primary outcomes and statistically significant secondary end points or subgroup analyses.3,19 Conscious or unconscious instances of spin have the potential to lead to subsequent research or clinical interventions without adequate supporting evidence.20,21 It is also important to mention that the responsibility for decreasing promotional language in part belongs to peer reviewers and scientific journals, who serve as the final gatekeepers for scientific text before publication. Practitioners should also not rely on abstracts as a substitute for reading the entirety of scientific manuscripts.
While the strengths of this article include the large sample size of orthopaedic abstracts representing a range of subspecialties, there are several limitations. First, this analysis only includes public facing abstracts available on the PubMed database. Second, there may be disagreement as to whether some of the identified hype adjectives truly indicate exaggerated promotion of results, and disagreement may exist over the classification of hype adjectives. While the subjectivity in the identification of adjectives and their categorization is acknowledged by the original authors, they note high between-rater agreement in their analysis.4 As a sensitivity analysis, the methods were repeated excluding the words “systematic”, “significant”, and “rigorous” because these words have explicit appropriate uses (eg, “non-hype usages”) in research design and were commonly used. When excluding these words, the key finding of the study does not change. The number of hype adjectives used per orthopaedic abstract still increases from 0.9 (1161/1265) in 1985 to 2.0 (11056/5565) in 2020. This is a similar increase to that observed when including all hype adjectives, with 1.1 (1407/1265) per abstract in 1985 to 2.6 (14529/5565) in 2020.
Conclusion
This study used an automated abstract search algorithm to demonstrate a 136% increase in hype adjectives in orthopaedic surgery abstracts from 1985 to 2020. This increase in use of promotional language is likely multifactorial and may be related to increased competitiveness of careers in academic orthopaedic surgery. Regardless, this trend in hype language, whether intentional or unintentional, is concerning in that it diminishes clarity and undermines the ethical responsibility of researchers (and also peer reviewers and journals) to portray findings accurately and objectively. These results should instigate heightened awareness and purposeful assessment of publication language by researchers, reviewers, and editors. Future work could evaluate how hype language influences consumers of literature and how tools could be developed to limit the bias of abstracts.
Footnotes
None of the following authors or any immediate family member has received anything of value from or has stock or stock options held in a commercial company or institution related directly or indirectly to the subject of this article: Dr. Halvorson, Dr. Wong, and Dr. Feeley.
References
- 1.Bauchner H: Hype, the responsibility of authors and editors, and the subjective interpretation of evidence. JAMA Netw Open 2023;6:e2349125. [DOI] [PubMed] [Google Scholar]
- 2.Bauchner H, Rivara FP: The scientific communication ecosystem: The responsibility of investigators. Lancet (London, England) 2022;400:1289-1290. [DOI] [PubMed] [Google Scholar]
- 3.Boutron I, Dutton S, Ravaud P, Altman DG: Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 2010;303:2058-2064. [DOI] [PubMed] [Google Scholar]
- 4.Millar N, Batalo B, Budgell B: Trends in the use of Promotional Language (hype) in abstracts of successful national Institutes of Health grant applications, 1985-2020. JAMA Netw Open 2022;5:e2228676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Millar N, Salager-Meyer F, Budgell B. “It is important to reinforce the importance of …”: ‘Hype’ in reports of randomized controlled trials. English Specif Purposes. 2019;54:139-151. [Google Scholar]
- 6.Cohen BA: How should novelty be valued in science? Elife 2017;6:e28699. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Kumar S, Krishna LG, Singh D, Kumar P, Arora S, Dhaka S: Evaluation of functional outcome and complications of locking calcaneum plate for fracture calcaneum. J Clin Orthop Trauma 2015;6:147-152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Millar N, Batalo B, Budgell B: Trends in the use of Promotional Language (hype) in national Institutes of Health funding opportunity announcements, 1992-2020. JAMA Netw Open 2022;5:e2243221. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Millar N, Batalo B, Budgell B: Promotional Language (hype) in abstracts of publications of national Institutes of health–funded research, 1985-2020. JAMA Netw Open 2023;6:e2348706. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Winter DJ: Rentrez: An R package for the NCBI eUtils API. The R J 2017;9:520-526. [Google Scholar]
- 11.Carr M, Anderson JM, Shepard S, et al. : An analysis of publication trends of orthopedic surgery residency graduates in relation to academic achievement. J Osteopathic Med 2022;122:195-202. [DOI] [PubMed] [Google Scholar]
- 12.Ngaage LM, Mb C, Xue S, et al. : The orthopaedic match: Defining the academic profile of successful candidates. J Am Acad Orthop Surg 2021;29:921-928. [DOI] [PubMed] [Google Scholar]
- 13.Toci GR, Elsner JA, Bigelow BF, Bryant BR, LaPorte DM: Medical student research productivity: Which variables are associated with matching to a highly ranked orthopaedic residency program? J Surg Educ 2021;78:512-518. [DOI] [PubMed] [Google Scholar]
- 14.Suganuma AM, Shinohara K, Imai H, Takeshima N, Hayasaka Y, Furukawa TA: Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: A study protocol for a meta-epidemiological investigation. BMJ Open 2016;6:e009832. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Tsujimoto Y, Aoki T, Shinohara K, et al. : Physician characteristics associated with proper assessment of overstated conclusions in research abstracts: A secondary analysis of a randomized controlled trial. PLoS One 2019;14:e0211206. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.van den Besselaar P, Mom C: The effect of writing style on success in grant applications. J Informetrics 2022;16:101257. [Google Scholar]
- 17.Shinohara K, Suganuma AM, Imai H, Takeshima N, Hayasaka Y, Furukawa TA: Overstatements in abstract conclusions claiming effectiveness of interventions in psychiatry: A meta-epidemiological investigation. PLoS One 2017;12:e0184786. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Woloshin S, Schwartz LM: Press releases: Translating research into news. JAMA 2002;287:2856-2858. [DOI] [PubMed] [Google Scholar]
- 19.Boutron I, Altman DG, Hopewell S, Vera-Badillo F, Tannock I, Ravaud P: Impact of spin in the abstracts of articles reporting results of randomized controlled trials in the field of cancer: The SPIIN randomized controlled trial. J Clin Oncol 2014;32:4120-4126. [DOI] [PubMed] [Google Scholar]
- 20.Khan MS, Lateef N, Siddiqi TJ, et al. : Level and prevalence of spin in published cardiovascular randomized clinical trial reports with statistically nonsignificant primary outcomes: A systematic review. JAMA Netw Open 2019;2:e192622. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Chiu K, Grundy Q, Bero L: ‘Spin’ in published biomedical literature: A methodological systematic review. Plos Biol 2017;15:e2002173. [DOI] [PMC free article] [PubMed] [Google Scholar]