Abstract
Objectives/Hypothesis
To understand the effect of the COVID‐19 pandemic on the volume, quality, and impact of otolaryngology publications.
Study Design
Retrospective analysis.
Methods
Fifteen of the top peer‐reviewed otolaryngology journals were queried on PubMed for COVID and non‐COVID‐related articles from April 1, 2020 to March 31, 2021 (pandemic period) and pre‐COVID articles from the year prior. Information on total number of submissions and rate of acceptance were collected from seven top‐ranked journals.
Results
Our PubMed query returned 759 COVID articles, 4,885 non‐COVID articles, and 4,200 pre‐COVID articles, corresponding to a 34% increase in otolaryngology publications during the pandemic period. Meta‐analysis/reviews and miscellaneous publication types made up a larger portion of COVID publications than that of non‐COVID and pre‐COVID publications. Compared to pre‐COVID articles, citations per article 120 days after publication and Altmetric Attention Score were higher in both COVID articles (citations/article: 2.75 ± 0.45, P < .001; Altmetric Attention Score: 2.05 ± 0.60, P = .001) and non‐COVID articles (citations/article: 0.03 ± 0.01, P = .002; Altmetric Attention Score: 0.67 ± 0.28, P = .016). COVID manuscripts were associated with a 1.65 times higher acceptance rate compared to non‐COVID articles (P < .001).
Conclusions
COVID‐19 was associated with an increase in volume, citations, and attention for both COVID and non‐COVID articles compared to pre‐COVID articles. However, COVID articles were associated with lower evidence levels than non‐COVID and pre‐COVID articles.
Level of Evidence
3 Laryngoscope, 132:1364–1373, 2022
Keywords: Bibliometrics, COVID‐19, publication trends, scientific publication
INTRODUCTION
The COVID‐19 pandemic has led to an upsurge in publications aiming to elucidate the clinical features, pathophysiology, and management of the disease. 1 Due to the urgency of the pandemic, the scientific community has been forced to balance rigorous, time‐consuming research with the need for rapid publication and dissemination of knowledge. 2 , 3
Striking this balance has been challenging, and multiple studies have analyzed research challenges within the COVID‐19 literature. 1 , 4 , 5 One study investigating the highest impact medical journals found that COVID‐19 publications were associated with lower levels of evidence and higher citation rates than that of non‐COVID‐19‐related research. 6 Pandemic‐related research has also been criticized for small sample sizes, lack of control groups, and premature access to pre‐prints. 7 Additionally, a high retraction rate of COVID‐19 publications has caused great alarm in the scientific community, affecting even the most prominent medical journals. 8 , 9 , 10 Though COVID‐19 publication quality has been criticized, the research community has been ultimately successful in identifying treatments and developing vaccines. 11 , 12 , 13 , 14
The otolaryngology (ENT) research community has been similarly affected by the pandemic. Two publications have demonstrated an increase of 58% and 42% in ENT publications from German and Italian universities during the pandemic, compared to prior years. 15 , 16 Another study of the top 20 ENT journals found that 75% of the 166 COVID‐19 related publications were either editorials, comments, or letters, which generally correspond to lower levels of scientific evidence. 17 Additionally, little attention has been given to non‐traditional, but increasingly influential, measures of ENT research impact, like social media reach. 18 , 19 The comprehensive impact of COVID‐19 on ENT research quantity and quality has yet to be formally described. We sought to elucidate these effects by analyzing the publications of 15 top‐ranked ENT journals during the COVID‐19 pandemic.
MATERIALS AND METHODS
Journal Selection
We identified the top 20 otolaryngology journals by h5‐index as listed on Google Scholar. 20 From this list, we excluded the Otolaryngology Clinics of North America because it does not publish peer‐reviewed publications. We also excluded open access journals (i.e., Brazilian Journal of Otorhinolaryngology and Journal of Otolaryngology‐Head & Neck Surgery). After initial data collection was performed, we excluded Hearing Research and Audiology and Neurotology because neither journal published any COVID‐related articles. Thus, 15 journals were included in the final analysis.
Data Collection
This study met exemption criteria established by The George Washington University Institutional Review Board. We collected data from PubMed, utilizing its Entrez Programming Utilities (E‐utilities) interface. 21 The articles from the aforementioned 15 otolaryngology journals were divided into three groups to compare the impact of COVID on otolaryngology literature. These article groups will be henceforth referred to as COVID articles, non‐COVID articles published during the pandemic, and pre‐COVID articles.
COVID articles from PubMed were queried from April 1, 2020 to March 31, 2021 using the search terms: COVID, COVID 19, COVID‐19, Coronavirus, Coronavirus 19, Coronavirus‐19, or SARS‐COV‐2. Non‐COVID articles were queried on PubMed from April 1, 2020 to March 31, 2021 by excluding COVID search terms. Pre‐COVID articles were queried from April 1, 2019 to March 31, 2020. While COVID‐19 was known as early as December 2019, we defined the “pandemic period” from April 1, 2020 to March 31, 2021 to account for publication lag. We separately queried publication data from PubMed on common ENT topics using the search terms otitis media, obstructive sleep apnea, chronic sinusitis, hoarseness, and sensorineural hearing loss from April 1, 2011 to March 31, 2021.
Data on unique COVID and non‐COVID article submission and acceptance count were requested from 9 of the top‐ranked ENT journals. Resubmissions were excluded. Data included the number of articles submitted and accepted for the pre‐COVID (April 1, 2019–March 31, 2020), non‐COVID (April 1, 2020–March 31, 2021), and COVID (April 2020–March 31, 2021) groups. Seven of the nine journals provided complete data.
Article Characteristics
Data collected from PubMed for each article included authorship information, publication date and type, and overall citation count at time of data collection (May 27, 2021) and within 120 days of publication. The top 5 countries by number of COVID articles published were identified for analysis and the remainder of countries were grouped together.
Publication type refers to PubMed‐specified terms and were grouped into five categories for analysis that aimed to parallel Oxford level of evidence groups. 22 , 23 “Meta‐analysis/Reviews” included Meta‐Analysis, Systematic Review, Review, and Practice Guideline publication types. “Randomized Control Trials” included Randomized Controlled Trial, Clinical Trial, Clinical Trial Phases I‐IV, Controlled Clinical Trial, and Equivalence Trial publication types. “Observation/Cohort/Other Studies” included Evaluation Study, Comparative Study, Multicenter Study, Observational Study, Validation Study, and Clinical Study publication types. “Case Studies” included Case Report and Twin Study publication types. “Miscellaneous” included all other publication types, a majority of which were Letter, Comment, and Editorial publication types. The generic publication type Journal Article is listed for a majority of publications without a more specific tag and were included in the “Observation/Cohort/Other Studies” group.
Altmetric (London, UK) was used to obtain information on article “attention.” The Altmetric Attention Score is a weighted function of article mentions across mainstream news outlets, social media websites, blogs, policy documents, and other online platforms. We also used Altmetric data to obtain the number of times an article is mentioned specifically on social media platforms.
Statistical Analysis
Descriptive statistics were used to compare article features using Chi‐square test for categorical variables and ANOVA test for continuous variables. Multivariate regression was performed to analyze the association between COVID publications and citation/attention metrics. All analyses were completed using Python (Python Software Foundation, Delaware, United States) and R (R Foundation for Statistical Computing, Vienna, Austria). A two‐sided P‐value less than .05 was considered statistically significant. All data collection and analysis code are available at https://bit.ly/3fXwfOy.
RESULTS
Publication Trends
The query on PubMed returned 9,847 articles. Three studies with “veterinary” publication types were excluded, resulting in 9,844 articles for final inclusion: 759 COVID articles, 4,885 non‐COVID articles, and 4,200 pre‐COVID articles. This corresponds to a 34.4% increase in publications from the pre‐pandemic period (April 1, 2019–March 31, 2020) to the pandemic period (April 1, 2020–March 31, 2021). During the pandemic period, over three times as many COVID articles were published as those on other popular ENT topics (Table I). However, during the pandemic period, linear regression of monthly publications demonstrated a gradual decline in COVID publications over time (−6.26 ± 1.42 publications per month, P < .001). No significant change was seen in the monthly number of non‐COVID publications over time (P = .571).
TABLE I.
Number of Publications by Selected ENT Topic.
| Topic | Number of Publications During COVID period (April 1, 2020–March 31, 2021) | Number of Publications pre‐COVID period (April 1, 2019–March 31, 2020) | Number of Publications in Past Decade (April 1, 2011–March 31, 2021) |
|---|---|---|---|
| COVID | 759 | 0 | 0 |
| Chronic sinusitis | 120 | 211 | 2,067 |
| Hoarseness | 26 | 21 | 242 |
| Obstructive sleep apnea | 175 | 148 | 1,525 |
| Otitis media | 167 | 155 | 1,754 |
| Sensorineural hearing loss | 237 | 279 | 2,865 |
Journal of Laryngology & Otology (30.0% COVID publications, n = 66) and Otolaryngology‐Head and Neck Surgery (29.6%, n = 175) had the highest proportion of COVID publications during the pandemic period, while Otology & Neurotology (2.3%, n = 9) and Ear and Hearing (2.0%, n = 3) had the fewest (Fig. 1). Most journals published more COVID “miscellaneous” articles and fewer COVID “Observation/Cohort/Other studies” than those among non‐COVID or pre‐COVID articles (Fig. 2).
Fig. 1.
COVID and non‐COVID publication by journal. Percent of non‐COVID versus COVID publication by 15 ENT journals from April 1, 2020 to March 31, 2021 demonstrate that certain journals published a higher portion of COVID articles, especially J Laryngol Otol (30.0% COVID publications, n = 66), Otolaryngol Head Neck Surg (29.6%, n = 175), Int Forum of Allergy Rhinol (21.4%, n = 49), JAMA Otolaryngol Head Neck (21.4%, n = 59), and Head Neck (20.6%, n = 104).
Fig. 2.
Publication type by journal and publication period. Data acquired from PubMed demonstrates that a majority of top ENT journals published more “Miscellaneous” and fewer “Observation/cohort/other” studies than non‐COVID and pre‐COVID articles. Additionally, journals tended to publish more “meta‐analysis/review” articles on COVID.
Manuscript Submissions and Acceptances
Seven journals contributed data on submitted and accepted publications for the COVID, non‐COVID, and pre‐COVID groups. There was a 38.8% increase in number of submissions from the pre‐pandemic period (10,538 manuscripts) to the pandemic period (14,622 manuscripts), with only a 21.2% increase in the number of accepted publications (2,567 increased to 3,111 publications). In 5 of 7 journals, COVID articles had a significantly higher acceptance rate than non‐COVID articles (Fig. 3). Overall, COVID articles had a 1.65 times higher acceptance rate than non‐COVID articles (P < .001). There was a significant decrease in acceptance rate from pre‐COVID to non‐COVID groups in 4 of 7 journals (Fig. 3).
Fig. 3.
COVID, non‐COVID, and pre‐COVID submissions. Number of unique submissions and accepted publications from April 2019–April 2020 (pre‐COVID) and April 2020–April 2021 (non‐COVID and COVID) show that a majority of selected top ENT journals had higher acceptance rates for COVID articles, compared to non‐COVID articles. Percent of articles accepted annotated on graph. Chi‐square P‐values identifying significant differences in acceptance rates between time period for each journal is shown on the figure as follows: *P < .05, **P < .01, ***P < .001.
Two journals were only able to provide partial data. International Journal of Pediatric Otorhinolaryngology noted a 36.3% acceptance rate in 2019, 33.0% in 2020, and approximately 30% for COVID articles. European Archives of Oto‐Rhino‐Laryngology noted a pre‐COVID acceptance rate of 26.0% and a combined COVID and non‐COVID acceptance rate of 25.7%.
Publication Statistics
Comparison of COVID, non‐COVID, and pre‐COVID article features is shown in Table II. Meta‐analysis/reviews and miscellaneous publication types made up a larger portion of COVID publications than non‐COVID and pre‐COVID (Meta‐analysis/reviews: 17.1%, 7.2%, and 8.6%, respectively, P < .001; miscellaneous: 21.3%, 8.7%, 8.2%, P < .001). Fewer non‐COVID randomized control trials (n = 73, 1.5%) were published compared to pre‐COVID randomized control trials (n = 137, 3.3%, P < .001). The United States published a larger portion of COVID publications (29.0%) and non‐COVID (23.2%) publications than pre‐COVID publications (15.5%, P < .001). Altmetric Attention Score is significantly higher in COVID articles than non‐COVID or pre‐COVID (24.9 vs. 2.0 vs. 2.9 points, P < .001). When analyzing the top 100 most cited publications in each group, most of the previous trends persisted (Table II).
TABLE II.
Publication Statistics.
| Variable | COVID † Publications, n (%) | Non‐COVID † Publications, n (%) | Pre‐COVID † Publications, n (%) | P |
|---|---|---|---|---|
| Among all publications | n = 759 | n = 4,885 | n = 4,200 | |
| Publication type | <.001* | |||
| Meta‐analysis/reviews | 130 (17.1) | 352 (7.2) | 363 (8.6) | |
| Randomized control trials | 2 (0.3) | 73 (1.5) | 137 (3.3) | |
| Observational/cohort/other studies | 438 (57.7) | 3,880 (79.4) | 3,150 (75.0) | |
| Case studies | 27 (3.6) | 155 (3.2) | 206 (4.9) | |
| Miscellaneous | 162 (21.3) | 425 (8.7) | 344 (8.2) | |
| Top 5 countries by most COVID articles | <.001* | |||
| United States | 220 (29.0) | 1,133 (23.2) | 653 (15.5) | |
| Italy | 89 (11.7) | 181 (3.7) | 153 (3.6) | |
| France | 25 (3.3) | 99 (2.0) | 97 (2.3) | |
| India | 24 (3.2) | 124 (2.5) | 70 (1.7) | |
| Canada | 21 (2.8) | 161 (3.3) | 161 (3.8) | |
| Others | 380 (50.0) | 3,187 (65.2) | 3,066 (73.0) | |
| h5‐index of journal (mean ± SD) | 40.7 ± 8.5 | 39.7 ± 9.1 | 39.0 ± 9.2 | <.001* |
| Altmetric Attention Score (mean ± SD) | ||||
| Total score | 24.9 ± 173.2 | 2.0 ± 18.2 | 2.9 ± 26.5 | <.001* |
| Score 3 mo after publication | 2.1 ± 14.9 | 0.6 ± 15.7 | 0.1 ± 1.0 | <.001* |
| Score 6 mo after publication | 5.0 ± 23.1 | 1.2 ± 17.0 | 0.2 ± 1.5 | <.001* |
| Social media posts | 21.0 ± 211.5 | 2.3 ± 6.5 | 3.2 ± 10.8 | <.001* |
| Among top 100 most cited publications | n = 100 | n = 100 | n = 100 | |
| Publication type | .020* | |||
| Meta‐analysis/reviews | 25 | 21 | 25 | |
| Randomized control trials | 1 | 3 | 6 | |
| Observation/cohort/other studies | 61 | 73 | 65 | |
| Case studies | 0 | 1 | 0 | |
| Miscellaneous | 13 | 2 | 4 | |
| Top 5 countries by most COVID articles | <.001* | |||
| United States | 28 | 17 | 14 | |
| Italy | 25 | 8 | 8 | |
| France | 6 | 1 | 2 | |
| India | 0 | 0 | 0 | |
| Canada | 3 | 6 | 4 | |
| Others | 38 | 68 | 72 | |
| h5‐index of journal (mean ± SD) | 42.7 ± 6.8 | 43.1 ± 6.9 | 42.4 ± 7.5 | .777 |
| Altmetric Attention Score (mean ± SD) | ||||
| Total score | 87.7 ± 255.7 | 9.0 ± 37.9 | 17.5 ± 52.0 | <.001* |
| Score 3 mo after publication | 3.7 ± 19.1 | 2.0 ± 12.8 | 1.1 ± 5.6 | .396 |
| Score 6 mo after publication | 10.5 ± 38.8 | 5.8 ± 36.8 | 1.5 ± 6.2 | .124 |
| Social media posts | 48.8 ± 135.7 | 6.0 ± 13.3 | 14.8 ± 31.2 | <.001* |
P < .05.
COVID publications refers to publications queried from April 1, 2020–March 31, 2021 regarding COVID‐19. Non‐COVID refers to publications during the COVID time period (April 1, 2020–March 31, 2021) that are unrelated to COVID‐19. Pre‐COVID refers to publications unrelated to COVID in the 1 year period before COVID (April 1, 2019–March 31, 2020).
SD = standard deviation.
Citation Statistics
Total citation count and citation count 120 days postpublication are detailed in Table III. COVID publications were cited significantly more overall (P < .001) and at 120 days (P < .001), and this significance persisted when broken down by publication type (Table III). For the top 100 most cited articles in each group, COVID articles were still cited more often overall (46.8 vs. 3.4 vs. 8.0 citations/article, P < .001) and at 120 days postpublication (16.0 vs. 1.0 vs. 0.4 citations/article, P < .001). This trend persisted in all publication types except “miscellaneous” articles. Table IV lists the 10 most cited nonmiscellaneous COVID articles. They focus on characterizing olfactory and gustatory symptoms and outlining safe practices for otolaryngologic procedures.
TABLE III.
Citation Statistics.
| Variable | COVID † Citations/Article (mean ± SD) | Non‐COVID † Citations/Article (mean ± SD) | Pre‐COVID † Citations/Article (mean ± SD) | P |
|---|---|---|---|---|
| Among all publications | ||||
| Overall citations per paper | ||||
| Total | 8.25 ± 33.92 | 0.20 ± 0.68 | 0.88 ± 1.65 | <.001* |
| Within 120 d | 2.82 ± 12.11 | 0.06 ± 0.31 | 0.04 ± 0.21 | <.001* |
| Publication type | ||||
| Meta‐analysis/reviews | ||||
| Total | 12.16 ± 25.79 | 0.43 ± 1.12 | 1.79 ± 3.11 | <.001* |
| Within 120 d | 4.41 ± 11.91 | 0.08 ± 0.33 | 0.7 ± 0.31 | <.001* |
| Randomized control trials | ||||
| Total | 9.50 ± 4.95 | 0.22 ± 0.51 | 1.18 ± 1.67 | <.001* |
| Within 120 d | 1.50 ± 0.71 | 0.04 ± 0.20 | 0.04 ± 0.19 | <.001* |
| Observation/cohort/other studies | ||||
| Total | 8.61 ± 40.40 | 0.20 ± 0.67 | 0.85 ± 1.44 | <.001* |
| Within 120 d | 2.79 ± 13.83 | 0.06 ± 0.32 | 0.04 ± 0.21 | <.001* |
| Case studies | ||||
| Total | 3.37 ± 2.84 | 0.12 ± 0.41 | 0.40 ± 0.74 | <.001* |
| Within 120 d | 1.26 ± 1.51 | 0.02 ± 0.18 | 0.00 ± 0.00 | <.001* |
| Miscellaneous | ||||
| Total | 4.94 ± 20.60 | 0.08 ± 0.35 | 0.29 ± 1.14 | <.001* |
| Within 120 d | 1.92 ± 7.38 | 0.04 ± 0.25 | 0.03 ± 0.20 | <.001* |
| Among top 100 most cited publications | ||||
| Overall citations per paper | ||||
| Total | 46.78 ± 83.80 | 3.38 ± 2.38 | 8.00 ± 4.24 | <.001* |
| Within 120 d | 16.03 ± 30.17 | 1.01 ± 1.37 | 0.41 ± 0.70 | <.001* |
| Publication type | ||||
| Meta‐analysis/reviews | ||||
| Total | 45.60 ± 45.62 | 3.86 ± 2.24 | 9.80 ± 7.03 | <.001* |
| Within 120 d | 17.92 ± 22.76 | 0.71 ± 0.96 | 0.36 ± 0.70 | <.001* |
| Randomized control trials | ||||
| Total | 13.0 ± N/A‡ | 2.00 ± N/A‡ | 6.67 ± 1.37 | <.001* |
| Within 120 d | 2.0 ± N/A‡ | 0.00 ± N/A† | 0.33 ± 0.52 | .015* |
| Observation/cohort/other studies | — | |||
| Total | 48.39 ± 99.84 | 3.32 ± 2.49 | 7.37 ± 2.59 | <.001* |
| Within 120 d | 15.57 ± 34.52 | 1.08 ± 1.46 | 0.42 ± 0.70 | <.001* |
| Case studies | ||||
| Total | N/A‡ | 3.00 ± N/A‡ | N/A‡ | |
| Within 120 d | N/A‡ | 2.00 ± N/A‡ | N/A‡ | |
| Miscellaneous | ||||
| Total | 44.08 ± 61.77 | 3.00 ± 1.41 | 9.00 ± 3.16 | .392 |
| Within 120 d | 15.62 ± 22.14 | 2.50 ± 2.12 | 0.75 ± 0.96 | .344 |
P < .05.
COVID publications refers to publications queried from April 1, 2020–March 31, 2021 regarding COVID‐19. Non‐COVID refers to publications during the COVID time period (April 1, 2020–March 31, 2021) that are unrelated to COVID‐19. Pre‐COVID refers to publications unrelated to COVID in the 1 year period before COVID (April 1, 2019–March 31, 2020).
N/A noted for mean if no publications in category and for standard deviation if only 1 publication noted in that category.
SD = standard deviation.
TABLE IV.
Ten Most Cited COVID Articles.
| Citations | Citations at 120 d | Title | Journal | Publication Type |
|---|---|---|---|---|
| 735 | 252 | Olfactory and gustatory dysfunctions as a clinical presentation of mild‐to‐moderate forms of the coronavirus disease (COVID‐19): A multicenter European study. | Eur Arch Otorhinolaryngol | Observational/cohort/other studies |
| 244 | 85 | Association of chemosensory dysfunction and COVID‐19 in patients presenting with influenza‐like symptoms. | Int Forum Allergy Rhinol | Observational/cohort/other studies |
| 208 | 113 | Safety recommendations for evaluation and surgery of the head and neck during the COVID‐19 pandemic. | JAMA Otolaryngol Head Neck Surg | Meta‐analysis/reviews |
| 207 | 62 | Smell dysfunction: A biomarker for COVID‐19. | Int Forum Allergy Rhinol | Observational/cohort/other studies |
| 145 | 40 | Objective evaluation of anosmia and ageusia in COVID‐19 patients: Single‐center experience on 72 cases. | Head Neck | Observational/Cohort/Other Studies |
| 126 | 54 | Endonasal instrumentation and aerosolization risk in the era of COVID‐19: Simulation, literature review, and proposed mitigation strategies. | Int Forum Allergy Rhinol | Observational/cohort/other studies |
| 126 | 31 | The prevalence of olfactory and gustatory dysfunction in COVID‐19 patients: A systematic review and meta‐analysis. | Otolaryngol Head Neck Surg | Meta‐analysis/reviews |
| 111 | 36 | Self‐reported olfactory loss associates with outpatient clinical course in COVID‐19. | Int Forum Allergy Rhinol | Observational/cohort/other studies |
| 108 | 33 | COVID‐19 anosmia reporting tool: Initial findings. | Otolaryngol Head Neck Surg | Meta‐analysis/reviews |
| 103 | 52 | Surgical considerations for tracheostomy during the COVID‐19 pandemic: Lessons learned from the severe acute respiratory syndrome outbreak. | JAMA Otolaryngol Head Neck Surg | Meta‐analysis/reviews |
Table excludes miscellaneous publication types.
Multivariate regression analysis of the relationship between article characteristics and citations and attention metrics is detailed in Table V. When controlling for publication type, country of publication, and journal published, COVID publications were cited 7.25 ± 1.27 (P < .001) more times than pre‐COVID articles, while non‐COVID articles were cited 0.64 ± 0.36 fewer times (P < .001). When analyzing citations within 120 days postpublication, both COVID and non‐COVID articles were associated with 2.75 ± 0.45 (P < .001) and 0.03 ± 0.01 (P = .002) more citations than pre‐COVID articles. This trend persisted with the 3‐month Altmetric Attention Score. When analyzing social media posts per article, no difference existed between pre‐COVID and non‐COVID publications although COVID publications were posted 18.3 ± 8.3 more times than pre‐COVID articles (P < .001) when controlling for the same article characteristics.
TABLE V.
Multivariate Regression of Article Characteristics on Citation and Attention Metrics.
| Variable Coefficient ± Standard Error | Citations/Paper | Citations/Paper Within 120 d | Social Media Posts | Altmetric Attention Score at 3 mo |
|---|---|---|---|---|
| Publication period | ||||
| Pre‐COVID | ref. | ref. | ref. | ref. |
| Non‐COVID | −0.64 ± 0.36 *** | 0.03 ± 0.01 ** | −0.10 ± 0.48 | 0.67 ± 0.28 * |
| COVID | 7.25 ± 1.27 *** | 2.75 ± 0.45 *** | 18.32 ± 8.28 * | 2.05 ± 0.60 ** |
| Publication type | ||||
| Observation/cohort/other studies | ref. | ref. | ref. | ref. |
| Meta‐analysis/reviews | 1.00 ± 0.41 * | 0.25 ± 0.17 | 0.24 ± 0.82 | 1.57 ± 1.33 |
| Randomized control trials | 0.25 ± 0.13 | 0.00 ± 0.03 | 1.34 ± 0.65 * | −0.17 ± 0.10 |
| Case studies | −0.54 ± 0.13 *** | −0.12 ± 0.05 * | 12.10 ± 13.79 | 0.06 ± 0.28 |
| Miscellaneous | 0.62 ± 0.13 *** | −0.26 ± 0.14 | −5.28 ± 1.76 ** | −0.80 ± 0.21 *** |
| Top 5 countries by most COVID articles | ||||
| United States | ref. | ref. | ref. | ref. |
| Canada | 0.67 ± 0.70 | 0.25 ± 0.24 | 2.35 ± 2.13 | 0.11 ± 0.73 |
| France | 3.95 ± 3.21 | 1.37 ± 1.11 | 2.29 ± 2.66 | −0.08 ± 0.25 |
| India | −0.38 ± 0.22 | −0.13 ± 0.08 | −1.03 ± 0.56 | −0.01 ± 0.19 |
| Italy | 1.47 ± 0.74 * | 0.42 ± 0.24 | −0.26 ± 2.21 | −0.14 ± 0.25 |
| Others | 0.13 ± 0.16 | 0.04 ± 0.06 | 0.25 ± 0.89 | 0.12 ± 0.19 |
| Journal | ||||
| Acta Otolaryngol | −0.26 ± 0.19 | −0.01 ± 0.06 | −1.36 ± 0.47 ** | −0.33 ± 0.16 * |
| Am J Rhinol Allergy | −0.38 ± 0.23 | −0.05 ± 0.08 | 0.22 ± 0.56 | −0.40 ± 0.18 * |
| Ann Otol Rhinol Laryngol | −0.44 ± 0.19 * | −0.09 ± 0.06 | −1.61 ± 0.71 * | −0.41 ± 0.19 * |
| Auris Nasus Larynx | −0.39 ± 0.23 | −0.08 ± 0.08 | −2.31 ± 1.16 * | −0.51 ± 0.18 ** |
| Clin Otolaryngol | −0.20 ± 0.26 | −0.03 ± 0.09 | 1.68 ± 0.77 * | 0.13 ± 0.20 |
| Ear Hear | 0.45 ± 0.18 * | 0.26 ± 0.07 *** | 1.18 ± 0.72 | −0.22 ± 0.20 |
| Eur Arch Otorhinolaryngol | 0.15 ± 0.51 | 0.08 ± 0.17 | −1.17 ± 0.43 ** | 0.19 ± 0.24 |
| Head Neck | 0.54 ± 0.30 | 0.17 ± 0.10 | 0.11 ± 0.51 | −0.42 ± 0.21 * |
| Int Forum Allergy Rhinol | 1.99 ± 0.82 * | 0.63 ± 0.28 * | 5.31 ± 1.81 ** | 3.64 ± 2.57 |
| Int J Pediatr Otorhinolaryngol | −0.37 ± 0.17 * | −0.05 ± 0.06 | −1.56 ± 0.47 ** | −0.47 ± 0.18 ** |
| J Laryngol Otol | −1.13 ± 0.26 *** | −0.31 ± 0.09 *** | −3.09 ± 1.26 * | −0.28 ± 0.25 |
| JAMA Otolaryngol Head Neck Surg | 0.68 ± 0.53 | 0.34 ± 0.26 | 32.94 ± 9.75 ** | 1.27 ± 0.38 ** |
| Laryngoscope † | ref. | ref. | ref. | ref. |
| Otol Neurotol | 0.26 ± 0.21 | 0.05 ± 0.07 | 0.17 ± 0.46 | −0.09 ± 0.27 |
| Otolaryngol Head Neck Surg | −0.10 ± 0.31 | −0.05 ± 0.10 | −1.71 ± 0.78 * | −0.55 ± 0.26 * |
All significant values are bolded for clarity.
P < .05.
P < .01.
P < .001.
Laryngoscope was chosen as the reference group because this journal had the most publications overall in our dataset (n = 1,486).
Notably, the country of publication was generally not associated with citation or attention metrics (Table V). Similarly, publication type and journal of publication had variable associations with only minor effect sizes on citation and attention metrics (Table V). However, more journals were significantly associated with the Altmetric Attention Score at 3 months and the number of social media posts than were associated with citations per article (Table V). A separate multivariate regression found no significant relationship between citations per article and social media posts (β coefficient of 0.02 ± 0.15, P = .908) when controlling for publication type, COVID group, journal, and country.
DISCUSSION
COVID‐19 had a significant impact on ENT publications. There was a 34.4% increase in publications during the COVID period compared to the year prior with a greater focus on COVID‐19 than other common ENT topics. However, we found overall research output on COVID‐19 in ENT is already beginning to decline. We utilized citations per article as a marker for impact, publication type as a proxy for quality, and Altmetric Attention Score and social media mentions as a representation of attention across a variety of media (e.g., social media, Wikipedia, blogs, and policy documents). While we acknowledge the imperfect nature of these metrics, we find strong evidence that the COVID‐19 pandemic period was correlated with higher impact and attention for both COVID and non‐COVID ENT articles, as well as a dramatic increase in volume of publications.
Both COVID (n = 759) and non‐COVID articles (n = 4,885) contributed to the increase in publications (pre‐COVID, n = 4,200). There was a corresponding increase in submission in non‐COVID articles compared to the pre‐COVID period, which may be due to increased research productivity among surgeons as elective surgeries were postponed. 24 , 25 , 26 There was a significant increase in “Observation/Cohort/Other” articles, which often includes research that can be completed remotely, such as retrospective analyses. There were 685 more publications in the non‐COVID group compared to the pre‐COVID group, of which the United States produced 480. Thus, the United States represented a larger share of non‐COVID ENT publications (23%) than in the prior year (16%). This increase in research productivity was likely due to the pandemic's limiting effect on clinical practice, particularly elective procedures, of which the U.S. cancelled more than other countries. 27 , 28 , 29 , 30
Studies have demonstrated lower levels of evidence among COVID compared to non‐COVID articles. Among the top medical journals, COVID publications were 186‐fold more likely to be of lower evidence. 6 High‐ and low‐ranking journals were equally as likely to publish COVID articles that were ultimately retracted. 31 Another study found 72 retractions of COVID‐19 articles, including those published in well‐regarded journals like the New England Journal of Medicine and The Lancet. 32 These retractions heighten the concern regarding COVID research quality. We found that ENT COVID publications were more likely to be of “Miscellaneous” and “Meta‐analysis/Reviews” publication types. While meta‐analysis and reviews are generally associated with higher levels of evidence, we believe this to be the exception for COVID articles. Given the relatively fewer number of “Randomized Controlled Trial” and “Observational/Cohort/Other Study” publication types, these review articles are likely based on weaker evidence.
Notably, we found that the COVID‐19 pandemic impacted non‐COVID ENT publications. Compared to the pre‐pandemic period, fewer non‐COVID “Meta‐analysis/Review” and “Randomized Control Trial” articles were published during COVID‐19. This change in publication pattern may be due to logistical challenges associated with COVID‐19. For example, it has been more difficult to run clinical trials during COVID‐19 due to safety challenges, patient appointment interruptions, and the shift in focus to COVID‐related research. 33 , 34
Our study showed that COVID articles generally had a higher acceptance rate than non‐COVID articles across several top ENT journals. Within ENT, COVID articles have focused on reducing the spread of disease during ENT procedures and on investigating olfactory symptoms, which is consistent with our findings. 17 However, COVID‐19 publications across a variety of scientific disciplines have been associated with lower levels of evidence and a shorter time to publication, likely due to the urgency of the pandemic. 6 , 32 While COVID‐19 research has led to the eventual discovery of important treatments like remdesivir, they have also led to misuse, as in the case of hydroxychloroquine. 35 It is important to understand the limitations and cost of COVID‐19 on ENT research, specifically noted by the lower levels of evidence in COVID articles and the decrease in randomized control trials and meta‐analysis/reviews in non‐COVID publications.
Nonetheless, during COVID‐19, an increase in the impact and attention was noted among top ENT journals for both COVID and non‐COVID articles. While the pandemic period was associated with more citations and attention for COVID articles, it was also associated with a modest increase in citations and attention score for non‐COVID articles when compared to pre‐COVID articles. This suggests that the readership brought to ENT journals from COVID articles had a spillover effect for non‐COVID articles. Overall citations per article and attention is higher in pre‐COVID than non‐COVID articles likely because pre‐COVID articles have been in circulation for a longer period of time. Generally, the journal of publication had little to no impact on citations per article. Int Forum Allergy Rhinol and Ear Hear were associated with the highest citations per article, while JAMA Otolaryngol Head Neck Surg was associated with the highest attention scores and number of social media mentions per article. Publication in JAMA Otolaryngol Head Neck Surg was associated with 32 more social media posts per article, likely due to their large online presence associated with the JAMA Network. However, it is important to note, we found no relationship between social media posts and citations per publication when controlling for journal and article characteristics, which differs from published findings. 36 , 37 , 38 , 39 This difference could be due to our analysis involving only the highest impact journals in a small specialty. Thus, scientific readership is relatively fixed and social media may have a more limited impact on citations. Additionally, publication type had little to no impact on citation or attention metrics.
The major limitation of this study is in its analysis of the level of evidence of published articles. Because we utilized PubMed “publication type” tags, we were not able to directly analyze the level of evidence as specified by the Oxford Center for Evidence‐Based Medicine. 23 While we lacked the ability to differentiate nuances in studies, such as nonrandomized cohort studies (level of evidence III) versus observational studies with dramatic evidence (level of evidence II), we grouped publication types in categories that generally correspond to appropriate levels of evidence. An additional limitation is that publishing patterns may have changed in the study timeframe within the journals. For example, the Int Forum Allergy Rhinol and the Laryngoscope had new editors‐in‐chief during the study period. While these changes may affect how individual journals made editorial decisions, we believe the overall trends in our analysis are relatively unaffected.
CONCLUSION
COVID‐19 resulted in dramatic changes in otolaryngology research. There was a significant increase in publications among both COVID and non‐COVID articles but a decrease in the publication of clinical trials. COVID articles drew significantly more focus than non‐COVID articles from researchers and the general public, despite being of lower evidence levels. However, non‐COVID research was also cited more often and received more attention than in the prior year.
ACKNOWLEDGMENTS
We would like to thank the editorial departments of all the journals contributing data to this study.
Editor's Note: This Manuscript was accepted for publication on October 01, 2021.
The authors have no funding, financial relationships, or conflicts of interest to disclose.
References
- 1. Liu N, Chee ML, Niu C, et al. Coronavirus disease 2019 (COVID‐19): an evidence map of medical literature. BMC Med Res Methodol 2020;20:177. 10.1186/s12874-020-01059-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2. Angus DC. Optimizing the trade‐off between learning and doing in a pandemic. JAMA 2020;323:1895–1896. 10.1001/jama.2020.4984. [DOI] [PubMed] [Google Scholar]
- 3. London AJ, Kimmelman J. Against pandemic research exceptionalism. Science 2020;368:476–477. 10.1126/science.abc1731. [DOI] [PubMed] [Google Scholar]
- 4. Älgå A, Eriksson O, Nordberg M. Analysis of scientific publications during the early phase of the COVID‐19 pandemic: topic modeling study. J Med Internet Res 2020;22:e21559. 10.2196/21559. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Yu Y, Li Y, Zhang Z, et al. A bibliometric analysis using VOSviewer of publications on COVID‐19. Ann Transl Med 2020;8:816. 10.21037/atm-20-4235. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Zdravkovic M, Berger‐Estilita J, Zdravkovic B, Berger D. Scientific quality of COVID‐19 and SARS CoV‐2 publications in the highest impact medical journals during the early phase of the pandemic: a case control study. PLoS One 2020;15:e0241826. 10.1371/journal.pone.0241826. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Glasziou PP, Sanders S, Hoffmann T. Waste in covid‐19 research. BMJ 2020;369:m1847. 10.1136/bmj.m1847. [DOI] [PubMed] [Google Scholar]
- 8. Yeo‐Teh NSL, Tang BL. An alarming retraction rate for scientific publications on coronavirus disease 2019 (COVID‐19). Account Res 2021;28:47–53. 10.1080/08989621.2020.1782203. [DOI] [PubMed] [Google Scholar]
- 9. Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED: hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID‐19: a multinational registry analysis. Lancet 2020;395:1820. 10.1016/S0140-6736(20)31180-6. [DOI] [PMC free article] [PubMed] [Google Scholar] [Retracted]
- 10. Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Cardiovascular disease, drug therapy, and mortality in Covid‐19. N Engl J Med 2020;382:e102. 10.1056/NEJMoa2007621. [DOI] [PMC free article] [PubMed] [Google Scholar] [Retracted]
- 11. Siemieniuk RA, Bartoszko JJ, Ge L, et al. Drug treatments for covid‐19: living systematic review and network meta‐analysis. BMJ 2020;370:m2980. 10.1136/bmj.m2980. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Baden LR, El Sahly HM, Essink B, et al. Efficacy and safety of the mRNA‐1273 SARS‐CoV‐2 vaccine. N Engl J Med 2021;384:403–416. 10.1056/NEJMoa2035389. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Polack FP, Thomas SJ, Kitchin N, et al. Safety and efficacy of the BNT162b2 mRNA Covid‐19 vaccine. N Engl J Med 2020;383:2603–2615. 10.1056/NEJMoa2034577. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Sadoff J, Le Gars M, Shukarev G, et al. Interim results of a phase 1‐2a trial of Ad26.COV2.S Covid‐19 vaccine. N Engl J Med 2021;384:1824–1835. 10.1056/NEJMoa2034201. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Trecca E, Marano PG, Gelardi M, et al. Is 2020 the golden year of otolaryngology research? The impact of COVID‐19 on the Italian academic production. Acta Biomed 2021;92:e2021207. 10.23750/abm.v92i2.11557. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Guderian D, Leinung M, Weiss R, Diensthuber M, Stöver T, Loth A. Development of the publication activity at the German university hospitals for otorhinolaryngology, head and neck surgery during the SARS‐CoV‐2‐pandemic in 2020. Laryngorhinootologie 2021;100:364–371. 10.1055/a-1430-7735. [DOI] [PubMed] [Google Scholar]
- 17. Zocchi J, Pietrobon G, Moretto S, et al. Literature in the time of COVID‐19: the “phase two.”. Oral Oncol 2020;109:104837. 10.1016/j.oraloncology.2020.104837. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Chan AKM, Nickson CP, Rudolph JW, Lee A, Joynt GM. Social media for rapid knowledge dissemination: early experience from the COVID‐19 pandemic. Anaesthesia 2020;75:1579–1582. 10.1111/anae.15057. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Venegas‐Vera AV, Colbert GB, Lerma EV. Positive and negative impact of social media in the COVID‐19 era. Rev Cardiovasc Med 2020;21:561–564. 10.31083/j.rcm.2020.04.195. [DOI] [PubMed] [Google Scholar]
- 20. Otolaryngology ‐ Google Scholar Metrics. Available at: https://scholar.google.com/citations?view_op=top_venues&hl=en&vq=med_otolaryngology. Accessed June 6, 2021.
- 21. Sayers E. A General Introduction to the E‐Utilities: Bethesda, MD, USA: National Center for Biotechnology Information (US); 2010. Available at: https://www.ncbi.nlm.nih.gov/books/NBK25497/. Accessed June 5, 2021. [Google Scholar]
- 22. Publication Characteristics (Publication Types) with Scope Notes. Available at: https://www.nlm.nih.gov/mesh/pubtypes.html. Accessed June 6, 2021.
- 23. Howick J, Chalmers I, Glasziou P, et al. The Oxford 2011 Levels of Evidence. Available at: https://www.cebm.ox.ac.uk/resources/levels-of-evidence/ocebm-levels-of-evidence. Accessed June 13, 2021.
- 24. Jain A, Jain P, Aggarwal S. SARS‐CoV‐2 impact on elective orthopaedic surgery: implications for post‐pandemic recovery. JBJS 2020;102:e68. 10.2106/JBJS.20.00602. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Allevi F, Dionisio A, Baciliero U, et al. Impact of COVID‐19 epidemic on maxillofacial surgery in Italy. Br J Oral Maxillofac Surg 2020;58:692–697. 10.1016/j.bjoms.2020.04.035. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. The Lancet Rheumatology . Too long to wait: the impact of COVID‐19 on elective surgery. Lancet Rheumatol 2021;3:e83. 10.1016/S2665-9913(21)00001-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Prasad NK, Englum BR, Turner DJ, et al. A nation‐wide review of elective surgery and COVID‐surge capacity. J Surg Res 2021;267:211–216. 10.1016/j.jss.2021.05.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. Khan A, Mao JZ, Soliman MAR, et al. The effect of COVID‐19 on trainee operative experience at a multihospital academic neurosurgical practice: a first look at case numbers. Surg Neurol Int 2021;12:271. 10.25259/SNI_240_2021. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29. Rocco N, Montagna G, Di Micco R, et al. The impact of the COVID‐19 pandemic on surgical management of breast cancer: global trends and future perspectives. Oncologist 2021;26:e66–e77. 10.1002/onco.13560. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. COVIDSurg Collaborative . Elective surgery cancellations due to the COVID‐19 pandemic: global predictive modelling to inform surgical recovery plans. Br J Surg 2020;107:1440–1449. 10.1002/bjs.11746. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31. Cortegiani A, Catalisano G, Ippolito M, Giarratano A, Absalom AR, Einav S. Retracted papers on SARS‐CoV‐2 and COVID‐19. Br J Anaesth 2021;126:e155–e156. 10.1016/j.bja.2021.01.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Becker RC, Bode C. Medical publishing during the COVID‐19 pandemic: then and now. J Thromb Thrombolysis 2021;51:1101–1106. 10.1007/s11239-021-02422-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Anker SD, Butler J, Khan MS, et al. Conducting clinical trials in heart failure during (and after) the COVID‐19 pandemic: an Expert Consensus Position Paper from the Heart Failure Association (HFA) of the European Society of Cardiology (ESC). Eur Heart J 2020;41:2109–2117. 10.1093/eurheartj/ehaa461. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Jeon J, Kim H, Yu KS. The impact of COVID‐19 on the conduct of clinical trials for medical products in Korea. J Korean Med Sci 2020;35:e329. 10.3346/jkms.2020.35.e329. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Saag MS. Misguided use of hydroxychloroquine for COVID‐19: the infusion of politics into science. JAMA 2020;324:2161–2162. 10.1001/jama.2020.22389. [DOI] [PubMed] [Google Scholar]
- 36. Araujo AC, Vanin AA, Nascimento DP, Gonzalez GZ, Costa LOP. What are the variables associated with Altmetric scores? Syst Rev 2021;10:193. 10.1186/s13643-021-01735-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Chi AJ, Lopes AJ, Rong LQ, Charlson ME, Alvarez RD, Boerner T. Examining the correlation between Altmetric Attention Score and citation count in the gynecologic oncology literature: does it have an impact? Gynecol Oncol Rep 2021;37:100778. 10.1016/j.gore.2021.100778. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38. de Oliveira SD, Taborda B, Pazzinatto MF, Ardern C, Barton CJ. Altmetric score has a stronger relationship with article citations than journal impact factor and open access status: a cross‐sectional analysis of 4,022 sports science articles. J Orthop Sports Phys Ther 2021;1–19. 10.2519/jospt.2021.10598. [DOI] [PubMed] [Google Scholar]
- 39. Yoshimura R, Grant MC, Gardiner MD, Wade RG. Disseminating hand surgery research using social media: the relationship between altmetrics and citations. J Hand Surg 2021;S0363‐5023:00206–9. 10.1016/j.jhsa.2021.03.028. [DOI] [PubMed] [Google Scholar]