Abstract
Background:
Whether research engagement with social media and other public platforms results in increased citations in obstetrics and gynecology (OB/GYN) remains uncertain. The Altmetric Attention Score (AAS) is a metric of research influence that is based on mentions in social media and public platforms, like newsfeeds and Wikipedia. The correlation between AAS, absolute citation rates and the Relative Citation Ratio (RCR), a novel metric of research engagement that is also based on citation rates, for research in OB/GYN is uncertain.
Objective:
To evaluate the correlation between AAS, absolute citation rate, and RCR for articles published in OB/GYN journals from 2004–2019. Our second objective was to identify, characterize, and compare the 100 articles with highest AAS, the 100 top-cited articles, and the 100 articles with highest RCR.
Study Design:
We performed a cross-sectional altmetric and bibliometric study of all OB/GYN articles indexed in the National Institutes of Health (NIH) Open Citation Collection (OCC) from 2004–2019. Articles were included if they were published in OB/GYN journals according to InCites Journal Citation Reports indexing. Citations data, including citation numbers and RCRs, were downloaded on 20 March 2021 and merged with altmetric data from the Altmetric Explorer based on each article’s unique PubMed identification number. We assessed correlation between AASs and number of citations as well as AASs and RCRs by calculating Pearson correlation coefficient. The 100 articles with highest AAS, the 100 top-cited articles, and the 100 articles with highest RCRs were characterized and compared using means (standard deviations; SD) and mean differences (95% confidence intervals; CI).
Results:
There were 156,592 articles published in 82 OB/GYN journals and indexed in the NIH OCC between 2004–2019. The correlation coefficient was 0.18 (95% CI 0.17−0.19) for AASs versus number of citations and 0.10 (95% CI 0.09–0.11) for AASs versus RCRs. There was no overlap among the 100 articles on the highest AAS list and the 100 top-cited list and minimal overlap among the 100 articles on the highest AAS list and the 100 highest RCR list (98 unique articles on each list). Articles with highest AASs generated substantially more engagement with social media and other public platforms compared to top-cited articles (mean [SD] AAS 763.1 (520.8) versus 49.9 [SD 81.6]; mean difference −713.2 [95% CI −819.9 − −606.6]) and highest-RCR articles (mean 116.2 [SD 415.9]; mean difference −661.5 [95% CI −746.2−−576.9]). In contrast, the articles with highest AASs generated much fewer citations compared to top-cited articles (mean [SD] 39.7 [SD 47.6] versus 541.8 [SD 312.8]; mean difference 502.0 [95% CI 439.0–565.0]) and highest RCR articles (mean 458.9 [SD 363.5]; mean difference 427.7 [95% CI 353.8–501.6]). Nearly half of articles with highest AASs were basic/translational studies and prominently featured articles about menopause and environmental factors that impact fertility whereas top-cited articles and highest RCR were more likely to be reviews and consensus statements, respectively, and featured articles about the placentation and poly cystic ovarian disease, respectively. Articles with highest AASs were more likely to be published open access.
Conclusion:
There appears to be weak short-term correlation between AASs and citation rates. Further study is warranted to ascertain whether there may be long-term correlation between alternative metrics and citation rates in OB/GYN.
Key Words/phrases: Altmetric attention score, bibliometrics, citation rates, relative citation ratio, obstetrics and gynecology, journals
Condensation:
In this cross-sectional study of OB/GYN journals (2004–2019), Altmetric Attention Scores, a marker of research’s engagement with social media and public platforms, had weak short-term correlation with citation rates.
Introduction
Bibliometrics refers to the application of statistical methods to assess the impact of research articles and other publications (1). Bibliometric analysis based on absolute citation rates can also be used to evaluate performance of academic journals and individual researchers. Several citation-based metrics have generated research interest, including absolute citation number and the Relative Citation Ratio (RCR) (2). Absolute citation number favors older articles and has limited utility when comparing articles from different fields or different subspecialities within the same field (3). The RCR is another citation-based metric, which is defined as the total number of citations that an article receives per year divided by the average field-specific citations rate for a peer comparison group of National Institutes of Health (NIH)-funded articles (4). The RCR allows comparison of articles in different fields (2), which is an advantage over absolute citation number, but it is still based on citation rates and its impact can only be assessed retrospectively (5).
Due to limitations of conventional bibliometric analysis based on citation rates, there is interest in applying alternative metrics to gauge the influence of research. One such tool is the Altmetric Attention Score (AAS), which is a composite score of research influence that is based on research mentions on research blogs, news media coverage, bookmarks on reference managers like Mendeley, citations on Wikipedia, and engagement on social media platforms such as Twitter and Facebook (6, 7).
Prior bibliometric studies in OB/GYN have focused on citation rates (8, 9) and the RCR (10), but it remains unclear whether greater research engagement with social media and public platforms results in higher citation rates. Therefore, we performed a cross-sectional altmetric and bibliometric analysis of published articles in all OB/GYN journals from 2004–2019. The objectives were to evaluate the correlation between AASs and two citation-based metrics, namely absolute citation number and the RCR. The second objective was to identify, characterize, and compare the 100 articles with highest AAS, the 100 top-cited articles, and the 100 articles with highest RCR. We hypothesized that there would be correlation between altmetrics and bibliometrics and that articles that generated substantial interest in social media and public platforms would share similarities with articles that generated frequent citations.
Methods
We performed a cross-sectional altmetric and bibliometric study of all OB/GYN articles that were published from 2004–2019. Given that social media companies such as Twitter and Facebook were founded in the early 2000s, we selected a period that would ensure that altmetric and bibliometric data would be available for most articles. Articles were included if they were published in “obstetrics and gynecology” journals according to Incites Journal Citations Reports (JCR) and indexed in the NIH Open Citation Collection (OCC) and available for free download from the iCite website (11). The NIH OCC is a scientometric database that is derived from multiple sources including PubMed Central, Medline, Entrez, and CrossRef. The database also includes open access data that is obtained through a machine learning pipeline (12). Since this is a publicly available database that contains no protected health information, no Institutional Review Board approval was sought. The study followed the Strengthening the Reporting of Observational Studies in Epidemiology reporting guideline for cross-sectional studies.
Data sources
InCites JCR was used to identify all active OB/GYN journals. The iCite database was queried to identify all articles published in each OB/GYN journal between 2004–2019 and downloaded into a Microsoft Excel file. These articles were then merged by their unique PubMed identification numbers with altmetrics from the Altmetric Explorer.
Bibliometrics, altmetrics, and article characteristics
Citation numbers and RCR data were downloaded from the iCite website, and altmetric data were downloaded from Altmetric Explorer on 20 March 2021. For Altmetric data, we recorded the AAS and the number of mentions in Twitter, newfeeds, blog posts, policy mentions, patent mentions, peer mentions, Facebook, Wikipedia, Reddit, and Mendeley for each article. For bibliometric data, we recorded the number of citations, the number of citations per year, and the RCR for each article.
To provide insight into the characteristics of OB/GYN research that led to substantial engagement in social media and other public platforms as well as citation rates, the 100 articles with highest AASs, the 100 top-cited articles, and the 100 articles with highest RCR were selected for further review. These articles were characterized based on prior bibliometric studies in OB/GYN (8, 9) by study design, topic, open access, authorship, and country of origin. Study design included observational, basic, and translational science, randomized clinical trials (RCT), reviews, consensus statements, new procedure, and other.
Statistical analysis
We assessed the correlation between AAS and citation rates, as well the AAS and RCR, based on the Pearson’s correlation coefficients (with 95% confidence intervals [CI]). We performed descriptive analyses to characterize the 100 articles that had highest AAS, top-cited, and highest RCR. In addition, after excluding articles that featured on both lists, we compared characteristics of articles with the highest AAS with the top-cited articles, and we compared articles with highest AAS with the articles with the highest RCR. We calculated mean differences (MD) with 95% CI. All analyses were performed with Stata version 10.1 (StataCorp LP, College Station, TX).
Results
All data were downloaded on 20 March 2021. There were 156,592 articles published in n=82 OB/GYN journals from 2004–2019. Citation data were available for 148,012 (94.5%) articles, of which 16,539 (11.2%) articles that were cited zero times, 128,991 (87.1%) articles were cited 1–99 times, and 2,482 (1.7%) articles were cited 100 times or more (Figure 1). RCR data were available for 156,223 (99.8%). There were 17,444 (11.2%) articles with RCR score of zero, 138,778 (88.8%) articles with RCR score between 0.1 to 99.9, and 1 (0.0%) article with RCR score 100 or higher. AAS was available for less than half (77,207 [49.3%] of articles with citation data, indicating that nearly half of published articles during the study period had no public footprint. Of these articles, there were 18,331 (23.7%) articles with AASs of zero, 58,128 (75.3%) articles with AASs between 1–99, and 748 (1.0%) articles had AASs more than 100. The correlation coefficient was 0.18 (95% CI 0.17–0.19) for AASs versus number of citations and 0.10 (95% CI 0.09–0.11) for AASs versus RCRs.
Figure 1.
Study flow diagram
RCR: Relative Citation Ratio: AAS: Altmetric Attention Score
Characteristics of the 100 articles with highest AAS, the 100 top-cited articles, and the 100 articles with highest RCR are summarized and compared in Table 1. The full list of the articles can be found in Supplemental Table 1. There was no overlap among the 100 articles on the highest AAS list and the 100 top-cited list, and there was minimal overlap among the 100 articles on the highest AAS list and the 100 highest RCR list (98 unique articles on each list). Compared to top-cited articles and articles with highest RCRs, articles with highest AASs were more recently published and more likely to be open access and have non-US authorship. Articles with highest AASs generated substantially more engagement with social media and other public platforms compared to top-cited articles and highest-RCR articles. The articles with highest AASs articles generated much fewer citations compared to top-cited articles (mean 39.7 [SD 47.6] versus 541.8 [SD 312.8]; mean difference 502.0 [95% CI 439.0–565.0]) and highest RCR articles (mean 458.9 [SD 363.5]; mean difference 427.7 [95% CI 353.8–501.6]).
Table 1.
Characteristics of 100 articles with highest Altmetric Attention Scores, 100 top-cited articles, and 100 with highest Relative Citation Ratios in Obstetrics and Gynecology Journals, 2004–2019
| Top AAS N=100 | Top Cited N=100* | Top RCR N=100** | Top AAS vs Top cited*** Mean Difference (95% CI) | Top AAS vs Top RCR**** Mean Difference (95% CI) | |
|---|---|---|---|---|---|
| Year | 2015 (3) | 2008 (3) | 2012 (4) | −7.1 (−7.9– −6.3) | −3.4 (−4.4– −2.4) |
| Citations | 39.7 (47.6) | 541.8 (312.8) | 458.9 (363.5) | 502.0 (439.0–565.0) | 427.7 (353.8–501.6) |
| Citations per year | 7.2 (8.0) | 44.5 (27.4) | 49.5 (25.6) | 37.4 (31.7–43.0) | 43.2 (37.9–48.5) |
| RCR | 4.2 (4.7) | 23.9 (14.3) | 28.4 (12.6) | 19.6 (16.7–22.6) | 24.7 (22.1–27.3) |
| APT | 0.7 (0.3) | 0.9 (0.1) | 1.0 (0) | 0.3 (0.2–0.3) | 0.3 (0.2–0.3) |
| AAS | 763.1 (520.8) | 49.9 (81.6) | 116.2 (415.9) | −713.2 (−819.9– −606.6) | −661.5 (−746.2– −576.9) |
| 316.2 (1278.5) | 5.8 (17.0) | 27.5 (79.4) | −310.3 (−569.1– −51.6) | −294.7 (−556.1– −33.4) | |
| News | 92.3 (71.6) | 4.3 (9.0) | 12.8 (60.6) | −88.0 (−102.6– −73.4) | −81.2 (−91.8– −70.6) |
| Blog | 3.8 (5.8) | 1.2 (2.4) | 1.6 (4.8) | −2.6 (−3.8– −1.3) | −2.2 (−3.2– −1.2) |
| Policy | 0.4 (1.2) | 2.2 (3.1) | 2.5 (5.5) | 1.7 (1.1–2.4) | 2.1 (0.9–3.2) |
| Patent | 0.1 (0.3) | 3.2 (11.5) | 2.2 (11.0) | 3.2 (0.9–5.4) | 2.2 (0–4.4) |
| Peer | 0 (0.1) | 0.1 (0.5) | 0.1 (0.5) | 0 (−0.1–0.1) | 0 (0–0.1) |
| 16.6 (80.9) | 0.7 (3.0) | 3.6 (9.3) | −15.9 (−32.3–0.6) | −13.2 (−29.6– 3.2) | |
| Wiki | 0.4 (1.2) | 0.6 (1.2) | 0.6 (1.2) | 0.2 (−0.2–0.5) | 0.2 (−0.1–0.5) |
| 8.3 (48.8) | 0.1 (0.4) | 3.6 (9.4) | −8.1 (−18.0–1.7) | −8.1 (−18.1–1.9) | |
| 0 | 0 (0.1) | 0 (0.1) | 0 (0–0) | 0 (0–0) | |
| 0.5 (1.4) | 0 (0.2) | 0.1 (0.5) | −0.4 (−0.7– −0.2) | −0.4 (−0.7– −0.1) | |
| Mendeley | 101.3 (94.3) | 447.3 (274.7) | 449.3 (274.4) | 346.0 (288.6–403.4) | 355.2 (297.7–412.7) |
| Open access (%) | 82 | 54 | 61 | – | – |
| United States authorship (%) | 33 | 54 | 56 | – | – |
Altmetrics data available for n=95
Altmetrics data available for n=97
includes n=100 unique articles on top-cited and highest AAS lists
includes n=98 unique articles on highest RCR and highest AAS lists
Data is mean (SD) or n.
Nearly half of articles with highest AASs were basic/translational studies whereas and top-cited articles and highest RCR articles were more likely to be reviews and consensus statements, respectively (Table 2). Only 15 of these influential articles were RCTs, and 7 (46.7%) of them were in the top-cited list. The institutions that published the most articles are described in Table 3. The American College of Obstetrics and Gynecology, NIH, University of California at San Francisco, and North American Menopause Society all contributed 5 or more articles. The distribution of journals that published the most articles are described in Figure 2.
Table 2.
Study design of top-100 articles with highest Altmetric Attention Scores, citation rates, and relative citation ratios in Obstetrics and Gynecology Journals, 2004–2019
| Study Design | Top AAS N=100 | Top Cited N=100 | Top RCR N=100 |
|---|---|---|---|
| Observational | 10 | 12 | 11 |
| Basic/translational science | 49 | 14 | 7 |
| Review | 12 | 29 | 19 |
| Consensus statement | 3 | 20 | 38 |
| Metanalysis/system review | 5 | 15 | 20 |
| Randomized clinical trial | 5 | 7 | 3 |
| New procedure/assay | 3 | 1 | 1 |
| Other | 13 | 1 | 1 |
Table 3.
Institutions of origin of top-100 articles with highest Altmetric Attention Scores, citation rates, and relative citation ratios in Obstetrics and Gynecology Journals, 2004–2019 (listed if ≥5 articles)
| Institution | Country of Origin | Number of Publications |
|---|---|---|
| American College of Obstetrics and Gynecology | USA | 13 |
| National Institute of Child Health | USA | 6 |
| University of California San Francisco | USA | 6 |
| North American Menopause Society | USA | 5 |
Figure 2.
Journals in which the top-100 obstetrics and gynecology articles with highest Altmetric Attention Scores and highest citations were published, 2004–2018 (if ≥2 articles were published in that journal)
Topics that featured prominently in the top-AAS list include menopause (n=9), environmental factors that impact fertility (n=8), and abortion (n=6) whereas the placenta (n=7) and obesity in pregnancy (n=6) featured prominently in the top-cited list and polycystic ovarian disease (n=8) and maternal morbidity and mortality (n=7) featured prominently in the top-RCR list.
Comment
Principal Findings
In this cross-sectional altmetric and bibliometric study of articles published in OB/GYN journals between 2004–2019, we found weak short-term correlation between AASs, a marker of research’s engagement with social media and public platforms, and absolute citation number. The AASs also had weak correlation with the RCR, which is a novel metric of research influence that is based on citation rates, but has advantages over conventional citation analysis. This lack of correlation was further evident in an analysis that compared the 100 articles with highest AAS compared to the 100 top-cited articles and the 100 articles with highest RCRs. There was minimal overlap between these lists of influential articles, and the article characteristics were quite different. Compared to articles that received more citations (based on absolute number and RCRs), articles with highest AASs were more likely to be published recently, to be available as open access, to have non-US authorship, and to be basic science and translational research.
Result in Context
More than 70% of adults in the US use social media (13), and high impact journals use these platforms to share their research. The rationale for these efforts is based on limited evidence. For example, a 2011 study of tweets containing links to articles in the Journal of Medical Internet Research (July 2008 to November 2011) found moderate correlation between “tweetations” and citations (correlation coefficients ranged from 0.4 to 0.7 for the log-transformed Google Scholar citations), but the authors found that articles that were highly tweeted were 11 times more likely to be highly cited articles compared to articles that were not highly tweeted (rate ratio 10.8, 95% CI 3.4–33.6) (14). The results of two systematic reviews found some evidence that social media engagement may increase the number of citations that articles received, but the evidence was deemed “inconclusive” by one and its author suggested that further high quality studies are needed to quantify the impact of engagement with social media and public platforms on citation rates (15, 16).
As a marker of engagement with social media and public platforms, the AAS employs machine learning algorithms to evaluate engagement on multiple platforms. In the study, we found weak correlation between AAS and two citation-based metrics. Although this study is the first to focus on OB/GYN research, others have suggested poor correlation between AAS and citation rates. For example, in an analysis that included all pediatric articles published in 2014 in two general pediatric journals (Pediatrics and JAMA Pediatrics) and two general medicine journals (JAMA and the New England Journal of Medicine), the authors found a “modest” correlation between article citations and AASs (spearman correlation 0.53 (95% CI 0.49–0.57) (17). Weak or no correlation was seen in influential articles in dental research (18), medical education (19), anesthesia (20), and radiology (21), among other fields. The weak correlation could mean that altmetrics reflect other dimensions of impact that is independent of citations (22). There is evidence that articles shared on Twitter have higher download and view rates (23, 24), which reflects “uptake” of new research (14, 25). Given that the AAS is a relatively new metric, the lack of correlation may reflect a short-term relationship. Long-term correlation between AASs and citation-based metrics in OB/GYN remains uncertain.
Articles that were highly cited had substantial engagement with Mendeley. This is consistent with prior studies (26, 27). A meta-analysis evaluating correlation between altmetrics and citations rates found no correlation with micro-blogging (twitter), small correlation with blog counts, and moderate to large correlation with bookmark counts in Mendeley and CiteULike (26). Another study reported a positive correlation between traditional citations and Mendeley in all fields of academia, although the magnitude of correlation varied considerably across fields. The authors concluded that Mendeley reader counts could be used as an evidence of early citation impact in most fields (27).
The study identified numerical differences between the study types of articles with highest AASs and frequently cited articles. Approximately half of articles with highest AASs were basic/translational studies whereas top-cited and highest RCR articles were more likely to be reviews and consensus statements. To our knowledge, other studies have not characterized study types of OB/GYN articles that have substantial engagement in social media and public platforms. However, prior studies have evaluated study types of OB/GYN articles that are frequently cited. A prior study that evaluated top-RCR and top-cited articles in OB/GYN in the NIH OCC (1980–2019) suggested that these articles were frequently reviews and consensus statements, but the most common study type was observational. In another study of OB/GYN articles that included specialty OB/GYN and non-specialty journals in the Web of Science’s Science Citation Index Expanded (1980–2018), the most common study type among articles from specialty journals was observational as well. It is possible that the short time period of our study explains the difference in results from our study compared to these previously published studies
Further, compared to top-cited articles and articles with highest RCRs, articles with highest AASs were more likely to be open access and have non-US authorship. Reasons for these findings are not well elucidated. It is possible that open access articles are more frequently shared on social media and other public platforms. The machine learning algorithm employed by Altmetrics may preferentially identify open accesses articles as well, but this seems less likely. Only one-third of top-AAS articles were published by US authors. This finding warrants further study, especially since the main social media and public platforms that are incorporated into the AAS are based in the US.
Research Implications
Despite the study findings of weak correlation between AASs and citation rates, many academic journals (and most high impact journals) promote their research on social media and other public platforms. An analysis of more than 3000 journals that were indexed in Scopus demonstrated that more than 30% were linked to various social media sites (28). These journals use multiple strategies to promote their work on social media such as internet-based journal clubs (29), podcasts and other article promotion (30–32), and sharing infographics and videos abstracts (33–35). There may be several reasons why academic journals have embraced social media. There is a general sense that social media engagement has tangible, positive impact of citation rates (36, 37). Journals may also like how social media rapidly disseminates knowledge (faster than print). Research’s social media engagement provides insight into public engagement that is broader than citations (26), reflecting how it immediately reaches readers beyond (or perhaps in addition to) the usual research community (25) and serves to illustrate the potential social implications of the research (38).
Academic journals and research institutions continue to rely on conventional bibliometrics. Impact factors are the key metric for comparing the influence of academic journals, and the H-score is a key component of professional promotions applications. As many academic journals and researchers are promoting their findings on these public platforms, additional study and discussion regarding the utility of these efforts and the goals of research promotion in public spheres is warranted.
Strength and Limitations
Our study has several strengths. First, we evaluated correlation between alternative metrics and bibliometrics based on every article published in OB/GYN journals during the study period. Second, we compared AASs to two citation-based metrics, including citation rates and the RCR, and still found no correlation, which strengthens our findings. Third, to the best of our knowledge, this is the first study to undertake this type of analysis in OB/GYN.
Despite these strengths, our study has some limitations. Since the AAS is a newer metric of research impact and potential reach, we limited the study period to 15 years. This was necessary so that we could ensure that we had altmetric and bibliometric data for most articles. Even with this limited study period, less than 50% of all articles had an AAS and thus had no public footprint at all. As we acknowledged above, it is possible that we could not recognize a correlation with citations that requires more time to see. We also limited our analysis to articles published in OB/GYN journals. As a result, we excluded OB/GYN articles that were published in general medicine and surgery journals. Although a potential weakness, the study reflects correlation among articles published in OB/GYN specialty journals, and it is likely that these results are generalizable to other journal articles from specialty journals in other fields, although the results may not be generalizable to highest impact journals that are geared towards larger audiences. Finally, we must acknowledge that any discussion about metrics of research influence does not reflect quality or impact of health outcomes, which is a major limitation of these metrics and calls for improved metrics that reflect quality as well as engagement in social media and public platforms (as reflected by altmetrics) and impact as reflected by citation rates.
Conclusions
In this cross-sectional altmetric and bibliometric study of articles published in OB/GYN journals between 2004–2019, we found weak short-term correlation between AASs, a marker of research’s engagement with social media and public platforms, and two citation-based metrics. We also found that there were substantial differences between articles that had more engagement in social media and public platforms and those articles that received more citations. Although many academic journals and research institutions promote their findings on these public platforms, the results of the study question the utility of these efforts. Further study is warranted to ascertain whether there may be long term correlation between altmetrics and citation rates.
Supplementary Material
AJOG at a glance.
A. Why was this study conducted?
The study was conducted to evaluate whether there is a correlation between obstetrics and gynecology (OB/GYN) research’s engagement with social media and other public platforms and two citation-based metrics, including absolute citation number and the Relative Citation Ratio (RCR). We also wanted to characterize the 100 articles with highest Altmetric Attention Score (AAS), citation rates, and RCRs to gain insight into the characteristics that lead to greater research influence according to these metrics.
B. What are the key findings?
There was weak short-term correlation between AASs and citation-based metrics. Compared to top-cited articles and articles with highest RCRs, articles with highest AASs were more recently published and more likely to be open access and have non-US authorship. Nearly half of articles with highest AASs were basic/translational studies and prominently featured articles about menopause and environmental factors that impact fertility whereas top-cited articles and highest RCR were more likely to be reviews and consensus statements, respectively, and featured articles about the placentation and poly cystic ovarian disease, respectively.
C. What does this study add to what is already known?
The study quantifies the degree of correlation between AASs and two citation-based metrics in OB/GYN, including absolute citation number and the RCR. While the results of this study suggest there is weak short-term correlation between OB/GYN’s research engagement with social media and public platforms and two citation-based metrics, further study is warranted to ascertain whether there could be a stronger long-term correlation.
Acknowledgements
Altmetric provided no-cost access to their data for this project, but were not involved in the study design, analysis, or drafting the manuscript nor did they review or approve any version of the manuscript.
Funding
Drs. Cande Ananth is supported, in part, by the National Heart, Lung, and Blood Institute (grant R01-HL150065) and the National Institute of Environmental Health Sciences (grant R01-ES033190), National Institutes of Health.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Disclosures: The authors report no actual or potential conflicts of interest.
References
- 1.Franchignoni F, Ozcakar L, Negrini S. Basic bibliometrics for dummies and others: an overview of some journal-level indicators in physical and rehabilitation medicine. Eur J Phys Rehabil Med. 2018;54(5):792–6. [DOI] [PubMed] [Google Scholar]
- 2.Surkis A, Spore S. The relative citation ratio: what is it and why should medical librarians care? J Med Libr Assoc. 2018;106(4):508–13. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hutchins BI, Yuan X, Anderson JM, Santangelo GM. Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level. PLoS Biol. 2016;14(9):e1002541. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Reddy V, Gupta A, White MD, Gupta R, Agarwal P, Prabhu AV, et al. Assessment of the NIH-supported relative citation ratio as a measure of research productivity among 1687 academic neurological surgeons. J Neurosurg. 2020:1–8. [DOI] [PubMed] [Google Scholar]
- 5.Sathianathen NJ, Lane Iii R, Murphy DG, Loeb S, Bakker C, Lamb AD, et al. Social Media Coverage of Scientific Articles Immediately After Publication Predicts Subsequent Citations - #SoME_Impact Score: Observational Analysis. J Med Internet Res. 2020;22(4):e12288. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Robinson DBT, Powell A, Waterman J, Hopkins L, James OP, Egan RJ, et al. Predictive value of Altmetric score on citation rates and bibliometric impact. BJS Open. 2021;5(1). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Goksoy B, Bozkurt H. Social attention of the top 50 scientific articles on gastric cancer: Bibliometric and altmetric analysis. J BUON. 2020;25(5):2322–31. [PubMed] [Google Scholar]
- 8.Brandt JS, Hadaya O, Schuster M, Rosen T, Sauer MV, Ananth CV. A Bibliometric Analysis of Top-Cited Journal Articles in Obstetrics and Gynecology. JAMA Netw Open. 2019;2(12):e1918007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Brandt JS, Downing AC, Howard DL, Kofinas JD, Chasen ST. Citation classics in obstetrics and gynecology: the 100 most frequently cited journal articles in the last 50 years. Am J Obstet Gynecol. 2010;203(4):355.e1–7. [DOI] [PubMed] [Google Scholar]
- 10.Mitra AN, Aurora N, Grover S, Ananth CV, Brandt JS. A bibliometric analysis of obstetrics and gynecology articles with highest relative citation ratios, 1980 to 2019. Am J Obstet Gynecol MFM. 2021;3(1):100293. [DOI] [PubMed] [Google Scholar]
- 11.National Institute of Health: Office of Portfolio Analysis [updated Decmeber 4th, 2019. Available from: https://icite.od.nih.gov/.
- 12.Hutchins BI, Baker KL, Davis MT, Diwersy MA, Haque E, Harriman RM, et al. The NIH Open Citation Collection: A public access, broad coverage resource. PLoS biology. 2019;17(10):e3000385. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Share of U.S. adults using social media, including Facebook, is mostly unchanged since 2018 [Internet]. 2019. Available from: https://www.pewresearch.org/fact-tank/2019/04/10/share-of-u-s-adults-using-social-media-including-facebook-is-mostly-unchanged-since-2018/.
- 14.Eysenbach G Can tweets predict citations? Metrics of social impact based on Twitter and correlation with traditional metrics of scientific impact. J Med Internet Res. 2011;13(4):e123. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Bardus M, El Rassi R, Chahrour M, Akl EW, Raslan AS, Meho LI, et al. The Use of Social Media to Increase the Impact of Health Research: Systematic Review. J Med Internet Res. 2020;22(7):e15607. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Ruano J, Aguilar-Luque M, Gomez-Garcia F, Alcalde Mellado P, Gay-Mimbrera J, Carmona-Fernandez PJ, et al. The differential impact of scientific quality, bibliometric factors, and social media activity on the influence of systematic reviews and meta-analyses about psoriasis. PLoS One. 2018;13(1):e0191124. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Giustini AJ, Axelrod DM, Lucas BP, Schroeder AR. Association Between Citations, Altmetrics, and Article Views in Pediatric Research. JAMA Netw Open. 2020;3(7):e2010784. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Delli K, Livas C, Spijkervet FKL, Vissink A. Measuring the social impact of dental research: An insight into the most influential articles on the Web. Oral Dis. 2017;23(8):1155–61. [DOI] [PubMed] [Google Scholar]
- 19.Amath A, Ambacher K, Leddy JJ, Wood TJ, Ramnanan CJ. Comparing alternative and traditional dissemination metrics in medical education. Med Educ. 2017;51(9):935–41. [DOI] [PubMed] [Google Scholar]
- 20.Rong LQ, Lopes AJ, Hameed I, Gaudino M, Charlson ME. Examining the correlation between Altmetric score and citation count in the anaesthesiology literature. Br J Anaesth. 2020;125(2):e223–e6. [DOI] [PubMed] [Google Scholar]
- 21.Rosenkrantz AB, Ayoola A, Singh K, Duszak R Jr. Alternative Metrics (“Altmetrics”) for Assessing Article Impact in Popular General Radiology Journals. Acad Radiol. 2017;24(7):891–7. [DOI] [PubMed] [Google Scholar]
- 22.Dinsmore A, Allen L, Dolby K. Alternative perspectives on impact: the potential of ALMs and altmetrics to inform funders about research impact. PLoS Biol. 2014;12(11):e1002003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Shuai X, Pepe A, Bollen J. How the scientific community reacts to newly submitted preprints: article downloads, Twitter mentions, and citations. PLoS One. 2012;7(11):e47523. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Allen HG, Stanton TR, Di Pietro F, Moseley GL. Social media release increases dissemination of original articles in the clinical pain sciences. PLoS One. 2013;8(7):e68914. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Klar S, Krupnikov Y, Ryan JB, Searles K, Shmargad Y. Using social media to promote academic research: Identifying the benefits of twitter for sharing academic work. PLoS One. 2020;15(4):e0229446. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Bornemann L Alternative metrics in scientometrics: a meta-analysis of research into three altmetrics. Scientometrics. 2015;103:1123–44. [Google Scholar]
- 27.Thelwall M Are Mendeley Reader Counts Useful Impact Indicators in all Fields. Scientometrics. 2017. [Google Scholar]
- 28.Valerio-Ureña G, Herrera-Murillo D. Online social networks as a communication channel for Open Access journals. Revista Latina de Comunicación Social 2017;72:1341–50. [Google Scholar]
- 29.Hawkins CM, Hunter M, Kolenic GE, Carlos RC. Social Media and Peer-Reviewed Medical Journal Readership: A Randomized Prospective Controlled Trial. J Am Coll Radiol. 2017;14(5):596–602. [DOI] [PubMed] [Google Scholar]
- 30.Fox CS, Bonaca MA, Ryan JJ, Massaro JM, Barry K, Loscalzo J. A randomized trial of social media from Circulation. Circulation. 2015;131(1):28–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Fox CS, Gurary EB, Ryan J, Bonaca M, Barry K, Loscalzo J, et al. Randomized Controlled Trial of Social Media: Effect of Increased Intensity of the Intervention. J Am Heart Assoc. 2016;5(5). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Thoma B, Murray H, Huang SYM, Milne WK, Martin LJ, Bond CM, et al. The impact of social media promotion with infographics and podcasts on research dissemination and readership. CJEM. 2018;20(2):300–6. [DOI] [PubMed] [Google Scholar]
- 33.Erskine N, Hendricks S. The Use of Twitter by Medical Journals: Systematic Review of the Literature. J Med Internet Res. 2021;23(7):e26378. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34.Wadhwa V, Latimer E, Chatterjee K, McCarty J, Fitzgerald RT. Maximizing the Tweet Engagement Rate in Academia: Analysis of the AJNR Twitter Feed. AJNR Am J Neuroradiol. 2017;38(10):1866–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Chapman SJ, Grossman RC, FitzPatrick MEB, Brady RRW. Randomized controlled trial of plain English and visual abstracts for disseminating surgical research via social media. Br J Surg. 2019:1611–6. [DOI] [PubMed] [Google Scholar]
- 36.Thelwall M, Haustein S, Lariviere V, Sugimoto CR. Do altmetrics work? Twitter and ten other social web services. PLoS One. 2013;8(5):e64841. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Fox CS, Barry K, Colbert J. Importance of Social Media Alongside Traditional Medical Publications. Circulation. 2016;133. [Google Scholar]
- 38.Patel RB, Vaduganathan M, Bhatt DL, Bonow RO. Characterizing High-Performing Articles by Altmetric Score in Major Cardiovascular Journals. JAMA Cardiol. 2018;3(12):1249–51. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.