Skip to main content
NCBI home page
F1000Research logoLink to F1000Research
. 2019 Mar 7;8:266. [Version 1] doi: 10.12688/f1000research.17979.1

The impact of the open-access status on journal indices: a review of medical journals

Saif Aldeen AlRyalat 1,a, Mohammad Saleh 2, Mohammad Alaqraa 2, Alaa Alfukaha 2, Yara Alkayed 2, Maryann Abaza 2, Hadeel Abu Saa 2, Mohamed Alshamiry 2
PMCID: PMC6449789  PMID: 31001420

Abstract

Background: Over the past few decades, there has been an increase in the number of open access (OA) journals in almost all disciplines. This increase in OA journals was accompanied an increase in funding to support such movements. Medical fields are among the highest funded fields, which further promoted its journals to move toward OA publishing. Here, we aim to compare OA and non-OA journals in terms of citation metrics and other indices.

Methods: We collected data on the included journals from Scopus Source List on 1 st November 2018.  We filtered the list for medical journals only. For each journal, we extracted data regarding citation metrics, scholarly output, and wither the journal is OA or non-OA.

Results: On the 2017 Scopus list of journals, there was 5835 medical journals. Upon analyzing the difference between medical OA and non-OA journals, we found that OA journals had a significantly higher CiteScore (p< 0.001), percent cited (p< 0.001), and source normalized impact per paper (SNIP) (p< 0.001), whereas non-OA journals had higher scholarly output (p< 0.001). Among the five largest journal publishers, Springer Nature published the highest frequency of OA articles (31.5%), while Wiley-Blackwell had the lowest frequency among its medical journals (4.4%).

Conclusion: Among medical journals, although non-OA journals still have higher output in terms of articles per year, OA journals have higher citation metrics.

Keywords: Open access, Journal, Medicine, Bibliometrics, Citation

Introduction

Open access (OA) journals allow free (access to/availability of) academic articles, they enable any user to read, search, download, share, use them for indexing, print the full texts, or utilize them as data for software without being charged 1. Over the past 20 years, there has been an increase in the number of OA medical journals. According to Web of Science, published OA articles as a proportion of total publications increased from 9.5% to 24% from 1998 to 2018. These OA journals provide an easily accessed source of information, a source that is accessible even for developing and low income countries 2.

Bibliometric analysis are methods or applications used to measure the influence of authors or scientific papers, of which, citation analysis is the most commonly used methods 3. Now several citation databases have become available, with the three largest being Web of Science, Scopus and PubMed. These databases record the number of times that a journal article has been cited by other papers 4. The use of bibliometric analysis is becoming more popular to assess the performance of different aspects of the scholarly and scientific fields. Analysis can be at the level of the researchers themselves, journals, departments, universities, national organizations, and even entire nations 5– 8. There are several databases that can be used to perform the bibliometric analysis, with each database having its own characteristics; these include Google Scholar, Pubmed (Only biomedical citations), Scopus, and Web of Science 4. According to the number, coverage, and quality of citations covered by the databases, Scopus has wide coverage of high quality journals, compared to high number of citations at the expense of quality for Google Scholar, and high quality at the expense of number of citations for Web of Science 9– 11.

It is claimed that the emergence of OA journals has led to better dissemination of knowledge with the additional benefit of more citations for the authors, although this is still a matter of debate 12. In this study, we aim to study the OA status of medical journals and the impact of the open-access status on journal indices using the Scopus database.

Methods

Data collection

We collected data on the included journals from Scopus Source List on 1 st November 2018 (see Underlying data 13). We filtered the list for medical journals (which include all specialties in medicine, as per Scopus categorization).

Variables

For each journal, we extracted the following citation metrics: Citation count, Percent Cited, CiteScore, CiteScore Percentile, SCImago Journal Rank, Source Normalized Impact per Paper (SNIP), and SCImago Quartiles. Details about these metrics and how they are calculated can be found on Scopus website.

Moreover, scholarly output is defined as sum of documents published in the serial title (e.g. 2017) in the 3 years prior to the year of the metric (e.g. 2014 – 16). Open access Journals covered by Scopus are indicated as Open Access if the journal is listed in the Directory of Open Access Journals (DOAJ) and/or the Directory of Open Access Scholarly Resources (ROAD).

Statistical analysis

We used SPSS version 22.0 (Chicago, USA) in our analysis. We used means (± standard deviation) to describe continuous variables (i.e. journal indices). We used counts (frequency) to describe other nominal variables (i.e. publishers and OA journals). We performed Mann-Whitney tests to analyze the difference between measurements and OA status, and we presented data as medians (25% to 75% quartiles). To analyze open access journals between radiology and medicine, we used the weighting cases function in SPSS and a Chi-square test. All underlying assumptions were met, unless otherwise indicated. A p value of 0.05 was considered as significant.

Results

In the 2017 Scopus list of journals, there was 5835 medical journals. Regarding the 5 most common publishers, 890 (15.3%) journals were from Elsevier, 653 (11.2%) Springer Nature, 196 (6.8%) Taylor & Francis, 360 (6.2%) Wiley-Blackwell, and 304 (5.2%) Wolters Kluwer. 1293 (22.2%) journals were OA journals. Table 1 indicates the minimum, maximum, mean, and standard deviation of medical journal indices.

Table 1. Descriptive statistics for medical journals.

N Minimum Maximum Mean Std. Deviation
CiteScore 5835 0 130 1.58 2.588
Percentile 5835 0 99 48.03 28.666
Citation count 5835 0 77809 761.87 2221.841
Scholarly output 5835 1 11270 346.14 505.062
Percent cited 5835 0 100 45.67 26.800
SNIP 5835 0.000 88.164 0.75260 1.450990
SJR 5835 0.000 61.786 0.82674 1.572423
Rank 5835 1.00 785.00 162.7102 167.95247
Open in a new tab

SNIP: Source Normalized Impact per Paper; SJR: SCImago Journal Rank.

Upon analyzing the difference between medical OA and non-OA journals, we found significant differences in the following indices:

  • CiteScore (p< 0.001): with a median of 1.19 (25–75%: 0.53–2.21) for OA journals, and a median of 1.06 (25–75%: 0.26–2.18) for non-OA journals.

  • Scholarly output (p< 0.001): with a median of 157 (25–75%: 76–319.5) for OA, and a median of 205 (25–75%: 107–423) for non-OA journals.

  • Percent cited (p< 0.001): with a median of 52% (25–75%: 32%-70%) for OA, and a median of 48% (25–75%: 19%–68%) for non-OA journals.

  • SNIP (p< 0.001): with a median of 0.706 (25–75%: 0.370–1.023) for OA, and a median of 0.617 (25–75%: 0.176–1.013) for non-OA journals.

Upon comparing open access journals between the 5 most common publishers, we found a significant difference (p< 0.001). Post-hoc analysis showed that Wiley-Blackwell has significantly lower number of open access journals 16 (4.4%) open access journals compared to others. Table 2 shows the open access status for the most common publishers.

Table 2. A comparison in the percentage of open access (OA) journals between the top five publishers of medical journals.

Open access Total
No Yes
publishers Elsevier Count 756 134 890
84.9% 15.1% 100.0%
Springer Nature Count 447 206 653
68.5% 31.5% 100.0%
Taylor & Francis Count 324 72 396
81.8% 18.2% 100.0%
Wiley-Blackwell Count 344 16 360
95.6% 4.4% 100.0%
Wolters Kluwer Health Count 229 75 304
75.3% 24.7% 100.0%
Others Count 2442 790 3232
75.6% 24.4% 100.0%
Total Count 4542 1293 5835
77.8% 22.2% 100.0%
Open in a new tab

Discussion

Our study found that OA medical journals had significantly higher CiteScores, Percent cited and SNIP; which is consistent with a number of previous studies made across a variety of disciplines including philosophy, political science, engineering, mathematics, physics, computer science and agriculture; all of which concluded that open access publications have a greater research impact (higher citation rate) than non-open access publications 12, 14– 16. On the other hand, in a randomized controlled trial conducted on 11 biological and medical journals, it was found that only 2 of these journals showed positive and significant OA effects. In addition, it was found that OA advantage is declining by about 7% per year, from 32% in 2004 to 11% in 2007 17. Chua et al. found that there was significantly more citations in OA articles than in non-OA articles within almost identical journals’ impact factor 18. Moreover, comparing citations in OA and non-OA articles in the same journal showed significant citation privilege for OA publications in several studies. For example, for the Journal of Postgraduate Medicine a comparison of citations per 100 articles per year before and after the journal became open access showed an increase between 3 and 4.5 times in citations 19. In a longitudinal study of a cohort of OA and non-OA articles, it was shown that OA articles are cited earlier, and almost 2 times more frequently than non-OA articles in the first 4–16 months after publication in the same journal 20. Regardless of all the aforementioned findings, our study found that non-OA medical journals have significantly higher Scholarly Output which can be strongly linked to the fact that most non-OA medical journals have been established years before OA journals, which have only recently emerged 21.

We found that the number of OA journals varied among publishers, with Whiley-Blackwell having the least, with only 16 journals (4.4%), and the most with Springer Nature (206, 31.5%). In a previous study that analyzed OA articles published by different publishers, regardless of the discipline, they found that Elsevier had the highest number of OA articles, followed by Springer Nature and Whiley-Blackwell 22. A longitudinal study comparing hybrid open access articles between publishers found great variation depending on the discipline 23. For instance, medicine is the discipline which most frequently publishes in hybrid OA 23.

Our study has potential limitations. In this study, we didn’t account for the effect of publishing OA articles in non-OA journals (hybrid journals), as “Gold” OA publishing (i.e. fully OA journals) relates to publication of articles that are freely available to view and these may occur in OA or hybrid journals. Moreover, future studies should consider analyzing specialties within medicine (e.g. oncology), where we believe there will be variations in the effect of OA publishing within these specialties.

Data availability

Underlying data

Harvard Dataverse: Medical journals. https://doi.org/10.7910/DVN/YYUTGG 13.

This project contains the following underlying data:

  • Medical journals 2017 dataset.tab (Scopus search results from the 1 st November 2018)

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Funding Statement

The author(s) declared that no grants were involved in supporting this work.

[version 1; peer review: 3 approved

References

  • 1. Budapest Open Access Initiative (BOAI). Interlending & Document Supply.2002;30(2). 10.1108/ilds.2002.12230bab.012 [DOI] [Google Scholar]
  • 2. Nwagwu WE, Ahmed A: Building open access in Africa. Int J Technol Manage. 2009;45(1/2):82–101. 10.1504/IJTM.2009.021521 [DOI] [Google Scholar]
  • 3. Nigam A, Nigam PK: Citation Index and Impact factor. Indian J Dermatol Venereol Leprol. 2012;78(4):511–6. 10.4103/0378-6323.98093 [DOI] [PubMed] [Google Scholar]
  • 4. AlRyalat SA, Malkawi LW, Momani SM: Comparing Bibliometric Analysis Using PubMed, Scopus, and Web of Science Databases. J Vis Exp (Pending Publication). 2019;e58494 Reference Source [DOI] [PubMed] [Google Scholar]
  • 5. Hirsch JE: An index to quantify an individual's scientific research output. Proc Natl Acad Sci U S A. 2005;102(46):16569–16572. 10.1073/pnas.0507655102 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6. Adams J: The use of bibliometrics to measure research quality in UK higher education institutions. Arch Immunol Ther Exp (Warsz). 2009;57(1):19–32. 10.1007/s00005-009-0003-3 [DOI] [PubMed] [Google Scholar]
  • 7. Kinney AL: National scientific facilities and their science impact on nonbiomedical research. Proc Natl Acad Sci U S A. 2007;104(46):17943–17947. 10.1073/pnas.0704416104 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8. King DA: The scientific impact of nations. Nature. 2004;430(6997):311–316. 10.1038/430311a [DOI] [PubMed] [Google Scholar]
  • 9. Kulkarni AV, Aziz B, Shams I, et al. : Comparisons of citations in Web of Science, Scopus, and Google Scholar for articles published in general medical journals. JAMA. 2009;302(10):1092–6. 10.1001/jama.2009.1307 [DOI] [PubMed] [Google Scholar]
  • 10. Burnham JF: Scopus database: a review. Biomed Digit Libr. 2006;3:1. 10.1186/1742-5581-3-1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11. Bar-Ilan J: Citations to the “Introduction to informetrics” indexed by WOS, Scopus and Google Scholar. Scientometrics. 2010;82(3):495–506. 10.1007/s11192-010-0185-9 [DOI] [Google Scholar]
  • 12. Antelman K: Do Open-Access Articles Have a Greater Research Impact? Coll Res Libr. 2004;65(5):372–382. 10.5860/crl.65.5.372 [DOI] [Google Scholar]
  • 13. AlRyalat SA: Medical journals. Harvard Dataverse, V1.2019. 10.7910/DVN/YYUTGG [DOI] [Google Scholar]
  • 14. Kousha K, Thelwall M, Rezaie S: Using the Web for research evaluation: The Integrated Online Impact indicator. J Informetr. 2010;4(1):124–135. 10.1016/j.joi.2009.10.003 [DOI] [Google Scholar]
  • 15. Lawrence S: Free online availability substantially increases a paper's impact. Nature. 2001;411(6837):521. 10.1038/35079151 [DOI] [PubMed] [Google Scholar]
  • 16. mcveigh.2004. [Google Scholar]
  • 17. Davis PM, Lewenstein BV, Simon DH, et al. : Open access publishing, article downloads, and citations: randomised controlled trial. BMJ. 2008;337:a568. 10.1136/bmj.a568 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18. Chua SK, Qureshi AM, Krishnan V, et al. : The impact factor of an open access journal does not contribute to an article’s citations [version 1; referees: 2 approved]. F1000Res. 2017;6:208. 10.12688/f1000research.10892.1 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19. Sahu DK, Gogtay NJ, Bavdekar SB: Effect of open access on citation rates for a small biomedical journal.2005. Reference Source [Google Scholar]
  • 20. Eysenbach G: Citation advantage of open access articles. PLoS Biol. 2006;4(5):e157. 10.1371/journal.pbio.0040157 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21. Björk BC, Solomon D: Open access versus subscription journals: a comparison of scientific impact. BMC Med. 2012;10(1):73. 10.1186/1741-7015-10-73 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22. Piwowar H, Priem J, Larivière V, et al. : The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles. PeerJ. 2018;6:e4375. 10.7717/peerj.4375 [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23. Laakso M, Björk B: Hybrid open access—A longitudinal study. J Informetr. 2016;10(4):919–932. 10.1016/j.joi.2016.08.002 [DOI] [Google Scholar]
F1000Res. 2019 Apr 3. doi: 10.5256/f1000research.19663.r45999

Reviewer response for version 1

Sung-Tae Hong 1

General comment: This article describes difference of journal citation metrics between OA and non-OA journals in medical scope. The subjected journals are 5835 medical journals which are indexed in the Scopus and compared indices are metrics data supplied by Scopus and SCImago. The data showed significantly higher citation indices in OA journals than those in non-OA journals. The conclusion is that OA medical journals are cited more. The conclusion is known already in OA journals of other disciplines but it is valuable to publish.

The conclusion is sound and positive as expected, but I think we have to think about other issues on this topic. OA means no financial barrier for readers, which can facilitate citations. But most of researchers in established institutions have no barrier to non-OA journals because their institutions are able to subscribe to the non-OA journals. For those researchers, non-OA journals are free. I think that is the main reason of the small difference of those indices between the two groups although the difference is statistically significant. The statistics must be supported by large number of subject journals. This point must be discussed.

Specific comments

  1. Definition of OA: journals included in DOAJ or ROAD. There are more Scopus journals outside of the 2 directories. PubMed Central may add more OA journals.

  2. The core results are difference of the indices that must be presented in a Table. Tables 1 and 2 are supportive but not cores. For reader friendly writing, one Table is required for results described in the text.

  3. Table 2 analyzed total numbers of OA journals by publishers. It may be interesting for readers to summarize whether there are index differences by publishers.

  4. Add discussion on the difference of indices: CiteScore 0.19 vs. 1.06; Scholarly output median 157 vs. 205; Percent cited 52% vs. 48%; SNIP median 0.706 vs. 0.617. All are highly significant by p<0.001. The gaps are significant but not so big because there are many other factors of citation. We should approach more details of citation analysis. 

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2019 Apr 3. doi: 10.5256/f1000research.19663.r46001

Reviewer response for version 1

Deeb N Salem 1

  1. The article is generally well done with respect to its method and purpose. My major concern is its purpose: assuming adequate methods, statistical analyses and result, what is the significance?  If there are implications of the research or significance of the conclusions, they should be clearly stated.

  2. The article is dependent upon the accuracy and scope of the Scopus database. I wonder how that was chosen and whether Pub Med might have been consulted as well. The accuracy of the conclusions is no better than the comprehensiveness of Scorpus.

  3. Are articles that appear in print but are archived online included?

  4. Otherwise the study seems to have been conducted well (methodologically) and the results competently reported.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2019 Mar 19. doi: 10.5256/f1000research.19663.r45458

Reviewer response for version 1

Suneet Sood 1

This paper intends to measure journal indices for OA journals, and compare them with the indices of Non-OA journals. The authors used the Scopus database.

Although the theme is not new, I believe that the large number of indices calculated by the authors is a useful contribution to the literature.

Database: I am a little ambivalent about the Scopus database. It is an extensive one, but to my mind it has two characteristics that need to be kept in mind. First, Google Scholar has, in all probability, overtaken Scopus in the breadth of coverage of publications for indexing (though till about 8 years ago Google Scholar was inferior to Scopus). Secondly, GS is more lax in its selection of journals to index, and it is more than likely that several “poor quality” journals are included. The authors themselves make these observations. I submit that the so-called poor quality of the journals indexed by GS is not really a drawback, and inclusion of these journals is more likely to provide a realistic picture. That said, using the Scopus database is an acceptable choice, and the sampling is large.

Statistics: The choice of statistical tests is acceptable. I think that choosing two groups (OA vs Non-OA) is adequate for the objective of this paper. Would a third group, hybrid, have helped? Possibly, but hybrid journals are extremely varied in the number of articles they allow for open access, and I doubt that one hybrid journal, allowing 10% of its articles for free access, can be compared with another that allows 25% access immediately and full access after 6 months. However, I was not able to duplicate the calculations of the authors. The authors state as follows:

CiteScore median = 1.19 for OA journals, 1.06 for non-OA journals.

However, I calculate 1.2750 and 1.16 respectively, though the differences are still statistically significant.

I get different values for the other metrics as well. Perhaps the authors should consider rechecking the values with their statisticians.

Methods: Although the methods are described with reasonable clarity, I was unsure of the period covered by the data collection. The data file suggests that the data was collected for journal issues published from 30 April 2018 onwards. Have I understood correctly? I would request the authors to provide this detail in the methods.

Results: The authors have recorded data for SJR values but have not discussed the results. There may be no significant differences between the two groups (OA, NOA), but I believe that they must discuss the implication of this result. The median SJR for NOA papers is slightly (but not significantly) higher than that for OA papers. Is this because NOA papers have a higher “prestige”? (Compare with their comments for scholarly output.)

Recommendation: The authors should consider: A. cross-checking the results, B. clarifying the time span that is covered by the study, and C. commenting on the SJR results.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2019 Mar 12. doi: 10.5256/f1000research.19663.r45461

Reviewer response for version 1

Ernesto Roldan-Valadez 1

The authors have performed an interesting study comparing differences in selected metrics (CiteScore, percent cited, SNIP, scholarly output) between medical OA and non-OA journals.

This study, however, presents some limitations that require significant revisions:

 

  • Authors are using a robust number of journals (5835 medical journals) that will allow the performance of parametric tests. I would like to see that they applied at least an ANOVA comparing three groups: OA, non-OA and OA articles in non-OA journals (hybrid journals). The hybrid group is fundamental, and they do not include it. The authors should mention to the reader the bias of this decision.

  • Also in this point, it is essential the authors explain why, if previous studies have concluded that the Eigenfactorscore is the best predictor of citations 1 , 2, they did not include this metric in their analyses? Please also explain to the readers, why a linear-mixed-model design analysis was not performed.

  • It is necessary to mention references of recent articles 3 citing the existing correlations between the selected bibliometrics (CiteScore vs SNIP, Citescore vs IF, etc.) with at least two purposes: that the authors justified why they did not include a correlation analysis in their study, and that the readers be aware of the limitations in the correlation analysis, and also how the medical speciality may influence the results. 

  • It would be desirable to present a subgroup analysis of the medical specialities with the higher number of citations (for example oncology) as an example of the expected variability within subspecialties.

  • If you report in the methods section that you used the SCImago Quartiles, why not control the effect of this variable using ANCOVA O MANCOVA? For example, if the authors are using the data from 5835 medical journals, this data allows a more robust analysis besides descriptive statistics and Mann-Whitney tests.

I have read this submission. I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

References

  • 1. Roldan-Valadez E, Orbe-Arteaga U, Rios C: Eigenfactor score and alternative bibliometrics surpass the impact factor in a 2-years ahead annual-citation calculation: a linear mixed design model analysis of Radiology, Nuclear Medicine and Medical Imaging journals. Radiol Med.2018;123(7) : 10.1007/s11547-018-0870-y 524-534 10.1007/s11547-018-0870-y [DOI] [PubMed] [Google Scholar]
  • 2. Diaz-Ruiz A, Orbe-Arteaga U, Rios C, Roldan-Valadez E: Alternative bibliometrics from the web of knowledge surpasses the impact factor in a 2-year ahead annual citation calculation: Linear mixed-design models' analysis of neuroscience journals. Neurol India.66(1) : 10.4103/0028-3886.222880 96-104 10.4103/0028-3886.222880 [DOI] [PubMed] [Google Scholar]
  • 3. Villaseñor-Almaraz M, Islas-Serrano J, Murata C, Roldan-Valadez E: Impact factor correlations with Scimago Journal Rank, Source Normalized Impact per Paper, Eigenfactor Score, and the CiteScore in Radiology, Nuclear Medicine & Medical Imaging journals. Radiol Med.2019; 10.1007/s11547-019-00996-z 10.1007/s11547-019-00996-z [DOI] [PubMed] [Google Scholar]

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Availability Statement

    Underlying data

    Harvard Dataverse: Medical journals. https://doi.org/10.7910/DVN/YYUTGG 13.

    This project contains the following underlying data:

    • Medical journals 2017 dataset.tab (Scopus search results from the 1 st November 2018)

    Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

    Articles from F1000Research are provided here courtesy of F1000 Research Ltd

    RESOURCES