Open access publishing: a study of current practices in orthopaedic research

Abstract

Background

Open access (OA) publications have changed the paradigm of dissemination of scientific research. Their benefits to low-income countries underline their value; however, critics question exorbitant publication fees as well as their effect on the peer review process and research quality.

Purpose

This study reports on the prevalence of OA publishing in orthopaedic research and compares benchmark citation indices as well as evidence quality derived from OA journals with conventional subscription based orthopaedic journals.

Methods

All 63 orthopaedic journals listed in ISI’s Web of Knowledge Journal Citation Report (JCR) were examined. Bibliometric data attributed to each journal for the year 2012 was acquired from the JCR. Studies that fulfilled the criteria of level I evidence were identified for each journal within PubMed. Individual journal websites were reviewed to identify their open access policy. A total of 38 (60.3 %) journals did not offer any form of OA publishing; however, 20 (31.7 %) hybrid journals were identified which offered authors the choice to publish their work as OA if a publication fee was paid. Only five (8 %) journals published all their articles as OA. There was variability amongst the different publication fees for OA articles. Journals that published OA articles did not differ from subscription based journals on the basis of 2012 impact factor, citation number, self citation proportion or the volume of level I evidence published (p > 0.05).

Conclusions

OA journals are present in orthopaedic research, though in small numbers. Over a third of orthopaedic journals catalogued in the ISI Web of Knowledge JCR® are hybrid journals that provide authors with the opportunity to publish their articles as OA after a publication fee is paid. This study suggests equivalent importance and quality of articles between OA and subscription based orthopaedic journals based on bibliometric data and the volume of level I evidence produced. Orthopaedic researchers must recognize the potential benefits of OA publishing and its emerging presence within the field. Further examination and consensus is required in orthopaedic research to generate an OA system that is robustly regulated and maintains research quality.

Introduction

Since 1665 when the Journal des Sçavans became the world’s first publication dedicated to science, journals have been a primary means of disseminating scientific knowledge [1]. In 1995 the British Medical Journal published an article describing the world wide web as “the crowning glory of the Internet” and proposed it would be “a powerful medium for electronic publishing” [2]. Since that article was published there has been a revolution in the paradigm of scientific publishing with the number of online peer reviewed scientific journals having grown from nine to approximately 25,000 [2, 3]. The opportunities the Internet has provided for researchers to identify, acquire and consume information are immeasurable [4]. Although convention dictated that cost of publishing was covered by the subscription fees paid by institutions, libraries and individuals, this model was at odds with a key value of the digital age which stated that “information wants to be free” [5].

The first formal calls to action came from the Declaration of the Budapest Open Access Initiative [6] and the Bethesda Statement on Open Access Publishing [7] in 2002 and 2003, respectively. The core values that these groups had outlined were that peer reviewed articles should be donated for publication by their authors and should be available online, and free for all [8]. However, given that the cost to a publisher of producing an article can be as high as $4,000, the shift to Open Access (OA) publishing means that researchers often must reach into their own limited monetary resources to cover these costs [9]. Critics often highlight the irony of sparing taxpayers the cost of reading research that they have funded, whilst at the same time using more of their taxes to pay for publication costs [10]. Although the benefits of OA publishing have been suggested in relation to providing orthopaedic information and research within low-income countries [11], concerns and perceptions about its effect on the quality of research have yet to be explored. In light of a number of orthopaedic publications that have examined and reported on research quality as well as bibliometrics [12–14], it is surprising that such an intense debate within the medical research community [15] has yet to result in an evaluation of OA publishing in orthopaedics.

The aim of this study was to provide an overview of OA publishing in orthopaedic research. The three principle goals were (1) to report on the proportion of orthopaedic journals that publish OA articles, (2) to scrutinize bibliometric data arising from these journals which benchmark the journal’s importance within orthopaedic research, and (3) to determine whether the volume of level I evidence they produce is comparable to that of conventional subscription-based orthopaedic journals.

Material and methods

Information sources

The ISI Web of Knowledge journal citation report (JCR®) was used to generate a sample of 63 orthopaedic journals that were the focus of this study. Using the ISI Web of Knowledge as a means of sampling orthopaedic journals and scrutinizing citation data is commonly performed in orthopaedic research [16]. Citation data was generated within the JCR®.

PubMed was used to search each journal and identify level I studies published within the time period specified within the search criteria. Each journal’s website was examined to determine its Open Access publishing policy and publication fees.

Eligibility and search strategy

All journals listed in the orthopaedics section of the ISI Web of Knowledge were deemed eligible. Non-English language journals were not excluded. Citation information presented with the JCR® is based on journal bibliometrics for the year 2012.

The search strategy for identification of level I evidence used the ‘advanced search’ option in PubMed to search by individual journal for articles categorized as “randomized controlled trial” and “meta–analysis”. The search was limited to the year 2012 and individual articles were examined to ensure they represented randomized controlled trials (RCTs). Meta-analysis studies were only included if their results were derived from randomized controlled trials. Meta-analysis of prospective well-designed studies are by definition level II studies [17] and were therefore excluded. The Google search engine was used to identify individual journal editorial policy with regards to OA publishing.

Data extraction

The following information was extracted for each journal: its title, whether it was a subspecialty journal, the region it was published by continent, its language of publication and the number of level I studies published in 2012. Bibliometric data from 2012 included the total number of citations the journal had received, the proportion of citations that were self cites and its impact factor. Information specific to the journal’s OA publishing policy included whether there was a uniform policy to publish all articles as OA or the policy was optional to authors who were submitting their research. Furthermore, the cost to authors for publishing articles as Open Access was also recorded.

Statistical analysis

Data were analysed in SPSS 20.0 (SPSS Inc, Chicago, IL). The distribution of the data was assessed with a Kolmogorov-Smirnov test and found to be non-parametric. A comparison of open access journals and non-open access journals was performed with a Mann Whitney U test to determine whether there was a statistical difference in their bibliometrics and the of volume of level I evidence they published.

Results

Sixty-three orthopaedic journals were examined in this study. The origins of these publications were predominantly Europe (n = 32) and North America (n = 28). There were three journals that originated from India and Brazil as well as both Australia and New Zealand. The majority of journals were printed in English (n = 55), with a few non-English language journals published in Turkish, Portuguese, German and French (n = 8). A total of 37 were general orthopaedic journals and 26 were dedicated subspecialty publications.

Assessment of bibliometric data from the year 2012 for all the journals in the JCR® revealed that the median impact factor was 1.35 (IQR 0.66–2.26), the median number of total journal citations was 2,217 (IQR 358–5,651) and the median proportion of self-citations was 0.11 (IQR 0.05–0.16). The median number of level I studies published in 2012 by the selected journals was four (IQR 0–14).

Within the instruction to authors domain in each journal’s website, three distinct policies that related to OA publishing were identified. First, there were 38 (60. 3 %) journals that did not offer any form of OA publishing to researchers and access to their research was provided by institutional or personal subscription to the journal. Second, there were 20 (31.7 %) journals that were subscription based however offered researchers the option to publish their work as OA if they paid a specified publication fee. Such journals are often termed ‘hybrid journals’ [18]. Lastly, five (8 %) journals had a policy that stated that all papers would be published as OA to the public. Amongst these five journals, two did not charge authors a publication fee and the remaining three charged $1,715, $1,985 and $2,270, respectively. For the 20 hybrid journals that offered authors the option to pay for their article to be published as OA, the median publication fee was $3,000 (IQR $2,500–$3,000). Publication fees for both hybrid and fully OA journals in relation to their impact factor are reported in Fig. 1.

Fig. 1.

Variation in publication fees of different journals in relation to their impact factor (IF)

All five journals that printed all their articles as OA were from Europe. Fifteen (75 %) journals that provided the option of open access publishing (n = 20) were also from Europe. The variation of regional practices with regards to journals that offered some form of OA publishing (either compulsory or hybrid) are demonstrated in Fig. 2. A similar proportion of subspecialty orthopaedic journals (42.30 %) provided some form of open access publishing compared to their generalist counterparts (37.83 %).

Fig. 2.

Regional variations of journals offering OA publication. This includes combined data from hybrid journals and fully OA journals

For evaluation of statistical differences in impact factor, the overall citations received in 2012, the proportion of self-citations and the number of level I studies published, three journal pairings were compared. There was no statistical difference in the bibliometric data or volume of level I evidence published for journals that had a compulsory OA policy compared to those with subscription-based journals. Similarly, there was no statistical difference in these variables when hybrid journals were compared to subscription-based journals, and there was no statistical difference found when journals that offered any form of open access publishing were compared with purely subscription-based journals. A p-value of less than 0.05 was viewed to indicate statistical significance. Descriptive data and statistical analysis are summarized in Table 1.

Table 1.

Comparison of key citation indices and volume of level I studies produced in 2012 between different OA journal types and subscription-based journals

Indices	Hybrid journals (n = 20)	Subscription based journals (n = 38)	P value (Mann Whitney U test)
Median impact factor	1.57 (IQR 0.53–2.12)	1.28 (0.76–2.35)	0.45
Median citations in 2012	1,925.5 (IQR 273–5,618)	2,625 (IQR 560.75–7,444.50)	0.58
Median proportion of self cites	0.09 (IQR 0.02–0.16)	0.13 (IQR 0.06–0.20)	0.38
Median number of level I studies published in 2012	3 (IQR 0–18)	3.50 (IQR 0–12.5)	0.89
	Journals that are entirely open access (n = 5)	Subscription based journals (n = 38)	P value (Mann Whitney U test)
Median impact factor	1.47 (IQR 0.81–2.31)	1.28 (0.76–2.35)	0.97
Median citations in 2012	358 (IQR 270.50–4,984)	2,625 (IQR 560.75–7,444.50)	0.35
Median proportion of self cites	0.05 (IQR 0.015–0.13)	0.13 (IQR 0.06–0.20)	0.09
Median number of level I studies published in 2012	4 (IQR 1–30.50)	3.50 (IQR 0–12.5)	0.76
	Journals that publish open access papers (n = 25)	Subscription based journals (n = 38)	P value (Mann Whitney U test)
Median impact factor	1.47 (IQR 0.60–2.12)	1.28 (0.76–2.35)	0.51
Median citations in 2012	1,734 (IQR 278–5,605.5)	2625 (IQR 560.75–7,444.50)	0.4
Median proportion of self cites	0.08 (IQR 0.05–0.16)	0.13 (IQR 0.06–0.20)	0.17
Median number of level I studies published in 2012	4 (IQR 0–18)	3.50 (IQR 0–12.5)	0.99

Discussion

This study has demonstrated that although a small proportion of orthopaedic journals publish all their articles as OA, more than a third provide authors the option to publish their work as OA given that they pay a median publication fee of $3,000. Furthermore comparison of conventional subscription based journals with hybrid journals and those that published all their research as OA revealed there was no statistical difference in benchmark citation indices or the volume of level I evidence that was published in 2012.

Implications of study findings

Although the number of orthopaedic journals that have an entirely OA policy is relatively small, the concept of OA publishing is relatively new and many so-called hybrid journals are now prevalent in orthopaedics. These findings suggest that the growth of OA in orthopaedic research reflects trends that are seen across scientific publishing [4]. Given that many researchers have a limited knowledge of OA publishing and its implications [19], these findings are important to authors of orthopaedic research who will need a better understanding of this new paradigm in publication policy.

The financial implications of OA publishing in orthopaedics are significant. A key benefit of this model is that information will be easily accessible to researchers. In developed countries where many researchers have a wealth of access to research owing to their institutional subscriptions this may not be of great importance, however in low-income countries this freedom of access to information is invaluable [11]. Whether an OA model is actually cost effective to governments and research bodies remains a controversial topic [20]. Ruthless cooperate attitudes by publishers are often highlighted by their large profit margins [21] and underline that conventional subscription-based journals are far from cost effective for scientists; however, it is unlikely that all orthopaedic authors will have the financial means to choose an OA publication that requires a significant publication fee. Furthermore, in certain circumstances researchers may not be able to choose. The Welcome Trust requires all recipients of its grants to make their work freely available on PubMed Central no later than six months after publication [22]. In principle this willingness to disseminate research that is free for all to see is commendable, but there is basis to recent perspective that suggests that OA publishing is “Open but not free” [23]. PubMed Central diverts approximately $4 million from the National Institute of Health’s budget [24]. Suggestion that this may be money that would have been available for research is poignant [23].

Another key consideration with regards to OA publishing relates to a potential conflict of interest that the model inherently exhibits. These journals are generally not constrained by size and in view of the financial incentives to publish more research there have been suggestions that this may lead to lower rejection rates and subsequently publication of poor quality studies [25]. The results of this study are contrary to such a notion. OA and hybrid orthopaedic journals had similar citation indices to conventional subscription-based journals. There was no evidence of inflation of impact factors amongst journals that published OA research, as their self-citation rates were not statistically different when compared with subscription-based journals. This suggests that orthopaedic researchers view the evidence that originates from OA journals to be equally important; however, whether they are aware of the OA status of the work is not known. Furthermore, as the volume of level I evidence produced by OA and hybrid orthopaedic journals is similar to that of subscription-based journals, there is no clear evidence to suggest that orthopaedic journals with OA publication policies do not publish high quality research.

OA publishing is likely to continue to grow in orthopaedics as it does across all scientific fields [4]. Researchers should recognize the benefits of publishing their work as open access, especially to low income countries. Variability in publication costs between these journals continues to raise questions within the scientific community [9] and a consensus is required within orthopaedics how to manage such concerns. Although the results of this study suggest that the importance and quality of such journals is equivalent to subscription-based journals, transparency and a uniform policy to OA publishing across orthopaedics may dispel common beliefs that it breeds low quality research [25]. Further research to examine the proportion of articles within hybrid journals that are OA, as well as their sources of funding will shed further light on the uptake of this model within orthopaedic research and determine whether it has provided feasibility for publication by researchers who are not supported by institutional, industrial or government grants.

Study limitations

There are three main limitations to this study. Firstly, the search strategy would have failed to retrieve a number of orthopaedic journals. PubMed Central and the Directory of Open Access Journals hold a number of titles that are not included in ISI’s Web of Knowledge JCR®. Neglecting these publications may be viewed to be a shortcoming within the methodology. However, the search strategy was created to sample mainstream orthopaedic journals in a uniform manner, as it would be impossible to include every journal available. The exponential proliferation of new journals is well recognized and is visible in the regular emails that researchers receive inviting them to submit an article to a newly established journal [26]. This study limits is evaluation to journals in the JCR® so that conclusions can be drawn on a sample of well recognized journals that surpass a certain proxy level of quality which has been drawn from the JCR®’s entrance criteria. Examination of more orthopaedic journals is beyond the scope of this study however may be of interest as it could generate different conclusions when less recognized and lower quality publications are included in the analysis.

A second limitation in this study is that it did not include an examination and comparison of “gold” and “green” OA publishing. “Gold” OA publishing relates to publication of articles that are freely available to view and these may occur in OA or hybrid journals. “Green” OA publishing involves authors publishing their article in a subscription-based journal however making an OA version of it available in a central repository such as PubMed Central. Examination of “green” OA publishing practices would provide insight into whether orthopaedic researchers recognize the importance of contributing their work to a domain that allows individuals from low in-come countries to read their work. Future evaluation of OA practices will encompass this aspect of OA publishing.

Finally, although a significant difference was not found in comparisons of OA with subscription-based journals, in view of the small number of fully OA journals that were examined, there is potential for type II error in these results. This requires consideration when reporting at the conclusions of this study.

Conclusions

OA journals are present in orthopaedic research, though in small numbers. Over a third of orthopaedic journals catalogued in the ISI Web of Knowledge JCR® are hybrid journals that provide authors with the opportunity to publish their articles as OA after a publication fee is paid. The median cost of publication fees for hybrid journals is $3,000; however, there is variability amongst different journals, and a more transparent OA model is required to enable authors to understand how these fees are established. Comparison of OA and hybrid orthopaedic journals with subscription-based journals revealed no significant difference in impact factor, citation count and self-citation proportion in 2012. Furthermore, there was no significant difference in the number of level I studies published between these journal types. Orthopaedic researchers must recognize the potential benefits of OA publishing and its emerging presence within the field; however, further examination and consensus is required in orthopaedic research to generate an OA system that is robustly regulated and promotes quality in research.