Bias and other limitations affect measures of journals in integrative and complementary medicine

Abstract

Publishing articles in a prestigious journal is a golden rule for university professors and researchers nowadays. Impact factor, journal rank, and citation count, included in Science Citation Index managed by Thomson Reuters Web of Science, are the most important indicators for evaluating the quality of academic journals. By listing the journals encompassed in the “Integrative and Complementary Medicine” category of Science Citation Index from 2003 to 2013, this paper examines the publication trends of journals in the category. The examination includes number, country of origin, ranking, and languages of journals. Moreover, newly listed or removed journals in the category, journal publishers, and open access strategies are examined. It is concluded that the role of journal publisher should not be undermined in the “Integrative and Complementary Medicine” category.

INTRODUCTION

Impact factor (IF), journal rank (JR), and citation count are currently the most important indicators for evaluating the quality of academic journals and a person’s output of professional research. All of these indicators can be searched in the Thomson Reuters Web of Science (WoS) database, which, with its annually updated Journal Citation Report (JCR), yields a clear-cut indicator for assessing the quality of academic journals. However, numerous scholars challenge the idea that IF, JR, and citation count can evaluate the quality of academic journals effectively [1, 2].

This article analyzes the characteristics of journals listed in the “Integrative and Complementary Medicine” (ICM) category of the Science Citation Index (SCI) in WoS <http://ip-science.thomsonreuters.com/mjl/publist_sciex.pdf>. A review of the academic journals listed in the ICM category from 2003 to 2013 revealed publication trends. Consequently, an analysis of the characteristics of the journals in this category is likely to be of interest to researchers in this field.

METHODS

The author examined available WoS data for all the journals listed in the ICM category of SCI for the years 2003 through 2013 <http://wokinfo.com/products_tools/analytical/jcr/>. The information provided for ICM journals by WoS for this period of time included the number of journals, each journal’s country of origin, status of each journal as newly listed on or removed from the ICM list, and the ranking of each journal based on its IF.

RESULTS

The ICM category consisted of twenty-two journals in the 2013 JCR. The data retrieved from the WoS database from 2003 to 2013 revealed the characteristics of, and trends in, the journals listed in the ICM category over the last eleven years. Six major findings were derived from an examination of the data.

1. The number of journals listed in the ICM category of WoS gradually increased from nine to twenty-two journals in the past eleven years. It contained nine journals in 2003–2004, ten in 2005–2006, twelve in 2007, fourteen in 2008, seventeen in 2009, twenty-one in 2010, and twenty-two 2011–2013.

2. England and the United States were the countries with the greatest increases in the number of journals listed. The United States had five journals listed in 2003 and eight in 2013; England had one journal listed in 2003 and seven in 2013.

3. There were four journals listed in 2009 from non-European and non-US regions: one from Africa, one from the People’s Republic of China (PRC), and two from Australia, all of which were published in English.

4. Two Australian journals were not listed in the ICM category in 2012 and 2013, and their listing periods lasted only three years and four years, respectively.

5. European and American journals ranked in the first quartile (Q1) of the journals in this category from 2003 to 2013.

6. Open access (OA) journals have been listed since 2010.

DISCUSSION

The results indicate that English is still the major language used for journals in this field. According to the definition offered by the National Center for Complementary and Alternative Medicine, “Integrative medicine combines mainstream medical therapies and [complementary and alternative medicine (CAM)] therapies for which there is some high-quality scientific evidence of safety and effectiveness. Complementary and alternative medicine is a group of diverse medical and health care systems, practices, and products that are not presently considered as a part of conventional medicine” [3]. In general, ICM encompasses non-Western or conventional medicines, and these medicines have a profound relationship with non-Western cultures or traditions, such as Chinese medicine, ayurveda, Islamic medicine, and Kampo (Chinese medicine in Japan) [4]. Apparently, the research output of non-Western medical traditions can be accepted in WoS only when the authors translate them into English. This phenomenon is the most obvious in the case of Chinese medicine. In fact, no Chinese medical journals written in Chinese are listed in WoS, because WoS uses only bibliographic information in English to determine its IFs [5].

Scholars have long noted that WoS has an English-language bias [6, 7]. This English bias leads to a tremendous problem, because myriad research findings of Chinese medicine, Islamic medicine, and Kampo are written in their native languages, not in English. As ICM emphasizes non-Western conventional medicines, this English-language bias is an obstacle to the ability to reflect any developing trends in this field accurately.

Some of the journals listed in the ICM category address a particular therapy. For instance, Chinese medicine accounts for five such journals, homeopathy for one, and ethnomedicine for two, while other so-called integrative journals are not limited to a specific type of treatment system (for example, BMC Complementary Medicine, Evidence-Based Complementary and Alternative Medicine, Journal of Alternative and Complementary Medicine). The recognition of Chinese medicine in ICM is growing around the globe [8]. In general, integrative journals and Chinese medicine journals have gained more attention from academia in this field.

Since 2007, journals published in non-European and non-American regions have begun to be included in the ICM category. However, journal publication is still dominated by a few publishers, and as a result, only three journal publishers (African Journal of Traditional Complementary and Alternative Medicines, World Scientific, and Karger) are located in non-European and non-US regions. Ironically, the PRC’s Chinese Journal of Integrative Medicine is published by Springer, which is based in the United States.

Despite the fact that WoS considers journals published in different regions, journals are still dominated by 5 major publishers. Europe and the United States host 5 of the world-renowned scholarly publishers: Elsevier, Springer, Wiley, Taylor & Francis, and Sage. According to the websites of these 5 major scholarly publishers, Elsevier holds 1,640 titles, Springer holds 1,570 titles, Wiley holds 1,202 titles, Taylor & Francis holds 909 titles (in 2012), and Sage holds 470 titles in the 2013 JCR. Altogether, the SCI and Social Science Citation Index (SSCI) <http://ip-science.thomsonreuters.com/mjl/publist_ssci.pdf> catalog nearly 10,000 journals, according to the 2013 JCR. Approximately 50% of SCI/SSCI journals are published by the 5 major scholarly publishers [9]. Elsevier and Springer each have over 15% of all published journals. Journal publishers in the ICM category reflect a similar situation, in other words, being dominated by the major publishers. Elsevier accounts for 8, Springer for 3, and Sage for 1. It is not unexpected that Elsevier alone accounts for over 36% of all published journals in the ICM category.

Of the journals listed in the ICM category, 8 journals are jointly published by professional societies and European and US publishers, including Acupuncture in Medicine, BMC Complementary Medicine, the Chinese Journal of Integrative Medicine, Chinese Medicine, Forschende Komplementarmedizin und Klassische Naturheilkunde, the Journal of Alternative and Complementary Medicine, the Journal of Ethnopharmacology, and the Journal of Herbal Medicine. At present, large-scale European and US publishers are seeking the joint publication of journals with professional societies and academic institutions in order to guarantee the quality of their listed articles. Elsevier published 372 titles on behalf of societies in 2013 JCR. Springer also claimed that it had published many titles on behalf of professional societies with whom it partners [10].

Two Australian journals were removed from WoS: the Journal of Complementary Medicine in 2012 <http://www.appco.com.au/JCM/> and the Journal of the Australian Traditional-Medicine Society in 2013 <http://www.atms.com.au/about-atms/journal/>. The removal of these two journals occurred because their IF values in 2011 were too low, with the IF for the latter journal being zero. Three reasons contribute to their low IF values. First, the exposure of their articles was limited. The circulations of these two journals were not large enough, and they were not published by prominent publishers. Also, they were not entered into commonly used journal databases (such as ScienceDirect, JSTOR, Academic Prime Search, and ProQuest), so researchers could only gain access to them through their websites. Second, the full text of the journals was not available on their websites, and their tables of contents were provided only in 2009 and 2010. As these two journals are self-published by academic associations, they are not widely circulated and have only limited online promotion. Third, the Journal of the Australian Traditional Medicine Society publishes articles on the activities of the Australian Traditional Medicine Society and on all aspects of natural medicine as taught and practiced in Australia. The articles must be on a theme that is generally accepted as being part of natural medicine, as recognized by the Australian Traditional Medicine Society, and as being relevant to the clinical practice of natural medicine in Australia. Because of this prerequisite, it is difficult to get scholars to use these journals as references.

In contrast, journals published by large-scale scholarly publishers are included in large databases (such as ScienceDirect and Scopus, managed by Elsevier) that can be easily discovered by readers and researchers throughout the world, enhancing the citation count. Journals of large-scale scholarly publishers also usually occupied the Q1 ranking. For example, Elsevier journals occupied the top position in 61 subject categories in the 2013 JCR [11]. Journals not published by large-scale publishers may suffer from low visibility. As a result, European and US journals as well as journals of large-scale scholarly publishers consistently have been in the Q1 rankings. Based on the ICM evidence, the visibility of a journal will be negatively affected if it is not supported by large-scale scholarly publishers and it is not from European and US regions, even though it is written in English. The IF values of English-language journals published in non-European and non-US regions are still relatively low.

Within the four years from 2010 to 2013, there were, on average, five journals listed in Q1: the Journal of Ethnopharmacology, Phytomedicine, the American Journal of Chinese Medicine, Chinese Medicine, and Complementary Therapies in Medicine for 2013. Chinese medicine is getting greater and greater attention because two Chinese medical journals were listed in the Q1 in 2013. Meanwhile, two ethnopharmacology journals, both published by Elsevier and listed in three other categories of SCI simultaneously, have been listed in Q1 for many years, reflecting the fact that pharmacological journals receive higher exposure.

Journals may be listed in more than one subject category. It is worth noting that of the five Q1 journals in the ICM category in the 2013 JCR, the Journal of Ethnopharmacology and Phytomedicine were simultaneously listed in three other categories: “Chemistry, Medicinal”; “Pharmacology & Pharmacy”; and “Plant Sciences.” Also, the American Journal of Chinese Medicine is listed in “Medicine, General & Internal,” and Chinese Medicine-UK is listed in “Pharmacology & Pharmacy.” If a journal is concurrently listed in several categories belonging to a large field, this will have a conspicuous effect on its IF.

Open access (OA) journals have been accepted by WoS in recent years [12]. OA manuscripts also undergo a peer-review process, but the articles can be freely accessed only on journal websites. To ensure content quality, OA journals in the ICM category—such as BMC Complementary Medicine and the Journal of Traditional Chinese Medicine, which belong to professional societies—cooperate with Springer and Elsevier separately. Large-scale scholarly publishers are expanding their reach; for instance, 179 out of 500 Springer OA journals are listed by WoS and have an IF [13].

It is still arguable that OA is a strategy being used to boost IFs [14, 15]. Obviously, the effect of OA on journal citations varies in different fields [16]. However, the biggest challenge for an OA journal, at present, is that the market is saturated with “fake journals” [17]. Fake journals aim to collect a processing or publication fee from authors without conducting any peer review, which greatly diminishes the credibility of OA journals. Thus, cooperating with large-scale scholarly publishers is one of the approaches to make readers confident about articles.

SCImago, similar to SCI of WoS, is another popular portal that includes information like journal rank and citation analysis. If the journals in the SCI ICM category are compared with those in the SCImago CAM category <http://www.scimagojr.com>, the number of journals in the SCImago CAM category is not only greater, but also has more diversified countries of origin. Based on the 100 journals listed in SCImago for the CAM category in 2013, the ranking of the listed countries shows that Europe accounts for the largest proportion and the United States makes up the second largest proportion. As the United States is followed by China and India, the ranking at least reflects the uniqueness of ICM, in other words, that non-Western conventional medicine is a defining characteristic of this category. The total number of Q1 journals in the SCImago CAM category is 25. Regarding the publishers of Q1 journals in the SCImago CAM category, it is noted that over half of the Q1 journals are published by the 5 leading publishers collectively, and Elsevier and Springer are still the publishers with the most Q1 journals. As the total number of journals is rapidly increasing, the advantages enjoyed by these 5 publishers are apparently not as great as their share of journals in the SCI ICM category suggests. Nevertheless, European and US journals still have the lion’s share of Q1 journals: the US holds 7 titles, the United Kingdom holds 9 titles, and Germany holds 3 titles.

CONCLUSIONS

This examination looked at only a small sample from a representative database. Nevertheless, it can provide some insights into the impact evaluation of ICM (or CAM) journals and the related impact of their research. The most important consideration is not whether a journal is an OA journal, but whether a journal is published by a large-scale scholarly publisher. Today, the main problem with OA journals is that the market is flooded with fake journals, which prevents readers from distinguishing the real from the fake. Large-scale scholarly publishers can offer journal information to scholars through their management of databases and networks. Moreover, these publishers’ brand names make them attractive to potential authors as the first choice for their articles.

Biography

Ka-wai Fan, PhD, cikwfan@cityu.edu.hk, Associate Professor, Department of Chinese and History, City University of Hong Kong, Hong Kong, China