Questionable papers in citation databases as an issue for literature review

Abstract

In recent years, the academic world has been faced with much academic misconduct. Examples involve plagiarizing papers, manipulating data, and launching predatory or hijacked journals. The literature exposing these activities is growing exponentially, and so is the presentation of criteria or guidelines for counteracting the problem. Most of the research is focused on predatory or hijacked journal detection and providing suitable warnings. Overlooked in all this is the fact that papers published in these journals are questionable, but nevertheless show up in standard citation databases. We need some way to flag them so future researchers will be aware of their questionable nature and prevent their use in literature review.

Introduction

Nowadays, academic search engines and citation databases such as Google Scholar, Scopus, and Thomson Reuters Web of Science (WoS) are popular among researchers who hope to find suitable references for their research efforts by doing a literature review (Osipov et al. 2014). Scopus and WoS are accessible through payment or subscription, but Google Scholar is free for all researchers, so it is a good candidate for an initial literature search. Scopus and WoS only cover selected peer-reviewed journals, but Google Scholar also includes nontraditional sources such as Webpages and self-published articles (Kulkarni et al. 2009). In other words, WoS and Scopus use defined criteria for selecting journals to cover their contents (De Moya-Anegón et al. 2007), but Google Scholar does not (Harzing and Van der Wal 2008). Google merely uses its usual web technologies to find what appears to be academic content.

As most researchers use academic search engines and citation databases to find papers on prior work to complete their literature review, it is important that such databases contain adequately peer-reviewed content. Otherwise, researchers might establish their new research based on bogus and manipulated papers. In support, they may inadvertently use references based on invalid “facts.” Scientific databases should include papers with varying, even opposed interpretations, but the data used for those interpretations should be beyond reproach.

Questionable papers in predatory and hijacked journals

Usually, we consider papers published in predatory and hijacked journals as questionable. Predatory journals were first uncovered by Jeffrey Beall, a librarian at the University of Colorado, Denver, in 2009. He archived each suspicious “call for papers” email sent to him, took a picture of each journal website pitching the call, and then coined the term “predatory publisher” in 2010. His first list had 20 publishers that he considered to be predatory. He eventually created a website with two continually updated lists, one for predatory publishers, and another for standalone predatory journals (Beall 2013). Predatory journals have a primary distinguishing characteristic: they only exist to make money. They do not have adequate peer review and will publish almost anything—for a price. They typically exploit the open access model that is now becoming popular among legitimate publishers, but do not have the quality and validation to back it up (Bhad and Hazari 2015; Nahai 2015). In response, investigators, following Beall, have highlighted their negative effect on science and have presented guidelines to improve the awareness of prospective authors (Hansoti et al. 2016; Tin et al. 2014; Beall 2016).

Hijacked journals are another problem for scholarly publishing. Hijacked journals are websites that mimic authentic journals by using similar names and ISSN numbers (Jalalian and Dadkhah 2015; Dadkhah and Borchardt 2016). Like predatory journals, these typically exploit the open access model—again, for a price. These journals do not have peer review. The first hijacked journal was created on August 11, 2011. Cybercriminals registered the domain “sciencerecord.com” and launched three hijacked journals including: “Science Series Data Report,” “Innova Ciencia,” and “Science and Nature” (Jalalian and Mahboobi 2014). Mehrdad Jalalian is the first researcher to detect systematic journal hijacking and presented the term “hijacked journal” to the academic world in early 2012 (Jalalian and Dadkhah 2015). Nevertheless, our observations show that there is not sufficient research about hijacked journals in comparison to predatory journals. When we search for the term “hijacked journal” in Google Scholar, 38 results are reachable, but the term “predatory journal” yields 161 results (searched on 15 September 2016).

The newest twist in academic publishing involves computer generated papers and do not present real knowledge. SCIgen (http://pdos.csail.mit.edu/scigen/#generate) is a tool developed by the Massachusetts Institute of Technology to test peer review at low quality conferences. It has been used by some researchers to create fake papers in the attempt to publish them in conferences and academic journals (Bohannon 2015a, b). Bohanon found that in 2013, 85 fake papers were accepted at 24 different conferences (Bohannon 2015a). Some legitimate journals also do not provide essential peer review, with fake papers having a good chance of being published. As a test, Bohanon sent a fake paper to 304 selected journals. It was accepted by 157 of them (Bohannon 2013).

These questionable papers may be used by researchers for doing literature review or designing hypotheses. It is clear that using such papers will lead to unreliable and invalid science. Researchers need to know the current situation in their disciplines. They do not need to follow bogus information.

Existence of questionable papers in citation databases

As mentioned, citation databases must not include questionable papers. In particular, they should not include papers published by predatory or hijacked journals. Usually citation databases and academic search engines use advanced techniques to select content (such as journals and proceedings), but our observations show that many questionable papers and journals are included nonetheless.

Predatory journals and citation databases

Google Scholar’s aim is to cover comprehensive journals regardless of their quality and makes no effort to exclude papers published in predatory journals (Beall 2016). The publishers of most predatory journals seem to have no qualms about announcing their being indexed in Google Scholar. Thus, Google Scholar is worthless for detecting questionable papers published in predatory journals. As mentioned, WoS and Scopus use special criteria to select high quality journals for indexing (De Moya-Anegón et al. 2007), although some predatory journals slip through. For example, we searched a custom query in Scopus (Appendix 1) to find the top journals under the phrase “most published papers in Asia.” The fifteenth journal is the International Journal of Applied Engineering Research with 11,002 papers that were indexed between 2010 and 2016. Beall and some others believe that it has a predatory publisher (https://scholarlyoa.com/publishers) (Dyrud 2014), so Scopus includes papers published in predatory journals too. Recently, Beall wrote about the Thomson Reuters master list and announced that this list also contains many predatory journals (https://scholarlyoa.com/2016/04/28/the-tr-master-journal-list-is-not-a-journal-whitelist/#more-7354). As mentioned in a previous section, Beall believes that predatory journals have flawed peer review and are a source of questionable papers. They should not be included in citation databases of reviewed journals.

Hijacked journals and citation databases

As mentioned before, hijacked journals are fake websites made to look like those of reputable journals. As a result, papers published in association with these websites should not be used by researchers to conduct new research. Normally, citation databases do not index papers published in hijacked journals, but cybercriminals have found ways to include their hijacked journals’ papers in academic search engines and citation databases.

As a test, we used Publish or Perish tools (http://www.harzing.com/resources/publish-or-perish) to uncover some papers published in hijacked journals and subsequently indexed in Google Scholar. We used Beall’s updated list of hijacked journals (https://scholarlyoa.com/other-pages/hijacked-journals/) to compile Table 1.

Table 1.

Number of indexed papers for each hijacked journal in Google Scholar (2014–2016)

No.	Journal name	Number of indexed papers for each hijacked journal
1	Academie royale des sciences d outre-mer bulletin des seances	46
	Authentic URL: http://www.kaowarsom.be/fr/bulletins
	Hijacked URL: http://ardsj.com
2	Ayer	218
	Authentic URL: http://www.ahistcon.org/revistaayer.html
	Hijacked URL: http://ayeronline.com
3	Bulletin of the Georgian National Academy of Sciences	3
	Authentic URL: http://www.science.org.ge/moambe/moambe.html
	Hijacked URL: http://www.bnas.org
4	JNCC REPORT	21
	Authentic URL: http://jncc.defra.gov.uk/page-2132
	Hijacked URL: http://jncc.info/ojs-2.4.6/index.php/jncc/about
5	MAGNT Research Report	20
	Authentic URL: http://www.worldcat.org/oclc/223764115
	Hijacked URL: http://brisjast.com/
6	MULTITEMAS	181
	Authentic URL:
	http://site.ucdb.br/editora-ucdb/tematica/37/periodicos/multitemas-numero-2-dez-1996/rmt/
	Hijacked URL: http://multitemas.com/index.php/MTJournal
7	Walia Journal	16
	Authentic URL: http://www.worldcat.org/oclc/183351204
	Hijacked URL: http://www.waliaj.com/

We expect that popular citation databases such as Scopus and Thomson Reuters Web of Science will avoid indexing hijacked journals in the future. Nonetheless, we suspect that cybercriminals eventually will find a way to hack these scientific databases too. In November 2015, Bohannon (2015b) published a paper in Science disclosing that cybercriminals had hijacked 20 journals. These hijacked journals are accessible in Thomson Reuters master list along with their hijacked URLs. Researchers visiting the Thomson Reuters master list were victimized when they inadvertently entered the URLs of these hijacked journals instead of the URLs of the authentic journals. Again, we must warn that citation databases and academic search engines may include questionable papers from hijacked as well as predatory journals.

Fake papers and citation databases

As mentioned, computer-generated and other kinds of fake papers have been published in academic proceedings and journals. Most of these were nonsensical, but got through the submission process due to low quality or nonexistent review.

We planned a small experiment by using SCIgen to generate 10 fake papers. Then, following Google Scholar instructions for indexing academic papers (https://scholar.google.com/intl/en/scholar/inclusion.html), we were able to index our own fake papers to be included in Google’s search results. Others did a similar experiment by creating fake citations for selected papers (Lopez-Cozar et al. 2012). The main problem is this: Google crawls web contents and looks for PDF files that seem to be academic documents. According to Google’s instructions for indexing a document, each PDF file that is linked to a web page with a title, author name, and references, is a candidate for indexing in Google Scholar.

Some general guidelines for authors

As mentioned in previous sections, many questionable papers are indexed in citation databases and academic search engines. Obviously, authors should not cite questionable papers in their research. The use of such citations tends to invalidate their own research. Authors must use only legitimate journals in their work. By using Beall’s lists for predatory and hijacked journals, researchers will be able to detect the most questionable papers. In the future, we hope that citation databases will flag questionable papers. Figure 1 shows the conceptual image of such a citation database. In this figure, the first paper has been retracted, so it easily classifies as a questionable paper. The other results come from reputable journals that are not predatory or have not been hijacked. A “trusted sign” has been shown for them. At present, citation databases only flag retracted papers that have flaws in terms of reliability.

Fig. 1.

Conceptual Google Scholar search showing special flags for questionable and authentic papers

Conclusion

Nowadays, scholarly publishing is faced with much misconduct leading to questionable papers throughout the academic literature. In this paper, we reveal the most common types of questionable papers and show that citation databases and academic search engines are not immune to including them in their indices. Researchers can avoid their use of such bogus references by checking lists of known predatory and hijacked journals. Also, a new duty now arises for citation databases and academic search engines: they need to find solutions for detecting questionable papers, thus preventing their spread throughout academic literature without their true nature being known to prospective users. In some research methods, such as meta-analysis and meta-synthesis, the qualityof selected sources is important. Research that exposes fraudulent papers obviously is critical for such assessment.

Appendix 1.

Custom searched query in Scopus for finding top journals by using the phrase “most published papers in Asia”

(AFFILCOUNTRY(Afghanistan) OR AFFILCOUNTRY(Armenia) OR AFFILCOUNTRY(Azerbaijan) OR AFFILCOUNTRY(Bahrain) OR AFFILCOUNTRY(Bangladesh) OR AFFILCOUNTRY(Bhutan) OR AFFILCOUNTRY(Brunei) OR AFFILCOUNTRY(Myanmar) OR AFFILCOUNTRY(Cambodia) OR AFFILCOUNTRY(China) OR AFFILCOUNTRY(India) OR AFFILCOUNTRY(Indonesia) OR AFFILCOUNTRY(Iran) OR AFFILCOUNTRY(Iraq) OR AFFILCOUNTRY(Japan) OR AFFILCOUNTRY(Jordan) OR AFFILCOUNTRY(Kazakhstan) OR AFFILCOUNTRY(North Korea) OR AFFILCOUNTRY(South Korea) OR AFFILCOUNTRY(Kuwait) OR AFFILCOUNTRY(Kyrgyzstan) OR AFFILCOUNTRY(Laos) OR AFFILCOUNTRY(Lebanon) OR AFFILCOUNTRY(Malaysia) OR AFFILCOUNTRY(Maldives) OR AFFILCOUNTRY(Mongolia) OR AFFILCOUNTRY(Nepal) OR AFFILCOUNTRY(Oman) OR AFFILCOUNTRY(Pakistan) OR AFFILCOUNTRY(Philippines) OR AFFILCOUNTRY(Qatar) OR AFFILCOUNTRY(Saudi Arabia) OR AFFILCOUNTRY(Singapore) OR AFFILCOUNTRY(Sri Lanka) OR AFFILCOUNTRY(Syria) OR AFFILCOUNTRY(Taiwan) OR AFFILCOUNTRY(Tajikistan) OR AFFILCOUNTRY(Thailand) OR AFFILCOUNTRY(Timor-Leste) OR AFFILCOUNTRY(Turkmenistan) OR AFFILCOUNTRY(United Arab Emirates) OR AFFILCOUNTRY(Uzbekistan) OR AFFILCOUNTRY(Vietnam) OR AFFILCOUNTRY(Yemen) OR AFFILCOUNTRY(Israel)) AND PUBYEAR AFT 2010.