Major inaccuracies in articles citing occupational or environmental medicine papers and their implications

INTRODUCTION

Bibliographic citations or references are an important component of all scientific manuscripts, but sufficient attention is not always paid to them. In fact, more than fifteen years ago, de Lacey and colleagues reported an astonishingly high rate of reference errors in articles published in general medical journals [1], and some studies since then have confirmed this phenomenon in many different biomedical journals [2–4].

These studies selected samples of citations in various periodicals and checked their accuracy, in other words, whether or not the information found in the reference lists was correct. Currently, to our knowledge, no published data has described the percentage of inaccurate citations citing specific articles. The aim of this study was to assess the accuracy of references specifically citing manuscripts concerning occupational and environmental medicine, as well as to evaluate the pattern of the errors in citing papers and the implications of these errors in terms of article citation rate. The cited papers were those published in Occupational and Environmental Medicine (OEM), because this journal was one well-known, if not the best-known, journal in this discipline and had the highest impact factor (IF). The second reason was that the tables of contents and the abstracts were easily accessible from the journal's Website <http://oem.bmjjournals.com>.

MATERIAL AND METHODS

The ISI® Science Citation Index Expanded (SCIE) (Thomson ISI, Philadelphia, Pennsylvania) was used to search for all the published articles that cited one of the articles published in OEM in 1994 or 1998 in their references. The SCIE provided access to current and retrospective bibliographic information, author abstracts, and cited references in more than 6,000 journals. This database was searched using OEM as the cited work and the years 1994 or 1998 as the cited year. These years were chosen to take into account the long half-life of occupational and environmental health literature: The cited half-life of OEM was 4.8 years, and the cited half-lives of other periodicals in this field, the Journal of Occupational Medicine or the Scandinavian Journal of Work Environment and Health, were 8.2 and 8.4 years, respectively [5].

The accuracy of all the references recovered was controlled by comparing each reference with the article published in the printed version of OEM. When a reference was inaccurate, the Websites of OEM and Pubmed <http://www.ncbi.nlm.nih.gov/entrez/query.fcgi> were used to search for the published article that best matched the reference and to identify the type of error in the citation. Only major citation errors were assessed (i.e., errors that involved the data elements by which references are identified in SCIE): the first author's surname and initials, the year of publication, the title of the journal, the journal volume number or supplement designation, and the initial page number.

RESULTS

A total of 314 articles were published in OEM during the two years studied, consisting mainly (80%) of original papers.

The number of citations of an OEM publication from 1994 or 1998 that were recovered from the SCIE was 3,347. One hundred and twelve (3.35%) citations were inaccurate, with at least 1 major error. The most common errors concerned the name of the first author and the first page number of the article (Table 1).

Table 1 Types of errors in the citations

On the whole, these citations included 122 errors, because 9 citations included more than 1 major error. Eight citations included 2 errors in each one, and one citation included 3 errors (wrong year, volume, and first page number). Furthermore, 11 incorrect citations were found in duplicate or in triplicate. The same error in the author's name, first page number, volume, year, or journal was repeated in 2 or 3 citations originating from different citing papers.

Searching SCIE identified 3,347 articles citing 387 different 1994 or 1998 OEM articles. Different articles were defined as articles with different first author, volume, page number, or year. Each article was cited between 1 and 41 times.

Among the 387 cited articles, 99 articles did not exist; the citations did not permit us to identify the articles in the 1994 or 1998 volumes of OEM. Therefore, only 288 articles were correctly cited and in fact existed (i.e., had really been published in OEM in 1994 or 1998). Among the 99 ghost articles, the citations were inaccurate in 80 cases but concerned, in fact, articles that had been published in OEM in 1994 or 1998 and that had been correctly cited elsewhere. An editorial* had been cited only once, inaccurately. Finally, 18 articles cited as articles published in OEM in 1994 or 1998 had, in fact, been published in OEM but in another year (14 cases), had been published in another journal (2 cases), or did not exist at all (one case).

Lastly, among the 314 articles published in OEM in 1994 or 1998, 288 (92.04%) were cited at least one time in one of the 6,000 journals covered by the SCIE database. That means that 25 articles (7.96%), in fact, had never been cited.

DISCUSSION

The SCIE is the best-known method in the health sciences to identify citing papers of selected articles. It certainly underestimates the real number of citations, because the SCIE database covers “only” 6,083 journals. Nevertheless, nearly all the major journals (36 of 38) specializing in occupational health, which were previously identified [6], were indexed in SCIE; only Medicina del Lavoro and the International Journal of Occupational and Environmental Health were not. Therefore, the majority of the articles citing manuscripts originating from OEM were probably recovered by this study.

Only errors concerning the first author's surname and initials, the year of publication, the title of the journal, the journal volume number or supplement designation, and the initial page number were included, because errors in these fields can seriously hinder retrieval of the cited article, whereas errors concerning the other authors or the title of the manuscript would not affect the retrieval of the full article.

Papers dealing with accuracy of references usually included all types of errors, especially in the article title, which was the main source of mistakes [7], and the high rate of reference errors in articles found in some studies were not easily compared to those presented in this study. Nevertheless, Fenton et al. [2] also divided errors into minor, intermediate, and major. They found a rate of major errors of 11.90% but observed that the higher the IF for the journal, the lower the number of errors detected in its papers. The highest journal IF in their study was 1.118, whereas the IF of OEM was 1.958 in 1999. Therefore, the rate of 3.35% remained unacceptably high, especially considering that 0.27% of citations included more than 1 major error.

The onus should be on the authors only to quote references that they have, in fact, read to ensure the accuracy of the bibliography. Nevertheless, some citations are based solely on reading the abstract and, even more problematic, on the simple reproduction of a citation found in another article. In this study, the fact that some citation errors are found in duplicate or triplicate, originating from different citing articles, clearly indicates that the “copy and paste” function is sometimes used improperly. Furthermore, reading the abstract instead of the full article is demonstrated, for example, by the analysis of citations quoting the article by Stucker and colleagues.† Two independent citations reproduced the error in the author's name found in MEDLINE (Strucker instead of Stucker). This unscientific approach should be discouraged, because it can lead to erroneous interpretation of the real conclusions of a scientific study.

Because journal IFs are readily available, it has been tempting to use them for evaluating individual scientists or research groups, but the dramatic shortcomings of such an approach have been highlighted [8]. Garfield, creator of the journal IF, has therefore proposed relying on the actual citation counts for individual articles and authors when trying to evaluate a person's publication list [9]. However, apart from incomplete retrieval of information for practice and research, citation errors can result in authors not receiving credit for their publications and thus bias this citation count.

A good example is provided in the article by De Zotti and colleagues.‡ Looking at the results provided by SCIE, this article appeared to have been correctly cited 41 times (i.e., cited with the right first author, journal, year, volume, and first page number). Nevertheless, it was also cited once with a wrong first page number (552 instead of 548) and five times with an error in the name of the first author (twice as Larese F, who was the second author, twice as Zotti R, and once as De Zeotti R). The correct number of citations was therefore 47 and not 41, an underestimation by 15% of its real “citation score.”

Another striking example is the article by Ilg and colleagues,§ which had been correctly cited 8 times. It was, in fact, cited 4 more times, twice with an error in the first author's name and twice of the first page number. The correct citations score of this paper was, therefore, 12. Searching only for the name and initials of the first author would have retrieved only 10 citations, leading to an underestimation of the citation score for this author and this paper of up to 25%. Such underestimation of the citation score can be very prejudicial to individuals and institutions, because this score is becoming a key indicator that not only influences academic advancement but may also have an effect on an individual's or an institution's chances of attracting research funding [10].

CONCLUSION

This study demonstrates inaccuracies in references in 3.35% of the 3,347 papers, originating from different periodicals and citing articles concerning occupational or environmental medicine published in the leading journal in this field. Poor reference accuracy seems, therefore, to be a problem in occupational and environmental health literature. This phenomenon, which hinders the ability of readers to access the article and thus eventually to quote it and which directly affects the assessment of single authors' citation rates, should be taken into consideration by authors, reviewers, and editors. These results warrant efforts from all stakeholders to reduce these inaccuracy rates as much as possible.