Skip to main content
Springer logoLink to Springer
. 2015 Mar 6;22(1):199–225. doi: 10.1007/s11948-015-9638-0

Evaluation by Citation: Trends in Publication Behavior, Evaluation Criteria, and the Strive for High Impact Publications

Maarten van Wesel 1,2,
PMCID: PMC4750571  PMID: 25742806

Abstract

Criteria for the evaluation of most scholars’ work have recently received wider attention due to high-profile cases of scientific misconduct which are perceived to be linked to these criteria. However, in the competition for career advancement and funding opportunities almost all scholars are subjected to the same criteria. Therefore these evaluation criteria act as ‘switchmen’, determining the tracks along which scholarly work is pushed by the dynamic interplay of interests of both scholars and their institutions. Currently one of the most important criteria is the impact of publications. In this research, the extent to which publish or perish, a long standing evaluation criterion, led to scientific misconduct is examined briefly. After this the strive for high impact publications will be examined, firstly by identifying the period in which this became an important evaluation criterion, secondly by looking at variables contributing to the impact of scholarly papers by means of a non-structured literature study, and lastly by combining these data into a quantitative analysis.

Keywords: Publish or perish, High impact, Research evaluation, Publication practices

Introduction

Not ideas, but material and ideal interests, directly govern men’s conduct. Yet very frequently the ‘world images’ that have been created by ‘ideas’ have, like switchmen, determined the tracks along which action has been pushed by the dynamics of interest. ‘From what’ and ‘for what’ one wished to be redeemed and, let us not forget, ‘could be’ redeemed, depended upon one’s image of the world. (Weber 1970b)

Scientific misconduct has been increasing, however until recently awareness of these practices appears to be limited (Regmi 2011).1 Recent cases of scientific misconduct, such as the Stapel affaire (fabrication of data, see Levelt Committee et al. 2012), and the Karl-Theodor zu Guttenberg plagiarism affair, have created some awareness amongst both scholars and the general public. This sparked some national or field-specific movements such as Science in Transition in the Netherlands (Dijstelbloem et al. 2013) and the American Society of Cell Biology’s Declaration on Research Assessment (Moustafa 2014). These movements perceive a link between the system of evaluation of science and cases of scientific misconduct, such as some of the extreme cases mentioned above.

Reflecting on the switchmen metaphor put forth by Weber in his study of religion and its influence on economic systems, we can begin to see that what is deemed as important in science steers the behavior of scholars in a certain direction. Two crucial ideas about what is important in science are currently at work: the idea that publishing more is better and the idea that a journal, paper, scholar and institution should have a high Impact Factor.

From what and for what does a scholar wish to be redeemed, and can a scholar be redeemed? In most western countries a publication track record is necessary for obtaining tenure and it is hard to secure funds for research without one (Regmi 2011). Especially peer-reviewed publications are important for the career of scholars at all stages of employment (Elliott 2013; Moustafa 2014). Someone who only publishes work with a low impact factor will have difficulty obtaining tenure and funds. Thus both the number of publications as well as their impact are crucial for the career of a scholar, redeeming him/her from joblessness, or at least careerlessness, if their publication record is better than that of their peers. Because when there is stiff competition for positions, funding, and other academic rewards, those with slightly greater achievements will reap in a disproportionately larger share of the rewards (Anderson et al. 2007).

The ideas about what is important in science are important themselves. Scholars are constantly being reminded that these ideas are important, thus these ideas determine the tracks via which action has been pushed by the interest of scholars. This is coupled with a rise of careerism among scientists, in which, sometimes, the shortest routes to success are taken, including fraudulent behavior (Kumar 2008) or cutting a few corners (Anderson et al. 2007). Competition between scientists increases the chance of scientific misconduct (Anderson et al. 2007).

First, the literature on publish or perish will be examined as it has already been established how this idea has shaped scholarly publishing behavior, and contributed to unethical behavior. Secondly the idea of publishing for high impact will be studied, as this is less well researched. Here the focus will not be on unethical behavior, but rather trends in publishing behavior will be examined to establish the link between a paradigmatic shift in what is important and publishing behavior. The period in which high impact publications became a criteria for evaluation will have to be identified, after which variables contributing to the impact of papers will be identified by means of a small literature study. These variables will be used to conduct a quantitative analysis of how these factors have changed over the years.

Publish or Perish

Scholarly publication rates, Errami and Garner (2008) claim, are at an all-time high. This is not caused by an increase in productivity, but rather by changes in the way scholars publish (Broad 1981), which is linked to the pressure to publish (Errami and Garner 2008). A 1981 commentary in Science reported that co-authorship and multiple publication of the same data were on the rise, whilst the length of papers was decreasing (Broad 1981). The increase in co-authorship is attributable to interdisciplinary papers, multi-institutional clinical trials, but also to gratuitous listing of co-authors (Broad 1981). Gift authorship (Matías-Guiu and García-Ramos 2010), for instance including the head of a department or lab, is a common practice in some disciplines as is adding other researchers out of courtesy (Broad 1981), or an expectation of reciprocity (Webster et al. 2009).

In addition to a rise in co-authorship, a decrease in paper length was already noted in the early 1980s (Broad 1981). Scholars prefer to publish four short papers instead of one long paper (Broad 1981). They slice their data as if it were a salami, hence the term salami-slicing is sometimes used. The terms Least Publishable Unit (LPU) (Broad 1981), or Smallest Publishable Unit (SPU) (Elliott 2013) are used to describe these papers that contain the minimum amount of information needed to get published.

Another trend, closely related to salami slicing, is that of duplicate, or multiple, publication publishing articles that overlap substantially (Andreescu 2013; Kumar 2008; von Elm et al. 2004). This could be a simple copy (with the same authors, same data, basically same content, maybe a different title), a salami-sliced article without cross references to other articles based on the same data, a meat extender also called data augmentation (which is an expansion of an existing article with more data, sometimes without cross-reference), salami-sliced articles published by different authors (most common in multicenter trials), and a textual copy of an article with a different dataset and, possibly, different results and/or conclusion (Kumar 2008; von Elm et al. 2004).

Using text comparison software followed by manual verification, Errami and Garner (2008) uncovered a growing trend of duplicate publications in the biomedical literature; just below 2 per 1000 in 1975 to just below 10 in 1000 in 2005. Whilst this quintupled number still only represents 1 % of the papers, it is worrying since duplication represents just one possible mode of scientific misconduct.

The importance of having many publications is now declining in favor of the impact of publications (Franco 2013), although there are still scientists evaluated solely on the number of publications (Anderson et al. 2007). Impact is the focus of the next section.

High Impact Publications

The idea that the importance of a publication can be judged from the number of references it receives is not recent. Even before Garfield (2006) first published about a Citation Index for Science, in 1955, the idea already existed, as he himself readily acknowledges. Early in the twentieth century, Gross and Gross (1927) postulated that the number of references a journal receives from a set of representative journals suggests something about its importance to the field, aiding librarians in choosing journals to add to their collections.

“The impact factor” states Moustafa, “became a major detrimental factor of quality, creating huge pressures on authors, editors, stakeholders and funders” (2014). But when did the impact of a single scholar, as measured by the citations (s)he receives, become important? This seems hard to pin point. In 1990, Tsafrir and Reis state “administrators are turning more to the citation performance of individuals” (1990) suggesting an increase in importance in or shortly prior to 1990, at least for Medicine. But it seems to have started earlier, in 1975 Wade provides cases where scholars’ citation counts were used for tenure and funding decisions, but it was by no means commonly used as an assessment tool at that time (Wade 1975). If indeed the idea about the importance of being cited influences scholars, consciously or unconsciously, we would expect this to be reflected in their work, starting between 1975 and 1990, in at least some scientific fields.

Recent research, discussed below, has examined the characteristics of papers, such as their writing style, which have an influence on subsequent citations. Whilst we should look at factors influencing subsequent citations in papers published in the period that we are interested into truly understand what was relevant then, the factors identified in current research offer some insights. These factors are expected to differ between the period before 1975 and the period after 1990, as the transition by then has already started.

The number of references a paper contains has been found to be positively correlated with the number of times a paper is cited, and this holds for all fields researched (Vieira and Gomes 2010; Webster et al. 2009; Wesel et al. 2014). Having many references can be useful to defend a paper against attacks (Latour 1987). Whilst references should be relevant to the paper, their numbers could be inflated by simply copying references from other papers (Ramos et al. 2012) or via a process of I cite you, you cite me in a form of reciprocal altruism (Webster et al. 2009).

The number of authors contributing to a paper is also a stable positive influencer across fields (Frenken et al. 2005; Glänzel and Thijs 2004; Levitt and Thelwall 2009; Vieira and Gomes 2010; Webster et al. 2009; Wesel et al. 2014). The rise of multi-authored papers, already observed by de Solla Price (1963) in the early 1960s, is often thought of as resulting from a rise in multi-disciplinary research. However other explanations for this rise are gratuitous listing of co-authors and gift-authorship, already mentioned above in the context of publish or perish. In the context of high-impact publication the naming of extra authors not only helps these authors gather extra publications, but could also help the paper to become highly cited, by extending the network to which the paper can easily be introduced (Frenken et al. 2005). Especially when eminent co-authors are named this has an even greater effect on the number of times a paper is cited (Haslam et al. 2008).

Another factor which, in most fields, correlates positively with the times an article is cited is its total length (Haslam et al. 2008; Hudson 2007; Vieira and Gomes 2010; Wang et al. 2012; Wesel et al. 2014), although this does not seem to hold in Applied Physics (Wesel et al. 2014). Notice, this seems to contrast with a trend observed for publish or perish which stimulates short, sliced, papers. Some suggests that lengthening is done to meet a presumed standard (Andreescu 2013).

Other interesting factors include the presence of a colon in the title and the length of the title (Haslam et al. 2008; Jacques and Sebire 2010). The direction of the effect of title length seems to differ across fields. In Sociology, Applied Physics, and a sub-set of PLoS journals a shorter title is associated with more citations (Jamali and Nikzad 2011; Wesel et al. 2014). Whilst in General and Internal Medicine the effect is reversed (Wesel et al. 2014).

The readability of abstracts also influences the number of citations an article receives, at least in Applied Physics and General and Internal Medicine (Wesel et al. 2014). A less readable than average abstract, as measured by the Flesch Reading Ease Score (Flesch 1948), has a positive effect on the number of citations an article receives. More sentences in the abstract is also related to more frequent citation in Sociology, Applied Physics, and General and Internal Medicine (Wesel et al. 2014).

The mechanism by which these factors are understood to influence the number of incoming citations is not relevant for this work (for exploration see, for instance, Wesel et al. 2014). What does matter is if the utilization of tricks that increase the number of received citations is increasing. These tricks do not necessarily represent scientific misconduct, although artificially inflating the author count, adding unnecessary references, and purposely making the abstract hard to read clearly can be considered misconduct. Depending on the circumstances this could also be said for lengthening a paper, if this lengthening occurs without adding new, relevant, information, this could be seen as misconduct.

Historically some of these, or related, factors have been shown to be stable whilst others are known to have changed. According to Gross et al. (2002) the number of citations per 100 words has risen from 0.3 in the period 1901–1925 to 1.8 in the period 1976–1995. This rise has been quite steep, in the period 1926–1950 there were 0.8 citations per 100 words and 1.5 in the period 1951–1975. The number of references quoted in articles was quite stable over a long period, in 1955 Garfield calculated an average of ten (Garfield 2006), and in the early ‘60s de Solla Price (1963) gives just under ten as the norm, stating it has been stable for many years.

Reproducing of Practices

Scholars who have traits enabling them to produce more and higher cited papers than another scholar in the same field are more likely to secure resources, e.g. career, funding, PhD candidates and the like (Anderson et al. 2007). Since the relationship between a professor and a Ph.D. candidate is a socialization process, many Ph.D. candidates are influenced by the publishing style of their professors. Thus they pick up on traits about what constitutes good scholarly conduct and what constitutes misconduct. Furthermore, productive scholars will be read more, and are thus more likely to influence their readers with their style and approach to citation. Scholars, at all moments in their career but especially if they are new to the field, are further socialized by what they read, what they see, and what they hear from their peers and especially from those who are seen as successful.

Thus scholarly (mis)conduct is reproduced via a form of sociocultural evolution. The selection mechanism (Nolan and Lenski 2006) is evident, as described in the above paragraph. In other words; “selection theory takes the following from: when interactors interact, replicators create lineages by a process of selection” (Gross et al. 2002).

As such conducts becomes more widespread, scholars have come to see these practices as the norm, and as the accepted way to conduct science. As Elliott suggested when discussing salami-slicing “there is no intentional deceit taking place, just an assumption that this practice is perfectly acceptable” (2013).

Expected Results

Following the discussion above one would expect to observe the following:

  • A decrease in the length of the paper title, in most fields

  • A rise in the number of authors contributing to a paper

  • An increase in paper length

  • An increase in the number of sentences in the abstract

  • Most likely a decrease in readability of the abstract until it reaches an optimum

  • A rise in the number of references a paper contains

  • And an increase in paper titles with a (semi-)colon

Given the generational effect described above we would aspect these changes to accelerate, at least until reaching an optimum or plateau level.

Methodology

To select representative journals, 50 journals with the highest Impact Factor for the years 1997 and 2012 from Thomson Reuters Journal Citation Reports (JCR) Science and Social Science edition were compared to identify journals which have been influential for many years.2 There was an overlap of 18 journals in the JCR Science Edition and 20 journals in the JCR Social Science Edition. For these journals the availability of data in Thomson Reuters Web of Knowledge was checked, as data was required from 1960 till 2004 in order to create three 15 year periods (1960–1974, 1975–1989, and 1990–2004) of which the first and third can be compared. Eight journals in the JCR Science Edition and four journals in the JCR Social Science Edition met this criterion.

Information about the papers which appeared in these journals was downloaded from the Web of Knowledge (WoK). WoK data provided information on the publication year, the title, the authors, the DOI. From the title, the length in the number of words, and presence of a (semi-)colon were recorded. From the list of authors, the number of authors was counted, by counting the separating semi-colons and adding 1, for papers with an anonymous author the author count field was left blank. Data on the number of references contained in the paper were also extracted from WoK, however this data was unavailable for papers published before 1988,3 and thus this variable was not analyzed. Using CrossRef4 the DOI was translated to the URL of the papers at the publisher’s website. When the DOI was missing, the article name, journal, and year were used to query CrossRef for the DOI, which was only accepted if the first author was listed and the match had a 100 % score. From the publishers website the abstract and type of paper were acquired, as well as the start and end page, as there was incongruity between publisher and WoK data. For Chemical Reviews and Pharmacological Reviews it proved not possible to obtain information about the paper type, thus these journals were removed from the sample.

HTML codes5 were removed from the abstract when necessary. Using the built-in readability function in Microsoft Word 2010 the Flesch Reading Ease was calculated. The formula used by Word (Microsoft 2007) for this is as follows;

FleschReadingEaseScore=206.835-1.015×TotalWords/TotalSentences-84.6×TotalSyllables/TotalWords

The Flesch Reading Ease Score (FRES) is a readability scale in which a higher score indicates easier readability, for all practical considerations the scale can be thought of as ranging from 0 to 100, where a score from 0 to 30 indicates very difficult and a score from 90 to 100 very easy.

Three rough categories of paper types were deemed suitable for analysis; Articles (review and original), Letters, and short scientific communications. This leads to the fifteen journal paper type combinations shown in Table 1. Differences in naming had to be resolved, for instance Correspondence and Letters to the Editor in Lancet and in Nature were combined for their respective journals.

Table 1.

Number of papers in the dataset

Total 1960–1974 1975–1989 1990–2004

American Psychologist; Comment and Reply

JCR SSE: Psychology, multidisciplinary

1099 92 181 826

American Psychologist; Journal Article

JCR SSE: Psychology, multidisciplinary

3144 863 868 1413

Annual Review of Biochemistry; review-article

JCR SE: Biochemistry and molecular biology

952 351 321 280

Annual Review of Physiology; review-article

JCR SE: Physiology

890 258 333 299

Annual Review of Psychology; review-articlea

JCR SSE: Psychology, multidisciplinary

517 162 116 239

Lancet; articleb

JCR SE: Medicine, general and internal

6213 3777 0 2436

Lancet; hypothesisc

JCR SE: Medicine, general and internal

690 256 319 115

Lancet; letters to the editor

JCR SE: Medicine, general and internal

55,703 10,852 22,421 22,430

Nature; article

JCR SE: Multidisciplinary sciences

7522 4859 1481 1182

Nature; letters to editor and correspondence

JCR SE: Multidisciplinary sciences

50,450 18,812 13,389 18,249

Physiological Reviews; article

JCR SE: Physiology

1019 248 311 460

Psychological Bulletin; journal article

JCR SSE: Psychology, multidisciplinary

1795 620 547 628

Psychological Review; journal article

JCR SSE: Psychology, multidisciplinary

1021 392 174 455

Science; lettersd

JCR SE: Multidisciplinary sciences

3070 1400 752 918

Science; report

JCR SE: Multidisciplinary sciences

32,475 11,857 9882 10,736
Total 166,560 54,799 51,095 60,666

aSome years with zero papers, most likely due to a low number of papers overall

bFor the period 1972–1990 there were no papers identified as article, this is due to how Science Direct displayed paper identification for part of the papers

cSome years with zero papers, are most likely due to a low number of papers overall

dOne paper in 1997 and zero in 1998–1999, reason unknown

These variables were compared using an independent-samples t-test grouping the papers in the period 1960–1974 and 1990–2004. The presence of a (semi-)colon in the title was compared using the Chi square test. Effect size was calculated using the r = sqrt((t2/(t2 + df)) formula for the t test, resulting in only positive effect sizes. And ϕ = sqrt(x 2/n) for the Chi square test, also resulted in only positive effect sizes.

Results

Paper titles, measured by the number of words, are longer in the second period, 1990–2004, for fourteen of the fifteen sets, only Letters to editor and correspondence in Nature show a reduction in title length (see Table 8). An independent-samples T test determines these differences are significant (p < 0.05), even for this before mentioned outlier (Table 2). The effect sizes for the different sets vary from small to large, the effect size is smallest for Nature; letters to the editor and correspondence and largest for Annual Review of Psychology; review-article (Table 8). Fourteen out of fifteen sets behave contrary to the prediction.

Table 8.

Statistics for title length

N Mean SD Minimal Maximal r
American Psychologist; Comment and Reply
 1960–1974 92 5.32 2.94 1.00 19.00 .13
 1990–2004 826 6.65 2.96 1.00 24.00
American Psychologist; Journal Article
 1960–1974 863 7.95 3.93 1.00 36.00 .18
 1990–2004 1413 9.34 4.26 1.00 33.00
Annual Review of Biochemistry; review-article
 1960–1974 351 4.23 2.39 1.00 14.00 .44
 1990–2004 280 6.83 3.34 1.00 21.00
Annual Review of Physiology; review-article
 1960–1974 258 4.16 2.67 1.00 14.00 .53
 1990–2004 299 7.78 3.16 1.00 20.00
Annual Review of Psychology; review-article
 1960–1974 162 2.70 1.96 1.00 15.00 .58
 1990–2004 239 6.47 3.37 1.00 18.00
Lancet; article
 1960–1974 3777 8.34 3.60 1.00 33.00 .53
 1990–2004 2436 12.87 4.59 2.00 34.00
Lancet; hypothesis
 1960–1974 256 7.61 3.46 1.00 21.00 .24
 1990–2004 115 9.52 3.83 2.00 24.00
Lancet; letters to the editor
 1960–1974 10,852 5.12 2.21 1.00 21.00 .20
 1990–2004 22,430 6.00 2.61 1.00 22.00
Nature; article
 1960–1974 4859 9.03 4.02 1.00 28.00 .24
 1990–2004 1182 10.33 3.28 2.00 23.00
Nature; letters to editor and correspondence
 1960–1974 18,812 9.13 3.74 1.00 36.00 .09
 1990–2004 18,249 8.44 3.62 1.00 34.00
Physiological Reviews; article
 1960–1974 248 6.13 2.93 1.00 15.00 .27
 1990–2004 460 7.90 3.16 1.00 18.00
Psychological Bulletin; journal article
 1960–1974 620 7.83 3.34 1.00 22.00 .33
 1990–2004 628 10.38 3.98 2.00 24.00
Psychological Review; journal article
 1960–1974 392 7.85 3.39 1.00 25.00 .24
 1990–2004 455 9.65 3.99 2.00 26.00
Science; letters
 1960–1974 1400 3.76 1.53 1.00 14.00 .20
 1990–2004 918 4.39 1.92 1.00 19.00
Science; report
 1960–1974 11,857 8.87 2.96 1.00 58.00 .21
 1990–2004 10,736 10.09 2.82 1.00 23.00

Table 2.

T test statistics for title length

t Degrees of freedom p value
American Psychologist; Comment and Reply −4.091 916 .000
American Psychologist; Journal Article −7.908 1934.237 .000
Annual Review of Biochemistry; review-article −10.968 488.989 .000
Annual Review of Physiology; review-article −14.677 554.791 .000
Annual Review of Psychology; review-article −14.135 390.756 .000
Lancet; article −41.234 4311.862 .000
Lancet; hypothesis −4.751 369 .000
Lancet; letters to the editor −32.026 24,949.985 .000
Nature; article −11.622 2130.690 .000
Nature; letters to editor and correspondence 18.068 37,059 .000
Physiological Reviews; article −7.323 706 .000
Psychological Bulletin; journal article −12.287 1214.666 .000
Psychological Review; journal article −7.120 844.862 .000
Science; letters −8.300 1653.707 .000
Science; report −31.809 22,535.249 .000

For all fifteen sets the mean number of authors per paper increases between the two periods (Table 9) and these increases are significant for all sets (Table 3). The effect sizes for the different sets vary from halfway between small and medium to large, the effect size is smallest for American Psychologist; Comment and Reply and largest for Science; report (see Table 9). All fifteen sets follow the predicted behavior.

Table 9.

Statistics for author count

N Mean SD Minimal Maximala R
American Psychologist; Comment and Reply
 1960–1974 89 1.24 .87 1.00 7.00 .19
 1990–2004 825 1.43 .91 1.00 11.00
American Psychologist; Journal Article
 1960–1974 758 1.25 .62 1.00 6.00 .27
 1990–2004 1111 1.82 1.55 1.00 20.00
Annual Review of Biochemistry; review-article
 1960–1974 351 1.54 .67 1.00 5.00 .35
 1990–2004 280 2.15 1.17 1.00 9.00
Annual Review of Physiology; review-article
 1960–1974 258 1.41 .62 1.00 4.00 .32
 1990–2004 299 1.97 1.13 1.00 12.00
Annual Review of Psychology; review-article
 1960–1974 162 1.46 .64 1.00 4.00 .29
 1990–2004 239 1.95 .90 1.00 6.00
Lancet; article
 1960–1974 3777 2.69 1.49 1.00 12.00 .28
 1990–2004 2436 7.80 17.07 1.00 697.00b
Lancet; hypothesis
 1960–1974 256 1.69 1.00 1.00 8.00 .45
 1990–2004 115 2.67 1.49 1.00 8.00
Lancet; letters to the editor
 1960–1974 10,852 1.90 1.21 1.00 19.00 .21
 1990–2004 22,430 2.52 1.79 1.00 39.00
Nature; article
 1960–1974 4622 2.24 1.34 1.00 16.00 .32
 1990–2004 1153 9.07 20.20 1.00 241.00c
Nature; letters to editor and correspondence
 1960–1974 17,853 2.15 1.12 1.00 13.00 .34
 1990–2004 17,132 4.11 5.14 1.00 349.00d
Physiological Reviews; article
 1960–1974 248 1.40 .57 1.00 3.00 .43
 1990–2004 460 2.32 1.39 1.00 10.00
Psychological Bulletin; journal article
 1960–1974 620 1.36 .69 1.00 6.00 .41
 1990–2004 628 2.10 1.06 1.00 6.00
Psychological Review; journal article
 1960–1974 392 1.36 .62 1.00 5.00 .43
 1990–2004 455 2.16 1.16 1.00 9.00
Science; letters
 1960–1974 1400 1.35 1.08 1.00 16.00 .20
 1990–2004 918 1.89 2.04 1.00 29.00
Science; report
 1960–1974 11,857 2.37 1.34 1.00 31.00 .53
 1990–2004 10,736 5.35 4.22 1.00 93.00

aIn the whole sample of 166,560 papers only 27 papers have 100 or more authors, and 67 have 50 or more

bLedergerber and PLATO Collaboration (2004)

cGibbs et al. (2004)

dAbazov et al. (2004)

Table 3.

T test statistics for author count

t Degrees of freedom p value
American Psychologist; Comment and Reply −2.025 110.103 .045
American Psychologist; Journal Article −11.139 1559.752 .000
Annual Review of Biochemistry; review-article −7.776 420.278 .000
Annual Review of Physiology; review-article −7.293 475.893 .000
Annual Review of Psychology; review-article −5.945 399 .000
Lancet; article −14.719 2459.036 .000
Lancet; hypothesis −6.417 161.699 .000
Lancet; letters to the editor −37.279 29,862.353 .000
Nature; article −11.469 1154.539 .000
Nature; letters to editor and correspondence −48.675 18,690.428 .000
Physiological Reviews; article −12.311 667.857 .000
Psychological Bulletin; journal article −14.569 1079.846 .000
Psychological Review; journal article −12.698 713.441 .000
Science; letters −7.398 1254.141 .000
Science; report −70.026 12,685.621 .000

The page count increases for ten out of the fifteen sets, in the other five the page count decreases (Table 10). The changes are significant in twelve sets, for American Psychologist; Comment and Reply, American Psychologist; Journal Article, and Lancet; hypothesis the change is not significant at all (Table 4). The effect sizes for the sets in which the change is significant vary from very small for Lancet; letters to the editor to very large for Science; report (see Table 10). Most sets behave as predicted.

Table 10.

Statistics for page count

N Mean SD Minimal Maximal r
American Psychologist; Comment and Reply
 1960–1974 92 2.23 1.58 1.00 13.00 .01
 1990–2004 692 2.21 .99 1.00 13.00
American Psychologist; Journal Article
 1960–1974 863 7.72 11.87 1.00 166.00 .03
 1990–2004 1406 8.25 7.30 1.00 176.00
Annual Review of Biochemistry; review-article
 1960–1974 351 31.32 10.63 8.00 73.00 .10
 1990–2004 279 33.45 9.73 9.00 71.00
Annual Review of Physiology; review-article
 1960–1974 258 31.88 11.67 10.00 84.00 .38
 1990–2004 299 24.46 7.81 2.00 56.00
Annual Review of Psychology; review-article
 1960–1974 162 33.45 10.59 12.00 92.00 .33
 1990–2004 239 28.92 5.16 18.00 57.00
Lancet; article
 1960–1974 3777 3.44 1.22 1.00 12.00 .59
 1990–2004 2435 5.24 1.59 1.00 14.00
Lancet; hypothesis
 1960–1974 256 3.11 1.14 1.00 9.00 .01
 1990–2004 115 3.12 .75 1.00 5.00
Lancet; letters to the editor
 1960–1974 10,828 1.38 .49 1.00 4.00 .04
 1990–2004 22,413 1.49 2.37 1.00 346.00
Nature; article
 1960–1974 4859 3.44 2.50 1.00 93.00 .46
 1990–2004 1182 6.51 1.61 1.00 29.00
Nature; letters to editor and correspondence
 1960–1974 18,812 2.01 .71 1.00 34.00 .47
 1990–2004 18,249 3.06 1.52 1.00 52.00
Physiological Reviews; article
 1960–1974 248 49.08 29.37 7.00 321.00 .21
 1990–2004 460 40.38 20.55 1.00 194.00
Psychological Bulletin; journal article
 1960–1974 620 13.97 7.43 1.00 51.00 .37
 1990–2004 627 20.46 9.26 1.00 70.00
Psychological Review; journal article
 1960–1974 392 13.98 7.36 2.00 52.00 .47
 1990–2004 454 23.38 10.33 3.00 63.00
Science; letters
 1960–1974 1400 3.10 6.06 1.00 105.00 .07
 1990–2004 918 2.36 5.17 1.00 112.00
Science; report
 1960–1974 11,857 2.60 .86 1.00 23.00 .60
 1990–2004 10,736 3.87 .83 1.00 20.00

Table 4.

T test statistics for page count

t Degrees of freedom p value
American Psychologist; Comment and Reply .119 100.707 .905
American Psychologist; Journal Article −1.174 1266.782 .241
Annual Review of Biochemistry; review-article −2.585 626 .010
Annual Review of Physiology; review-article 8.671 437.489 .000
Annual Review of Psychology; review-article 5.054 213.284 .000
Lancet; article −47.424 4245.280 .000
Lancet; hypothesis −.163 320.211 .871
Lancet; letters to the editor −6.702 26,140.010 .000
Nature; article −40.168 6039 .000
Nature; letters to editor and correspondence −84.856 25,727.743 .000
Physiological Reviews; article 4.149 380.383 .000
Psychological Bulletin; journal article −13.658 1194.349 .000
Psychological Review; journal article −15.381 815.745 .000
Science; letters 3.144 2163.076 .002
Science; report −112.927 22,494.270 .000

The number of sentences in the abstract rises in seven of the nine sets for which statistics for the number of sentences in the abstract could be calculated, for the other two, from the same journal, this number decreased (Table 11). These differences are significant (Table 5). The effect sizes for the different sets vary from small to very large the effect size is smallest for American Psychologist; Journal Article and largest for Lancet; article and Nature; Article (Table 11). This rise is in line with the predicted behavior.

Table 11.

Statistics for number of sentences in abstract

N Mean SD Minimal Maximal r
American Psychologist; Comment and Reply
 1960–1974 92 4.02 2.72 .00 16.00 .34
 1990–2004 793 2.94 1.33 .00 9.00
American Psychologist; Journal Article
 1960–1974 815 4.38 2.28 .00 17.00 .06
 1990–2004 1405 4.17 1.69 .00 11.00
Annual Review of Biochemistry; review-article
 1960–1974 1 11.00 11.00 11.00
 1990–2004 233 5.87 2.50 1.00 17.00
Annual Review of Physiology; review-article
 1960–1974 2 8.00 7.07 3.00 13.00
 1990–2004 253 5.80 2.67 1.00 16.00
Annual Review of Psychology; review-article
 1960–1974 0
 1990–2004 210 4.73 1.95 .00 11.00
Lancet; article
 1960–1974 1264 4.70 2.23 .00 19.00 .78
 1990–2004 2424 11.12 3.07 1.00 25.00
Lancet; hypothesis
 1960–1974 181 4.07 2.35 .00 11.00 .33
 1990–2004 104 5.72 2.21 1.00 13.00
Lancet; letters to the editor
 1960–1974 0
 1990–2004 10,370 2.70 1.64 .00 17.00
Nature; article
 1960–1974 1683 1.86 .84 .00 8.00 .78
 1990–2004 1179 4.74 1.91 1.00 13.00
Nature; letters to editor and correspondence
 1960–1974 18,288 4.11 2.29 .00 36.00 .57
 1990–2004 12,865 6.93 1.82 .00 26.00
Physiological Reviews; article
 1960–1974 0
 1990–2004 338 9.96 3.94 2.00 26.00
Psychological Bulletin; journal article
 1960–1974 617 4.61 1.79 .00 11.00 .19
 1990–2004 628 5.29 1.69 .00 11.00
Psychological Review; journal article
 1960–1974 389 5.06 2.05 .00 16.00 .12
 1990–2004 455 5.49 1.73 .00 14.00
Science; letters
 1960–1974 4 3.00 .82 2.00 4.00
 1990–2004 6 2.67 1.21 2.00 5.00
Science; report
 1960–1974 11,701 3.05 1.28 .00 18.00 .55
 1990–2004 10,724 4.80 1.40 .00 21.00

Table 5.

T test statistics for number of sentences in the abstract

t Degrees of freedom p value
American Psychologist; Comment and Reply 3.745 104.597 .000
American Psychologist; Journal Article 2.290 1336.504 .022
Lancet; article −72.459 3308.154 .000
Lancet; hypothesis −5.853 283 .000
Nature; article −48.565 1497.515 .000
Nature; letters to editor and correspondence −120.598 30,682.537 .000
Psychological Bulletin; journal article −6.858 1243 .000
Psychological Review; journal article −3.198 762.092 .001
Science; report −97.789 21,751.445 .000

In eight out of nine sets examined, the Flesch Reading Ease Score of the abstract is lower in the period 1990–2004 then it was in the period 1960–1974 (Table 12), these differences are significant (Table 6). This suggests that for eight of these sets the abstracts became harder to read, only Lancet; Articles became easier to read. The effect sizes for the different sets vary from small to halfway between small and medium, the effect size is smallest for Lancet; article and Nature; letters to editor and correspondence and largest for American Psychologist; Comment and Reply (see Table 12). As predicted, abstracts became harder to read.

Table 12.

Statistics for Flesch Reading Ease Score for the abstract

N Mean SD Minimal Maximal r
American Psychologist; Comment and Reply
 1960–1974 92 18.85 14.32 .00 65.70 .45
 1990–2004 793 10.88 11.32 .00 53.20
American Psychologist; Journal Article
 1960–1974 815 20.46 14.68 .00 100.00 .26
 1990–2004 1405 14.24 12.04 .00 67.20
Annual Review of Biochemistry; review-article
 1960–1974 1 34.80 34.80 34.80
 1990–2004 233 15.16 13.06 .00 55.90
Annual Review of Physiology; review-article
 1960–1974 2 42.40 2.55 40.60 44.20
 1990–2004 253 13.42 11.44 .00 54.70
Annual Review of Psychology; review-article
 1960–1974 0
 1990–2004 210 13.29 11.17 .00 42.20
Lancet; article
 1960–1974 1264 20.01 13.55 .00 60.50 .10
 1990–2004 2424 22.12 10.85 .00 53.60
Lancet; hypothesis
 1960–1974 181 14.21 12.61 .00 49.70 .16
 1990–2004 104 10.67 9.66 .00 38.80
Lancet; letters to the editor
 1960–1974 0
 1990–2004 10,370 22.01 16.13 .00 87.40
Nature; article
 1960–1974 1683 26.29 16.91 .00 80.70 .25
 1990–2004 1179 18.85 12.37 .00 67.60
Nature; letters to editor and correspondence
 1960–1974 18,288 20.66 13.89 .00 80.80 .08
 1990–2004 12,865 18.64 11.65 .00 90.00
Physiological Reviews; article
 1960–1974 0
 1990–2004 338 13.88 10.78 .00 48.70
Psychological Bulletin; journal article
 1960–1974 617 18.57 12.88 .00 57.50 .20
 1990–2004 628 13.49 11.60 .00 50.30
Psychological Review; journal article
 1960–1974 389 20.71 13.05 .00 62.10 .21
 1990–2004 455 15.25 11.95 .00 53.20
Science; letters
 1960–1974 4 19.30 5.38 15.10 26.60
 1990–2004 6 27.53 15.41 9.80 53.10
Science; report
 1960–1974 11,701 19.54 14.89 .00 81.30 .14
 1990–2004 10,724 15.86 12.07 .00 74.20

Table 6.

T test statistics for the abstracts Flesch Reading Ease score

t Degrees of freedom p value
American Psychologist; Comment and Reply 5.149 104.597 .000
American Psychologist; Journal Article 10.257 1445.817 .000
Lancet; article −4.796 2124.720 .000
Lancet; hypothesis 2.656 260.642 .008
Nature; article 13.584 2854.784 .000
Nature; letters to editor and correspondence 13.915 30,210.639 .000
Psychological Bulletin; journal article 7.309 1224.682 .000
Psychological Review; journal article 6.339 842 .000
Science; report 20.396 22,096.376 .000

For five of the fifteen sets the proportion of titles with a (semi-)colon in the title rises, for two the proportion stays about the same, and for seven the proportion is lower when we compare the period 1960–1974 to 1990–2004 (see Tables 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27). And these changes were found to be significant for the twelve sets in which we see a rise or drop (Table 7). The effect sizes for the sets in which the change is significant vary from small to halfway between medium and large, the effect size is smallest for Nature; article and largest for Lancet; article (Table 7). For some of the sets the predicted behavior is followed, others follow the opposite behavior.

Table 13.

Crosstabulation Period* Colon in title American Psychologist; Comment and reply

Colon in title Total
No Yes
Period
 1960–1974 67 (72.8 %) 25 (27.2 %) 92
 1990–2004 753 (91.2 %) 73 (8.8 %) 826
Total 820 98

Table 14.

Crosstabulation Period* Colon in title American Psychologist; Journal article

Colon in title Total
No Yes
Period
 1960–1974 635 (73.6 %) 228 (26.4 %) 863
 1990–2004 1363 (96.5 %) 50 (3.5 %) 1413
Total 1998 278

Table 15.

Crosstabulation Period* Colon in title Annual Review of Biochemistry; review-article

Colon in title Total
No Yes
Period
 1960–1974 333 (94.9 %) 18 (5.1 %) 351
 1990–2004 222 (79.3 %) 58 (20.7 %) 280
Total 555 76 631

Table 16.

Crosstabulation Period* Colon in title Annual Review of Physiology; review-article

Colon in title Total
No Yes
Period
 1960–1974 198 (76.7 %) 60 (23.3 %) 258
 1990–2004 228 (76.3 %) 71 (23.7 %) 299
Total 426 131

Table 17.

Crosstabulation Period* Colon in title Annual Review of Psychology; review-article

Colon in title Total
No Yes
Period
 1960–1974 155 (95.7 %) 7 (4.3 %) 162
 1990–2004 165 (69.0 %) 74 (31.0 %) 239
Total 320 81

Table 18.

Crosstabulation Period* Colon in title Lancet; article

Colon in title Total
No Yes
Period
 1960–1974 3669 (97.1 %) 108 (2.9 %) 3777
 1990–2004 1573 (64.6 %) 863 (35.4 %) 2436
Total 5242 971

Table 19.

Crosstabulation Period* Colon in title Lancet; hypothesis

Colon in title Total
No Yes
Period
 1960–1974 233 (91.0 %) 23 (9.0 %) 256
 1990–2004 102 (88.7 %) 13 (11.3 %) 115
Total 335 36

Table 20.

Crosstabulation Period* Colon in title Lancet; letters to the editor

Colon in title Total
No Yes
Period
 1960–1974 8474 (78.1 %) 2378 (21.9 %) 10,852
 1990–2004 21,407 (95.4 %) 1023 (4.6 %) 22,430
Total 29,881 3401

Table 21.

Crosstabulation Period* Colon in title Nature; article

Colon in title Total
No Yes
Period
 1960–1974 4468 (92.0 %) 391 (8.0 %) 4859
 1990–2004 1169 (98.9 %) 13 (1.1 %) 1182
Total 5637 404

Table 22.

Crosstabulation Period* Colon in title Nature; Letters to editor and correspondence

Colon in title Total
No Yes
Period
 1960–1974 17,748 (94.3 %) 1064 (5.7 %) 18,812
 1990–2004 18,037 (98.8 %) 212 (1.2 %) 18,249
Total 35,785 1276

Table 23.

Crosstabulation Period* Colon in title Physiological Reviews; article

Colon in title Total
No Yes
Period
 1960–1974 237 (95.6 %) 11 (4.4 %) 248
 1990–2004 365 (79.3 %) 95 (20.7 %) 460
Total 602 106

Table 24.

Crosstabulation Period* Colon in title Psychological Bulletin; journal article

Colon in title Total
No Yes
Period
 1960–1974 434 (70.0 %) 186 (30.0 %) 620
 1990–2004 440 (70.1 %) 188 (29.9 %) 628
Total 874 374

Table 25.

Crosstabulation Period* Colon in title Psychological Review; Journal Article

Colon in title Total
No Yes
Period
 1960–1974 315 (80.4 %) 77 (19.6 %) 392
 1990–2004 318 (69.9 %) 137 (30.1 %) 455
Total 633 214

Table 26.

Crosstabulation Period* Colon in title Science; letters

Colon in title Total
No Yes
Period
 1960–1974 1265 (90.4 %) 135 (9.6 %) 1400
 1990–2004 878 (95.6 %) 40 (4.4 %) 918
Total 2143 175

Table 27.

Crosstabulation Period* Colon in title Science; report

Colon in title Total
No Yes
Period
 1960–1974 7431 (62.7 %) 4426 (37.3 %) 11,857
 1990–2004 9937 (92.6 %) 799 (7.4 %) 10,736
Total 17,368 5225

Table 7.

Chi square statistics for colon in title

Value Degrees of freedom p value ϕ
American Psychologist; Comment and Reply 29.187 1 .000 .18
American Psychologist; Journal Article 261.596 1 .000 .34
Annual Review of Biochemistry; review-article 35.716 1 .000 .24
Annual Review of Physiology; review-article .018 1 .892 .01
Annual Review of Psychology; review-article 42.514 1 .000 .33
Lancet; article 1191.184 1 .000 .44
Lancet; hypothesis .487 1 .485 .04
Lancet; letters to the editor 2400.234 1 .000 .27
Nature; article 73.528 1 .000 .11
Nature; letters to editor and correspondence 562.801 1 .000 .12
Physiological Reviews; article 33.286 1 .000 .22
Psychological Bulletin; journal article .001 1 .980 .00
Psychological Review; journal article 12.218 1 .000 .12
Science; letters 22.192 1 .000 .10
Science; report 2830.663 1 .000 .35

This observation might, however, be misguided, as it appears that in the period 1990–1994, and most likely some surrounding years, the number of (semi-)colon in titles was very low, which might point to a discrepancy in the way Web of Knowledge treated this character (see the exemplary graphs in Figs. 1, 2). For instance the articles noted on APA PsycNET as “Support theory: A nonextensional representation of subjective probability” and “Simultaneous over- and underconfidence: The role of error in judgment processes.” are registered in WOK as “Support theory—A nonextensional representation of subjective probability” and “Simultaneous over- and underconfidence—The role of error in judgment processes” respectively. Note this is not only limited to these two journals, but occurs throughout the dataset for this period.

Fig. 1.

Fig. 1

Use of colon in title for Psychological Bulletin; journal article

Fig. 2.

Fig. 2

Use of colon in title for Psychological Review; journal article

Conclusion and Discussion

Whilst the predicted pattern is followed in most cases there are notable exceptions such as the title length, for which the predicted pattern is only followed in one set. For other variables the predicted pattern is followed more closely, for every set the number of authors increases, although the single-authored paper did not die out, something de Solla Price (1963) predicted would happen for Chemical Abstracts, not included in this sample, by 1980. For the other cases there are one or more sets not following the predicted behavior. Exception being the presence of a colon with in the title, which only rises in six sets, but might have an alternative explanation (see “Results” section).

Given only fifteen journal/paper type sets, representing ten journals, which in turn represent four JCR Science Edition categories and only one JCR Social Science Edition, were included in this paper some explanation for changes in individual variables can be sought in changes in the journal’s editorial policies or in changes in the field. A change in policies could have caused the change in the number of authors, but this would then have to have happened to for all journals examined. External factors of influence, other than the ideas that citing for impact and publish or perish are important, could also explain changes. The decline in readability could be caused by the rise in the use of word-processing software. Tin and Inggeris (2000) did find that students produce more complex text when word processing than when writing with pen and paper. Changes in the field could both be caused by changes related and not related to evaluation criteria based on publish or perish or on publishing for impact.

The rise in authors could be explained by a rise in multidisciplinary research. However this has already been dismissed as the sole explanation in earlier writings (Broad 1981; Matías-Guiu and García-Ramos 2010). Maybe the extravagant number of authors listed on some articles is not the result of an attempt to beat the Publish or Perish game or an attempt to become highly cited. Maybe it is a genuine attempt to acknowledge the contribution of those without whose work the research would not have been possible, such as lab technicians, doctors collecting data in their practices, and technicians keeping the machines of big science up and running. Instrumental and valuable, but not authors. Their contributions are perhaps too great to merely thank them in the Acknowledgements. Thus perhaps one solution is to find an alternative form of recognition that fills this apparent void between ‘Acknowledgement’ and ‘author’ ought to be filled. Gratuitous listing of co-authors greatly devalues authorship, something overlooked when research performance is evaluated. With authorship comes rights, the right for individuals to put a publication in their CVs, the right for a department or institution to claim the output. But authorship also comes with responsibility, as all authors are responsible for the content, right or wrong (for the latter, including responsibility for fraudulent actions, such as plagiarism and data fraud). Is the 50th author willing to bear this responsibility, responsibility for an article (s)he did not witness being created and might not even have read before it was submitted for publication?

Given the predicted behavior is followed in most of the cases, and there is a realistic case for why these changes occurred, it is not unreasonable to link the observed changes in publication behavior to a change in evaluation criteria, which is also not out-of-line with what commonsense would predict. These findings combined with those of other researchers, for instance the link between high impact publications and hot topics (Moustafa 2014), lead to the conclusion that evaluation criteria act as switchmen, determining the tracks along which scholarly work is pushed by the dynamic of interests of both scholars and their institutions.

Both the changes which follow as well as those which are counter to the expected behavior could be explained by the journal’s editorial policies. It would be possible to discover such policies by studying editorials and comments on submitted manuscripts. This would establish if editorial policies could have influenced these variables, but will, most likely, not explain what caused the change in policies, which could also be a response to external evaluation criteria.

We also need to consider the underlying cause of the rise of these ideas: why have they become so dominant in science? This is a harder question to answer, and one needs to work through national and university policy documents in order to find an answer to this question. One explanation might be sought in changes in how governments try to justify expenditures. Starting in the early 1980s university policies have increasingly been influenced by a need for accountability, at least in EU countries (Geuna 2001). Without a Citation Index it is questionable if the number of citations would be as important as it is now. This is not a technologically deterministic stance (Smith and Marx 1994), a feasible way of counting citations was needed to facilitate the operationalization of the idea, or in other words “knowledge is embedded in and performed by infrastructures” (Wyatt et al. 2013). Most likely the negative effect of the competition system for distributing careers, funding and the like on the behavior of scientist has been underestimated by those who bestow the rewards (Anderson et al. 2007).

An interesting future research opportunity would be to compare how these trends differ between fields. Some are traditionally more book- than article-based, notably in the humanities, and still try to hold on to these traditions. Such publications are less easily evaluated by criteria which are predominantly used in the STEM and medical fields.

This research has aimed to find whether publications and citation pressures have resulted in changes in the number of papers scholars produce and in their characteristics, but there are other, potentially more fruitful approaches. Another method, more in line with Weber, would be to examine the teachings of practices, or to study the theology 6 of the ideas (for inspiration Weber 1958). These teachings could be distilled from scholarly guidebooks, such as methods texts, editorial guidelines or codes of practice, by looking for suggestions about salami slicing, duplicate publications, multi-authorship, inflating references, etc. The theology could be found in policies on promotion (such as the granting of tenure), basis for research funding, and the basis for rankings on university and (inter)national levels.

Acknowledgments

The author would like to thank the anonymous reviewer of JSEE, and the anonymous reviewers and visitors of the International Integrity and Plagiarism Conference for their suggestions and Sally Wyatt and Jeroen ten Haaf for their many insightful comments, suggestions, and discussions.

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical standard

This article does not contain any studies with human participants or animals performed by any of the authors.

Appendix

See Tables 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26 and 27.

Footnotes

1

See, for instance, Andreescu (2013), Broad (1981), Elliott (2013), Kumar (2008), and von Elm et al. (2004) for a compilation of types of misconduct.

2

The author is aware of the irony of this statement.

3

Which might be due to the conditions our institutes subscription.

5

Simple code link <p> but also more complex like <xml…>.

6

Theology is meant here not as the study of God(s) but as the science of things divine; “All theology represents an intellectual rationalization of the possession of sacred values” (Weber 1970a).

References

  1. Abazov VM, Abbott B, Abdesselam A, Abolins M, Abramov V, Acharya BS. A precision measurement of the mass of the top quark. Nature. 2004;429(6992):638–642. doi: 10.1038/nature02589. [DOI] [PubMed] [Google Scholar]
  2. Anderson MS, Ronning EA, De Vries R, Martinson BC. The perverse effects of competition on scientists’ work and relationships. Science and Engineering Ethics. 2007;13(4):437–461. doi: 10.1007/s11948-007-9042-5. [DOI] [PubMed] [Google Scholar]
  3. Andreescu L. Self-plagiarism in academic publishing: The anatomy of a misnomer. Science and Engineering Ethics. 2013;19(3):775–797. doi: 10.1007/s11948-012-9416-1. [DOI] [PubMed] [Google Scholar]
  4. Broad W. The publishing game: Getting more for less. Science. 1981;211(4487):1137–1139. doi: 10.1126/science.7008199. [DOI] [PubMed] [Google Scholar]
  5. de Solla Price DJ. Little science, big science (George B Pegram lectures, Vol. 1962) New York: Columbia University Press; 1963. [Google Scholar]
  6. Dijstelbloem, H., Huisman, F., Miedema, F., & Mijnhardt, W. (2013). Why science does not work as it should and what to do about it. http://www.scienceintransition.nl/wp-content/uploads/2013/10/Science-in-Transition-Position-Paper-final.pdf. Accessed March 5, 2015.
  7. Elliott DB. Salami slicing and the SPU: Publish or Perish? Ophthalmic and Physiological Optics. 2013;33(6):625–626. doi: 10.1111/opo.12090. [DOI] [PubMed] [Google Scholar]
  8. Errami M, Garner H. A tale of two citations. Nature. 2008;451(7177):397–399. doi: 10.1038/451397a. [DOI] [PubMed] [Google Scholar]
  9. Flesch R. A new readability yardstick. Journal of Applied Psychology. 1948;32(3):221–233. doi: 10.1037/h0057532. [DOI] [PubMed] [Google Scholar]
  10. Franco G. Research evaluation and competition for academic positions in occupational medicine. Archives of Environmental & Occupational Health. 2013;68(2):123–127. doi: 10.1080/19338244.2011.639819. [DOI] [PubMed] [Google Scholar]
  11. Frenken K, Hölzl W, Vor FD. The citation impact of research collaborations: The case of European biotechnology and applied microbiology (1988–2002) Journal of Engineering and Technology Management. 2005;22:9–30. doi: 10.1016/j.jengtecman.2004.11.002. [DOI] [Google Scholar]
  12. Garfield E. Citation indexes for science. A new dimension in documentation through association of ideas. International Journal of Epidemiology. 2006;35(5):1123–1127. doi: 10.1093/ije/dyl189. [DOI] [PubMed] [Google Scholar]
  13. Geuna A. The changing rationale for European university research funding: Are there negative unintended consequences? Journal of Economic Issues. 2001;35(3):607–632. doi: 10.1080/00213624.2001.11506393. [DOI] [Google Scholar]
  14. Gibbs RA, Weinstock GM, Metzker ML, Muzny DM, Sodergren EJ, Scherer S. Genome sequence of the Brown Norway rat yields insights into mammalian evolution. Nature. 2004;428(6982):493–521. doi: 10.1038/nature02426. [DOI] [PubMed] [Google Scholar]
  15. Glänzel W, Thijs B. Does co-authorship inflate the share of self-citations? Scientometrics. 2004;61(3):395–404. doi: 10.1023/B:SCIE.0000045117.13348.b1. [DOI] [Google Scholar]
  16. Gross PLK, Gross EM. College libraries and chemical education. Science. 1927;66(1713):385–389. doi: 10.1126/science.66.1713.385. [DOI] [PubMed] [Google Scholar]
  17. Gross AG, Harmon JE, Reidy MS. Communicating science: The scientific article from the 17th century to the present. Oxford/New York: Oxford University Press; 2002. [Google Scholar]
  18. Haslam N, Ban L, Kaufmann L, Loughnan S, Peters K, Whelan J, et al. What makes an article influential? Predicting impact in social and personality psychology. Scientometrics. 2008;76(1):169–185. doi: 10.1007/s11192-007-1892-8. [DOI] [Google Scholar]
  19. Hudson J. Be known by the company you keep: Citations—Quality or chance? Scientometrics. 2007;71(2):231–238. doi: 10.1007/s11192-007-1671-6. [DOI] [Google Scholar]
  20. Jacques TS, Sebire NJ. The impact of article titles on citation hits: An analysis of general and specialist medical journals. JRSM Short Reports. 2010 doi: 10.1258/shorts.2009.100020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Jamali H, Nikzad M. Article title type and its relation with the number of downloads and citations. Scientometrics. 2011;88(2):653–661. doi: 10.1007/s11192-011-0412-z. [DOI] [Google Scholar]
  22. Kumar MN. A review of the types of scientific misconduct in biomedical research. Journal of Academic Ethics. 2008;6(3):211–228. doi: 10.1007/s10805-008-9068-6. [DOI] [Google Scholar]
  23. Latour B. Science in action: How to follow scientists and engineers through society. Cambridge, MA: Harvard University Press; 1987. [Google Scholar]
  24. Ledergerber B, PLATO Collaboration Predictors of trend in CD4-positive T-cell count and mortality among HIV-1-infected individuals with virological failure to all three antiretroviral-drug classes. Lancet. 2004;364(9428):51–62. doi: 10.1016/S0140-6736(04)16589-6. [DOI] [PubMed] [Google Scholar]
  25. Levelt Committee, Noort Committee, & Drenth Committee. (2012). Flawed science: The fraudulent research practices of social psychologist Diederik Stapel. https://www.commissielevelt.nl/wp-content/uploads_per_blog/commissielevelt/2013/01/finalreportLevelt1.pdf. Accessed March 5, 2015.
  26. Levitt JM, Thelwall M. Citation levels and Collaboration within library and information science. Journal of the American Society for Information Science and Technology. 2009;60(3):434–442. doi: 10.1002/asi.21000. [DOI] [Google Scholar]
  27. Matías-Guiu J, García-Ramos R. Fraud and misconduct in scientific publications. Neurología. 2010;25(1):1–4. doi: 10.1016/S0213-4853(10)70015-3. [DOI] [PubMed] [Google Scholar]
  28. Microsoft. (2007). Test your document’s readability. https://support.office.com/en-us/article/Test-your-documents-readability-0adc0e9a-b3fb-4bde-85f4-c9e88926c6aa?ui=en-US&rs=en-US&ad=US#bm1. Accessed November 27, 2014.
  29. Moustafa K. The disaster of the impact factor. Science and Engineering Ethics. 2014 doi: 10.1007/s11948-014-9517-0. [DOI] [PubMed] [Google Scholar]
  30. Nolan P, Lenski GE. Human societies: An introduction to macrosociology. 10. Boulder, CO: Paradigm Publishers; 2006. [Google Scholar]
  31. Ramos MA, Melo JG, Albuquerque UP. Citation behavior in popular scientific papers: What is behind obscure citations? The case of ethnobotany. Scientometrics. 2012;92:711–719. doi: 10.1007/s11192-012-0662-4. [DOI] [Google Scholar]
  32. Regmi K. Ethical and legal issues in publication and dissemination of scholarly knowledge: A summary of the published evidence. Journal of Academic Ethics. 2011;9(1):71–81. doi: 10.1007/s10805-011-9133-4. [DOI] [Google Scholar]
  33. Smith MR, Marx L, editors. Does technology drive history? The dilemma of technological determinism. Cambridge, MA: MIT Press; 1994. [Google Scholar]
  34. Tin, O. S., & Inggeris, J. (2000). Writing with different tools: The computer vs the pen. Jurnal Pendidikan Tigaenf, 2(4), 69–93.
  35. Tsafrir JS, Reis T. Using the citation index to assess performance. British Medical Journal. 1990;301(6764):1333. doi: 10.1136/bmj.301.6764.1333-c. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Vieira ES, Gomes JANF. Citations to scientific articles: Its distribution and dependence on the article features. Journal of Informetrics. 2010;4:1–13. doi: 10.1016/j.joi.2009.06.002. [DOI] [Google Scholar]
  37. von Elm E, Poglia G, Walder B, Tramèr MR. Different patterns of duplicate publication: An analysis of articles used in systematic reviews. JAMA. 2004;291(8):974–980. doi: 10.1001/jama.291.8.974. [DOI] [PubMed] [Google Scholar]
  38. Wade N. Citation analysis: A new tool for science administrators. Science. 1975;188(4187):429–432. doi: 10.1126/science.188.4187.429. [DOI] [PubMed] [Google Scholar]
  39. Wang M, Yu G, An S, Yu D. Discovery of factors influencing citation impact based on a soft fuzzy rough set model. Scientometrics. 2012 [Google Scholar]
  40. Weber M. (1958) The Protestant ethic and the spirit of capitalism (Student’s ed.). New York: Scribner.
  41. Weber, M. (1970a). Science as a vocation. In H. H. Gerth & C. W. Mills (Eds.), From Max Weber: Essays in sociology (pp. 129–156). London: Routledge & Kegan Paul Ltd.
  42. Weber, M. (1970b). The social psychology of the world religions. In H. H. Gerth & C. W. Mills (Eds.), From Max Weber: Essays in sociology (pp. 267–301). London: Routledge & Kegan Paul Ltd.
  43. Webster GD, Jonason PK, Schember TO. Hot topics and popular papers in evolutionary psychology: Analyses of title words and citation counts in evolution and human behavior, 1979–2008. Evolutionary Psychology. 2009;7(3):348–362. doi: 10.1177/147470490900700301. [DOI] [Google Scholar]
  44. Wesel MV, Wyatt S, Haaf J. What a difference a colon makes: How superficial factors influence subsequent citation. Scientometrics. 2014;98(3):1601–1615. doi: 10.1007/s11192-013-1154-x. [DOI] [Google Scholar]
  45. Wyatt, S., Scharnhorst, A., Beaulieu, A., & Wouters, P. (2013). Introduction to virtual knowledge. In P. Wouters, A. Beaulieu, A. Scharnhorst, & S. Wyatt (Eds.), Virtual knowledge: Experimenting in the humanities and the social sciences (pp. 1–23). Cambridge, MA: MIT Press.

Articles from Science and Engineering Ethics are provided here courtesy of Springer

RESOURCES