The Last 50 Years: Mismeasurement and Mismanagement Are Impeding Scientific Research

Abstract

In the last 50 years, there have been many changes to the substance, conduct, and style of research. Many of these changes have proved disastrous to the life of scientists and to science itself. As a consequence, the near-romantic spirit of adventure and exploration that inspired young scientists of my own and earlier generations has become tarnished. Now, many of us feel beleaguered by bureaucrats and by politicians: they affect our lives profoundly, apparently without an understanding of the way discoveries are made or of the nature of science itself. The core purposes of universities, teaching and research, are being eroded by excessive administration. The number and locations of our publications are counted up like beans and the outcomes are used to rank us, one against another; a process of evaluation that has recast the purposes of publication. Applying for grants takes far too much time from a young scientist's life.

O good old man, how well in thee appears
The constant service of the antique world,
When service sweat for duty, not for meed.
Thou art not for the fashion of these times,
Where none will sweat but for promotion,
And having that do choke their service up
Even with the having

1599 Shakespeare, As You Like It II, ii

A man is nothing now unless he has within him an appreciation of the new era, an era in which it would seem that neither honesty nor truth is very desirable, but in which success is the only touchstone of merit

Trollope (1857)

If he had been quite a different person, he could now have made his position one of great influence, it being the time for sitting on coordinating committees, Anglo-American committees, future policy committees. But he had no concern for finding a place in anything but scientific research itself

Hodges (1983)

Let me tell you how you become a man of power…of influence: you trade your ideals for self-interest

Morton-Smith (2015)

1966 was the initial issue of CTDB. That year I was a starting postdoc in Charlottesville, Virginia and my first experimental papers on insect development were published in the same year. So, professionally, CTDB and I are almost exact contemporaries. In our 50 years, we have both seen much evolution in the substance, conduct, and style of research. Some of these many changes have proved disastrous to the life of scientists and to science itself. It is not to say there has not been progress, of course there has, but this has been accompanied by an insidious corruption of the practice of research. As a consequence, the near-romantic spirit of adventure and exploration that inspired young scientists of my own and earlier generations has become tarnished and almost extinguished. Now, many of us feel beleaguered by bureaucrats and by politicians: they affect our lives pro-foundly, apparently without an understanding of the way discoveries are made or of the nature of science itself. The core purposes of universities, teaching and research, are being eroded by excessive administration. The number and locations of our publications are counted up like beans and the outcomes are used to rank us, one against another; a process of evaluation that has recast the purposes of publication. In addition, the granting system is so dysfunctional it could not have been designed—it has evolved from benign beginnings into a clumsy and purblind monster that tramples on innovation and creativity.

Let us have a closer look into seven of these claims:

1. the near-romantic spirit of adventure and exploration that inspired young scientists of my own and earlier generations has become tarnished This is a subjective statement; it describes how we felt then and now feel about scientific research, how we were and are motivated. I have long been impressed with the story of “The Worst Journey in the World” (Cherry-Garrard, 1922) when, in the darkness and cold of an Antarctic winter (1911), a few intrepid men set off on foot to collect one egg of the emperor penguin. The main justification was that such an egg had never before been seen by science. They risked and nearly lost their lives. Yes, their motivation was romantic and proved so, for afterward when they took the egg to the Natural History Museum in London they did not get the heroes’ welcome they had expected (Cherry-Garrard, 1922). Later, on the same expedition, Scott and his companions continued to manhaul heavy rocks right up to their death. Common sense dictated they should have abandoned the rocks; but they saw their expedition as scientific and, therefore, these geological specimens symbolized a grander purpose than the aim of being first to reach the South Pole (Scott, 1913). Science was highly rated in the first half of the twentieth century.

Even when I started to do research in 1962, there was an atmosphere of excited curiosity among graduate students; yes there were career thoughts, but few. Such thoughts were secondary to the quest to discover, something, anything. Now, it is relatively rare for students to discuss their projects in the coffee room; much more time is spent worrying about the next career step, about publication and the politics of science. This change has even affected my life, I now get more invitations to talk about scientific politics and careers than about my research.

One consequence of this current pragmatism is that we have lost many gifted people who might otherwise have gone into pure science. Knowing how tough, insecure and financially unrewarded the scientific life is, most bright students prefer other paths such as finance, law, and medicine. The result, particularly in the prosperous regions of the west, is that we have been recruiting young scientists from abroad, including the third world. A large fraction of graduate students in science in the United Kingdom and in the United States did their first degrees outside these countries. I have not been able to recruit a British person to my (little) group for more than a decade, mainly because “the homegrown postgraduate population is pitifully thin” (Dyson, 2015). Nevertheless, when people of different backgrounds and education work together, the clash of perspectives may well enhance the quality of their research.

However, there are bright lights in the darkness and one of the most inspiring is the discovery by Emmanuelle Charpentier and Jennifer Doudna and colleagues. Their CRISPR cas9 genetic editing technique is a big advance and many applications to science and human health are expected. Their technique derives from basic research on bacterial immunity by Phillipe Horvath, Sylvain Moineau, Charpentier and others. All of these people researched in a subject that was out of fashion and dismissed as “old microbiology”. Yet they persisted and the genetic editing technique is “a great example of a project that came about really through serendipity…with a very little bit of grant support…to not follow the fashion but to follow your curiosity and…discover things you would not have expected.” (Breakthrough Prize, 2015). It is my opinion that this discovery was made in spite of the present systems and not because of them.

2. now many of us feel beleaguered by bureaucrats and politicians Another subjective statement: but there is no doubt that most scientists resent being assessed and ruled by those who do not seem to understand how we work or what we try to achieve. Our new overlords seem to think that innovation is like digging tunnels, it is just a matter of more effort and then we will get further. There seems to be little understanding of the summer lightning of creativity or the inconvenient truth that trying new things carries a big risk of failure. There are many examples that illustrate how our purposes have become despoiled by bureaucrats but one stands out. It is the “impact statement” that we must write when we apply for money to do research. In the United Kingdom, we are forced to “describe societal and economic deliverables and milestones instead of focusing on just scientific deliverables” (BBSRC, 2015). This demand is not only badly written but also impossible to fulfill. Research is investigating the unknown; we cannot predict what we will find. Even less can we predict how what we might find could yield “societal and economic deliverables.”

There are many instances that make my point, three clear cases will do:

First, when Milstein and Kohler discovered how to make monoclonal antibodies they consulted people (the National Research Development Council) especially employed to assess future “economic deliverables” and to patent discoveries that could be lucrative. After considering monoclonal antibodies, the Council concluded “It is certainly difficult for us to identify any immediate practical applications which could be pursued as a commercial venture” and decided to take no action (WhatisBiotechnology.org, 2013). Remember at the time of their evaluation the discovery had already been made, so that at least was no longer an unknown. Yet they were unable to see any value in it! For the year 2016, the sales attributable to monoclonal antibodies will be worth approximately 58 billion (sic) US dollars (BCCResearchLLC, 2012).

Second, Townes, who invented the laser, was told by the former and current chairmen of his department—both Nobel laureates—that “It isn’t going to work. You know it’s not going to work, we know it’s not going to work.” (Townes, 1999). Nevertheless, it did work and even Townes himself could not see much use for it…then!

Third, Turing and colleagues found it difficult to get adequate funds to make what would be the first digital computer (Hodges, 1983).

As we write our impact statements, we know we will be rewarded for writing hogwash and putting aside objectivity. But, even worse, the requirement for impact statements is an outcome of a more widespread corruption: increasingly, we scientists, both inside and outside of our scientific papers, are talking up our work. We are using “the alchemy of spin to transform leaden pieces of information into fool’s gold…” (Lawrence, 2001) and as we do, we forsake our scientific principles. Hyping our research has become institutionalized: we do it, our universities do it, our funding bodies do it, and press releases and glossy brochures do it, let’s give it up.

3. The core purposes of universities, teaching and research, are being eroded by excessive administration There has been a disproportionate increase in management posts within universities and research institutes including, for example, in human resources, “research support” (Colquhoun, 2007), and health and safety. There has been a phenomenal growth in jobs in various kinds of outreach, for example, befriending donors, relating with the media, and manipulating the presentation of research in order to maximize government support for the university (HEFCE, 2014). Jobs in “metascience” have increased hugely over the last decades. By metascience I mean administration within grant-giving bodies (much of the work consists of considering and rejecting applications) and editorship of journals (most of the work consists of considering and rejecting submitted articles). Within higher education in the United States, “nonclassroom costs have ballooned, administrative payrolls being a prime example. The number of employees hired by colleges and universities to manage or administer people, programs and regulations increased 50% faster than the number of instructors between 2001 and 2011,” according to the U.S. Department of Education (Belkin & Thurm, 2012). Many gifted young scientists, particularly women, have found these posts to be a wiser option than research. These posts are usually secure and carry pensions and therefore take a large amount of money from the total scientific and teaching effort. They can be contrasted with the ephemeral and insecure support given to many teachers, researchers, and their younger dependents (students, postdocs).

Administration, although its avowed purpose should be to facilitate the work of the researchers and teachers, actually eats up a lot of our time. Reports and evaluations are frequently sought and I am supposed to check, annotate and update all my publications on several independent web-based systems. The process of buying items with our grant money can become enmeshed in complex procedures of procurement, purchasing, and accountability. A main force driving this increase in bureaucracy is the institution’s wish to create a paper trail, one of whose purposes is to pass any risks of litigation onto us (Vrountas & Ayer, 2013).

And in a recent and cruel twist, we scientists are increasingly being seen by management as cash cows whose raison d’être is financial (Jump, 2015)—if we do not attract enough grant money, we are losing our jobs. Some who have been fulfilling the primary duties of our posts—teaching and research—are being unfairly humiliated and even dismissed (Garwood, 2012). The recent case of Stefan Grimm who committed suicide after being hounded by the head administrators of Imperial College, London for not winning enough grant money is a terrible example (Colquhoun, 2014; Parr, 2014).

4. The number and locations of our publications are counted up…and then used to rank us, one against another Over the last 30 years, we have been building an “audit society” (Power, 1997) in which almost everything, including the scientific value of individuals and of their published articles, is measured or rather, mismeasured. This matter has been discussed by many (see for example, Gruber, 2014; Lawrence, 2007, 2009; Metze, 2010; Simons, 2008; Wang, Song, & Barabasi, 2013) but we scientists have also been seduced by the meretricious appeal of bibliometrics. There are a number of consequences:

First, so-called top journals are now places of pilgrimages, worth sacrifices, and long painful journeys to reach. The reasons are both pragmatic and mythical. It is only partly true but widely believed that just one paper in Nature, right or wrong, can get a young scientist an entrée into a grant or post.

Second, papers in Nature or Science tend to be so dense and specialized that few can follow them. But this does not matter too much—it is more important to be published than to communicate: “papers have become competitive tokens for insertion into grant-dispensing gambling machines rather than bricks in the edifice of science” (Lawrence & Locke, 1997).

Third, the increasing use of citations as a measure of success has reinforced the hegemony of the top journals. Because citations to these journals give the writer’s paper an elitist glow, such citations tend to be preferred over more appropriate citations to a lesser journal and, therefore, papers in the top journals get more citations than they deserve.

5. the granting system is so dysfunctional it could not have been designed For a young researcher it is discouraging to navigate the maze of grants, all with different eligibility requirements, clashing deadlines, and byzantine web-based forms. Typical postdoc grants last 2–3 years and up to 50% of that entire time and effort can be taken up as the young scientists try to negotiate their futures (Germain, 2015; Lawrence, 2009). Note that the odds of winning a research project grant are around 1 in 5 (NIH, 2014) and grant applications take weeks or months to complete. Grant applications are part exercises in fiction, in which a long list of experiments is contrived to look feasible and fashionable. They are fictional because one cannot predict how experiments will turn out and, anyway, it is imprudent to put one’s best ideas into these applications as they are likely to be reviewed by competitors. For some grants, one is expected to write a specific timetable for the development of the project up to several years ahead, as if we scientists were soothsayers. Even worse, reports can be demanded and then judged as to how well these plans have been met. This process discourages original research by inhibiting changes in direction and making it less likely that new opportunities will be taken up. Science management should try to remember Max Perutz’s famous observation “Creativity in science, as in art, cannot be organised. It arises spontaneously from individual talent. Well-run laboratories can foster it, but hierarchical organisations, inflexible bureaucratic rules, and mountains of futile paperwork can kill it. Discoveries cannot be planned, they pop up, like Puck, in unexpected corners” (Perutz, 2002).

6. mismeasurement has damaged the practice of publication itself Let us look at how scientific articles have changed over the last 50 years. In the mid-1960s, some of my papers were quite easily published in Nature as preliminary notes, cut and tailored to reach a “general reader” (who may have actually existed then; the general reader is now a mythical beast). Later, I published the same pieces of work more completely and at greater length in specialized journals. Now Nature letters detail substantial pieces of work but they have to be prepared like dressed crabs and squashed into a tiny space. Figures are reduced to the size of postage stamps and the text is often a packed jumble of acronyms. Indeed, there cannot be many who understand more than a small proportion of the papers in scientific journals, yet that is not so important to the authors—because a paper’s main function is not communication but to be a badge of honor. This change of purpose has had fundamental consequences, one is that scientists make less time for reading and comprehending because, in order to survive, they must put so much effort into writing manuscripts and manoeuvring them into journals.

In my opinion, there are other factors, apart from scientific quality, that help papers into publication in the highest ranking journals. These factors have affected the way papers are written, changes that have not benefited readers. For example, journals and editors are in thrall to impact factors. During the time I edited Development, which I did for most of the 50 years under review, the impact factor grew from a somewhat irrelevant measure into a dominating force. The impact factor relates to the average number of citations given to papers in a journal (Garfield, 2006). Editors know there will be more citations if they select papers in areas where many scientists are currently working. This means that manuscripts in unfamiliar fields tend to be rejected, while articles in currently fashionable fields are preferred. Of course, papers that prove later on to have had real quality or novelty attract citations over time, but they can be difficult to identify when submitted and, anyway, for the impact factor only the first 2 years of citations count. Therefore, fashionable papers are a safer bet. Editors tend to like research with medical relevance, such as findings related to a common human genetic disease, as the medical literature is vast, yielding many citations. The widespread conviction that new, exploratory or unusual topics are unlikely to be appreciated by editors has bred a play-safe attitude in the choice of projects. Thus, over the years, the sense of adventure so vital to research has been dampened.

Then, authors also have to get their manuscripts past reviewers and for this it is wise to fill in small gaps in our knowledge (identifying new members of an established genetic or metabolic pathway is a good strategy). Such findings fit in well with current dogma and are likely to be welcomed by reviewers. Another safe strategy is to produce papers that are data heavy and hypothesis light. This is not as difficult as modern methods can yield data relatively easily and then, a mass of results can give an article momentum, making it more difficult for a reviewer to find reasons to reject. But, unfortunately, too much data and too little storyline do not help the reader, who increasingly, cannot see any connection between the hyped claims in the abstract (tailored for the editor) and the mind-numbing piles of numbers and statistics in the result sections (designed to get past the reviewers). Another point for authors to remember: it is not prudent to try to overturn current opinion as this is likely to irritate at least one reviewer. Editors, tending to err toward safety, give too much weight to a negative reviewer and, in practice, a single dissenting voice often vetoes a paper (Indusieumgresium, 2009). Consequently, a fear of antagonizing reviewers has led to an increasing tendency to be diplomatic, to write manuscripts where conclusions are obfuscated and opinion is hidden—this does not help readers either.

Review articles have also been affected because they have two mutually conflicting purposes: first, they should aim to help readers understand a field and second, for the journals, they have become a means to gain citations and increase impact factors. Critical or “biased” reviews are actually more useful for readers as they compare cases and present evidence and arguments. However, editors prefer reviews that are anodyne lists of references because these can be cited as representing a whole field. Consequently, most reviews simply repeat current dogma and fail to examine the evidence behind it.

7. There has been “an insidious corruption of the practice of research” Have these political changes impacted on research itself? I think the answer is yes. It is instructive to look at how the change in the primary purposes of publication (from communication and record to producing tokens that will yield salaries and grants) has affected the way we structure our research. All of us have had to focus our research to produce enough papers to compete and survive. Thus, projects are published as soon as possible and many therefore resemble lab reports rather than fully rounded and completed stories. There are many reasons why projects may not be pursued to a point of clarification, a clarification that would benefit everyone, particularly the reader. Often the person responsible only has support for 2 or 3 years and has to leave; therefore, passing on the project to another person can cause authorship disputes. Also clearing up inconsistencies in research can take too much time; it may be more productive to publish what we have and move on. Consequently, it may be more effective in terms of the numbers of papers to start a new project for each person (for each potential first author). Thus, I think this emphasis on article numbers has helped make papers poorer in quality. And, even more significant, there is an effect on the choice of projects:few can afford to do really adventurous research these days, it is just too risky. We stick with what we know, it is easier and it is safer.

Signs of Change?

We have to change things and, encouragingly, there are signs of reformation. There are attempts to improve the processes of publication (Adie et al., 2012). Governments have begun to worry that the huge investment in metrics has been a mistake (HEFCE, 2015). Changes are more feasible in private organizations, which can and should be more flexible than government institutions. For example, the Wellcome Trust in the United Kingdom has reduced the average time spent on preparing grant applications. There is a two or three step process of selection, starting with only a short written application. Applications for large grants have been reduced in length, only a brief “vision” of research aims is required. The Wellcome Trust has also begun to break away from too much reliance on metrics and brought back interviews. The HHMI in the United States has asked its fundees to select only their five best papers for evaluation, thus rewarding people for completing research projects rather than aiming to maximize the number of their publications. These are encouraging trends and I think they should be extended to other organizations and taken further (so we can get back to doing research!).

These last two agencies together with the Max Planck Society have started a new journal, eLife, that is open access and hopes to break the oligarchy of the three main biomedical journals. It has also begun to change a particular imbalance in the system: reviewers have gained too much power over authors. Power sharing was not the original purpose of peer review but, little by little, reviewers have taken control of authors’ research. Thus, if reviewers ask for an experiment, that request is usually passed on unvetted by the editor, even if this experiment has just occurred to the reviewer after little thought and is more of the sort of “wouldn’t it be nice if they did this extra experiment”… And the authors feel compelled to undertake the experiment because of their overweening desire to publish in that journal. Consequently, I believe that too much research is now dictated by anonymous reviewers who cannot know the full picture of the lab situation, other projects competing for time, personnel running out of grant support and all the other real life inputs into the formulation of an effective scientific strategy. This situation is being improved by eLife because they ask their reviewers to judge the paper as submitted and, while reviewers are encouraged to ask for checks and controls concerning the fabric of the work, they are not allowed to suggest more experiments that would simply extend the research.

Education that is founded on rational argument and scientific research must be the only feasible way to save our planet and our species. We therefore need to keep science at the forefront of all decision making and to continue to overcome the “hegemony of the supernatural” (Mellman & Warren, 2000). It is therefore crucial that the primary purposes of science to understand and to innovate can once again become the overt aims of researchers. To achieve this, our dependence on phony measurements and bureaucracy must be abandoned and our processes must again reward originality and risk taking. Universities and institutes must reduce investment in administration and increase investment in teaching and research. Then young people, many of whom still come into science with a sense of excitement, will no longer become disenchanted. We will need them for our survival.

Box 1.

Some data:

University of Cambridge, currently 65% of all graduate students in science are non-UK nationals (information from U of Cambridge).

In the United Kingdom, Chinese students account for 23% (26,860) of the total full-time postgraduate numbers in England, while UK students account for 26% (30,320).

Even in the United States, about half all graduate students in engineering are from other countries (Anderson, 2013). In Australia, about one-third of all graduate students come from elsewhere (University of Western Australia, 2013).

Box 2.

the U.S. National Institutes of Health has elevated “significance” to an explicit criterion in funding decisions. It requires that grant reviewers write a paragraph on “impact,” which it defines as the likelihood that the proposed work will have a sustained and powerful influence. Especially in fundamental research, which historically underlies the greatest innovation, the people doing the work often cannot themselves anticipate the ways in which it may bring human benefit. Thus, under the guise of an objective assessment of impact, such requirements invite exaggerated claims of the importance of the predictable outcomes—which are unlikely to be the most important ones. This is both misleading and dangerous.

Kirschner (2013)

Box 3.

In Australia: “Research as much as teaching has been eroded by the bureaucratic university…. The proliferation of research management bureaucracies in universities over the period is a parallel symptom of a system that values procedure rather than productivity.” (Murphy, 2013)

In America: “American academics on average teach twice as much as they research. They teach for eleven hours a week. In colleges they research two hours a week and in research universities five hours a week. The rest of the time (twenty hours plus) is made up of committee meetings, e-mail correspondence and various professional obligations. Time disappears in this vortex—time that academics once would have spent interacting with students, and students with them. Bureaucratic time now continuously eats away at the vocation of the university—until one day it will disappear, unless a new model of the university is conceived.” (Murphy, 2013; and see Arum, 2011 for a detailed analysis).

In the UK: “a study by Brighton University, which looked at the remuneration between 1998 and 2009 of 193 vice-chancellors leading 95 UK institutions, has uncovered a real-term pay increase of 59%. On average, vice-chancellors received pay awards that were four times those of lecturers and the differential has widened”. Neil Gorman, former vice-chancellor of Nottingham Trent University, received £623,000 in 2013 (Henry, 2015); a typical University Professor receives about one-tenth of that salary and a graduate student about one fortieth. In Oxford, “The university’s central administrative staff is now almost three times what it was 15 years ago. There was no similar increase in full-time academic staff ” (McDonagh, 2015).

Box 4.

A true story of impact: Henry Disney (Department of Zoology, University of Cambridge, rhld2@hermes.cam.ac.uk) was on leave from West Cameroon in 1969 in the City of Bath, UK: “I was struck by one respect in which Bath was less hygienic than the large market town of Kumba. In the latter, people openly urinated against the boundary wall of the market and deposited faeces in the drainage channels. Likewise, dogs deposited excrement everywhere. However, within a few hours the dung beetles rolled up the dung into balls and buried them along with their eggs. By contrast one could not walk 10 metres along the streets of the salubrious City of Bath without passing a deposit of dog faeces. I therefore recorded which flies were visiting these deposits, which fly larvae were developing in them and which of these (such as the lesser housefly Fannia canicularis) were also commonly found in the kitchens of our houses. I suggested that the prevalence of dog dung in English cities was directly related to the frequency of human enteric infections in England (Disney, 1972). Zakaria Erzinclioglu read my article and carried out a more extensive study in a north London suburb, confirming and extending my findings (Erzinclioglu, 1981).

The curiosity of neighbours observing him collecting samples gave rise to his work coming to the attention of one of the London evening newspapers. In the subsequent correspondence it was pointed out that a parasitic infection of the eye, then frequent among children, was also commonly derived from dog dung. The result was the introduction of the requirement for dog owners to collect up their dog’s excrement and for local councils to provide bins for its deposition. The end result has been a reduction in the infections derived from dog dung. This undoubted impact of my initial note is not reflected in the citations or impact factors used to measure the value of a scientific publication. Indeed, according to these measures my note was of no significance whatsoever!”