Important Science—It's All about the SPIN

Abstract

The importance of a scientific finding can be difficult to ascertain because it requires both subjective judgment and foresight. We propose criteria, based upon whether a discovery is sizeable (S), practical (P), integrated (I), and new (N), that can be used individually or collectively to systematically assess the importance of a finding.

“It is clear that without the distinction between the important and the unimportant at our disposal, mankind could neither adequately understand, successfully teach, or effectively practice science.”

—Nicholas Rescher (16)

Of all the epithets attached to the word “science,” perhaps the most desirable is “important.” Reviewers of Infection and Immunity and other journals are asked to assess the importance of a manuscript, which often becomes the critical parameter for deciding the manuscript's fate. In the parlance of current scientific literature, the word “priority” may be used as a synonym for importance. Although the two words do not mean precisely the same thing, both are comparative terms that describe the relationship between a work and other competing works. In previous essays, we have considered the modifiers “descriptive” and “mechanistic” (2, 3). We now consider the problem of “importance,” recognizing that this most critical of descriptors is also the most elusive, for it relies on judgment. We note that reviewers for Infection and Immunity are asked to rate manuscripts in terms of both general interest and significance to the field, suggesting a nuanced meaning to the criterion of importance.

The American Heritage Dictionary defines “important” as “strongly affecting the course of events or the nature of things” (http://dictionary.reference.com/browse/Important). This implies that the outcome of a scientific discovery or publication on subsequent events determines its importance. Hence, there is a separation between scientific quality and outcome, a distinction that raises interesting issues. For example, authors frequently complain that journals are biased against studies that yield negative results, a practice known as “publication bias” (15); this is unquestionably true and can be attributed to the greater perceived importance of positive studies. Moreover, the focus on outcome means that any judgment on assessing the importance of a scientific work is a function of time. This is apparent to any student of the history of science, which contains many instances in which the importance of a scientific finding was underappreciated or overvalued at the time. An oft-cited example of this phenomenon is the work of Mendel, whose importance was not recognized in the decades following publication in an obscure journal (14) but was subsequently rediscovered to become a foundation of the nascent science of genetics. In every field, there are examples of papers that made little splash when first published yet in retrospect were found to contain important information with a substantial impact on later work. Hence, reviewers are asked to judge the potential of a manuscript without the advantage of hindsight that only time can provide. It would seem at first glance that any assessment of importance must be a highly subjective and error-prone process. Nevertheless, even without a crystal ball one may systematically attempt to estimate the importance of scientific work. Here we propose four criteria by which one may assess scientific importance, forming the acronym “SPIN.” Important science should be sizeable (S), practical (P), integrated (I), and new (N), and these criteria can be used individually or collectively to estimate the importance of a manuscript.

Sizeable.

In scientific investigations, the parameter of size can refer to the magnitude of a problem or the size of a field. Consider two pathogenic microbes that cause comparable levels of morbidity and mortality in susceptible hosts: Mycobacterium microti and Mycobacterium tuberculosis. Both belong to the M. tuberculosis complex. However, M. microti is found in rodents and llamas and only very rarely in humans, whereas one-third of the world's population is infected with M. tuberculosis. This is paralleled by the more than 175,000 publications on tuberculosis listed in PubMed, compared with fewer than 200 on M. microti. Hence, following our criteria, a scientific observation would be considered more important if discovered pertaining to M. tuberculosis rather than to M. microti, because the problem of tuberculosis and the number of tuberculosis researchers are so much more sizeable. Acknowledgment of the contribution of size to the overall importance of scientific work is evident in the introductions of many Infection and Immunity papers, in which one often finds descriptions of the prevalence of microbial diseases. In gauging importance, size matters, but those of us who work on rare microbes and/or inhabit small scientific fields need not despair, for other components of the SPIN factor can compensate for size.

Practical.

Scientific findings with practical utility have great importance because they provide benefits to society. For the public and the political establishment, which currently supports most scientific research, the tangible rewards of science are the dominant measure of its importance. However, utility can come in many forms, and here the value system of the reviewer is paramount in assigning importance. For instance, a discovery can have utility in a theoretical context if it facilitates understanding, even if that information cannot be immediately translated into practical applications. Furthermore, history has repeatedly provided examples in which practical applications arise serendipitously from basic research initiated for other purposes (18). Discoveries may have enormous practical utility even though they are not conceptually new. For example, earlier work showed that conjugation of the Haemophilus influenzae capsular polysaccharide to a protein carrier results in an effective vaccine (5, 7). This breakthrough had a dramatic impact on the incidence of invasive H. influenzae infections in the United States (17). Subsequent research led to the development of Streptococcus pneumoniae capsular polysaccharides conjugated to carrier proteins, allowing the prevention of invasive pneumococcal infections in infants and toddlers (8). From a definitional standpoint and public perspective, the pneumococcal research is clearly of high importance because it impacts many susceptible individuals. However, despite its importance, the finding lacks conceptual originality, and individuals who value newness over practical utility might dismissively label the latter work as derivative. This example illustrates that although assigning importance to science has an inevitable subjective quality, one may nevertheless quantify importance by considering the different components of the SPIN factor independently.

Integrated.

Knowledge builds upon knowledge, and all scientific knowledge is interconnected. New information must become integrated with prior knowledge. The importance of a new discovery is therefore dependent on context and the readiness of existing knowledge to allow integration of the new information, which must sometimes await new tools or further understanding. Science that is initially underappreciated because it is ahead of its time has been referred to as “premature” (12). By “premature,” we do not mean that a work was published before it was ready for publication but rather that the results were presented before the field could properly grasp their significance. Arrhenius' warnings of global warming in 1896 and Wegener's proposal of continental drift in 1915 are two classic examples of premature science (9). When Avery, MacLeod, and McCarty identified DNA as the material responsible for heredity in 1944 (1), the initial reaction to this transcendent discovery was surprisingly muted (19). This is in part because prevailing concepts of DNA were not consistent with the coding of specific information, as DNA was then believed to consist of a monotonously repeating polymer of identical tetranucleotides (19). Only after Chargaff showed that DNA bases were not necessarily present in equal proportions (4) and Hershey and Chase demonstrated internalization of phage DNA by bacteria (11) were the observations of Avery et al. fully appreciated and integrated with existing knowledge.

Science may be prioritized on the basis of its level of integration. If one envisages scientific knowledge as a branching tree, then basic processes tend to occupy deeper positions within scientific branches and have broader implications. A discovery may be viewed as more important when it is situated more deeply within the tree of knowledge or possesses more interconnections with other findings. To illustrate this with an example, consider the discovery of nitric oxide biosynthesis from the amino acid l-arginine. This insight facilitated the elucidation of nitric oxide's role in vasoregulation, neurotransmission, signal transduction, and host defense (6). Knowledge of nitric oxide chemistry is located at a deeper level than knowledge of the actions of nitric oxide in a specific type of infection and, consequently, depending on the perspective of the reviewer, might be considered more important.

New.

The parameter of newness reflects the time since information has come into existence. Newness plays a key role in the economics of science, where priority of discovery can confer great rewards on the scientist (20). In most fields, there is no prize for coming in second or third, except for the knowledge that confirmation and validation reinforce the foundation of science. Although newness and novelty could be considered synonymous, we will avoid the word “novel” and its derivatives since it is used so frequently in the current scientific literature that we consider it tired and in need of rest (a search using the word “novel” in PubMed produced over 400,000 citations, whereas “newness” yielded less than 100). In incorporating newness into our definition of importance, it is apparent that this parameter is different than size, practical utility, and integration because, unlike the other qualities, newness alone does not necessarily imply that a discovery affects “the course of events or the nature of things” (http://dictionary.reference.com/browse/Important). In fact, the newness of a discovery appears to be more important to the personal satisfaction of scientists than to the assessment of importance. This leads to the uncomfortable realization that new things are not necessarily important, and older findings may subsequently be judged of greater importance, as in the case of Mendel's work. The emphasis on the term “novel” in manuscripts and grants as a measure of importance may therefore be somewhat misplaced. Nevertheless, the newness of information is relevant in judging the importance of a finding because immediacy can temper the other SPIN parameters. Fleming's initial discovery of penicillin production by the obscure saprophytic fungus Penicillium chrysogenum was undeniably important, but largely in hindsight, as treatment applications were not realized until more than a decade later. Although other fungi and actinomycetes have subsequently been shown to produce antibiotics, Fleming's observations have retained special importance because they were the first. Ironically, newness can become more significant over time, as a greater value is placed on priority once the importance of a finding is recognized.

Surrogate measures of importance.

Most scientists would agree that the Nobel Prize represents the highest accolade for important scientific work. Hence, the analysis of science recognized by Nobel Prizes can provide insight into the validity of the SPIN parameters, their relative value, and their changes over time. Historically, the emphasis on size (S) is apparent in many Nobel Prizes in Medicine and Physiology that recognize progress against diseases affecting large numbers of people. For example, in 2008, Nobel Prizes recognized the association of human immunodeficiency virus and human papilloma virus with AIDS and cervical cancer, two diseases that afflict millions each year, while the association of human T-cell lymphotropic virus type 1 with the much rarer disease tropical spastic paraparesis did not receive comparable recognition, even though comparable associations with a specific virus were shown for all three conditions. Size is also undoubtedly a factor in the Nobel Prizes awarded for recognition of mechanisms of malaria transmission (1902), tuberculosis etiology and therapy (1905 and 1952), typhus control (1928), penicillin (1946), poliovirus (1954), and the association of Helicobacter pylori with peptic ulcer disease (2005), as each of these awards recognized progress against diseases affecting many individuals. No Nobel Prizes have been given for breakthroughs related to infections involving relatively small numbers of individuals, such as Kaposi's sarcoma, Whipple's disease, and Lyme disease, or the discovery of the first effective antifungal agent, amphotericin B.

Of the remaining parameters, integration (I) plays a dominant role, with the overwhelming majority of Nobel Prizes in Medicine and Physiology being awarded for basic discoveries that have had great consequences for diverse lines of life science research. The contribution of practical application (P) is apparent in Nobel Prizes awarded for techniques with wide applicability, such as the development of the radioimmunoassay (1977) and hybridoma technology (1984). In contrast to the Medicine and Physiology prize, technological advances have been recognized frequently by the Chemistry prize, including protein sequencing (1958), polarography (1959), DNA sequencing (1980), nuclear magnetic resonance (1991), PCR (1993), site-directed mutagenesis (1993), and green fluorescence protein applications in biology (2008). Newness (N) pervades all discoveries but is clearly apparent when given for the discovery of a specific phenomenon leading to a wholesale paradigm shift, such as “slow viruses” or prions (1977 and 1997), ribozymes (1989), and nitric oxide biosynthesis by vascular tissues (1998). We note that newness can sometimes make a finding more difficult to integrate with existing science. For example, the discovery of prions as infectious agents challenged the central dogma of DNA → RNA → protein and could not be integrated into mainstream biology until subsequent studies provided a mechanism by which a proteinaceous infectious agent itself by transmitting information, leading to an aberrant folding state.

The evolution of the Nobel Prize also provides some insights into the perception of importance over time. As examples of science no longer considered as important today, the 1926 prize to Johannes Fibiger for his discovery that the nematode Spiroptera carcinoma causes gastric cancer and the 1949 prize to Egaz Moniz for his work on prefrontal lobotomies come to mind (pun intended). Early in the 20th century, Nobel Prizes in Medicine and Physiology were often given for contributions that had a great impact on health, such as serum therapy (1901), malaria transmission (1902), phototherapy (1903), and surgical advances (1912), suggesting a great emphasis by the selection committee on practical application. In contrast, later in the century the majority of prizes were for very basic discoveries that often could not immediately be realized into useful applications. Regarding the importance of time in the ultimate judgment of importance, we note that Alfred Nobel stipulated in his will that the prize should be awarded annually “to those who, during the preceding year, shall have conferred the greatest benefit on mankind” (http://nobelprize.org/alfred_nobel/will/index.html). Although this sentence is sufficiently ambiguous to allow some room for interpretation, the emphasis on the preceding year does suggest an initial intention for rewarding immediacy in discovery. However, Nobel Prizes are nowadays almost always awarded many years after the original discovery, because importance takes time to become evident.

For less spectacular discoveries, one must rely on surrogate measures of importance, such as the number of times that a paper is cited by others. Rescher has noted that the distribution of numbers of citations and scientific quality are described by similar exponential functions, suggesting that the former is a reasonable measure of the latter (16). However, he hastens to add, “Of course, no more than an estimate is at issue here. For it has to be acknowledged that in view of the ever moving boundary lines of the frontiers of knowledge and the shifting ebb and flow of fashions in matters of theorizing have the unavoidable consequence that importance as best we can judge it is not a fixity but an ongoingly varying parameter.” Nevertheless, we cannot resist mentioning here that Infection and Immunity is the most cited infectious diseases journal in the world (2008 Journal Citation Reports, Thomson Reuters, 2009).

Assessment of importance is subjective and imperfect.

Ultimately, the assessment of importance has a large subjective component that reflects the values, experience, interest, knowledge, and biases of the reviewer. Different individuals are inclined to give different weights to the SPIN parameters. For example, scientists concerned with global health and vaccines may have their SPIN algorithm altered to S²P²IN to reflect these priorities, while for scientists interested in basic research the algorithm may be SPI³N. The essential point is that a manuscript's importance may be assessed from different perspectives. Emphasis on S, P, I, or N is in the eye of the beholder, and it behooves one to be respectful of divergent views. The difficulties inherent in evaluating the importance of a manuscript should not deter reviewers from attempting their own assessments, with the important caveat that one should remain humble, for there is great uncertainty in the process. The journal PLoS ONE has decided to eliminate the assessment of importance from its review process altogether: “PLoS ONE will…publish all papers that are judged to be technically sound. Judgments about the importance of any particular paper are then made after publication by the readership (who are the most qualified to determine what is of interest to them)” (http://www.plosone.org/static/information.action). Since PLoS ONE is an open-access journal that is available to anyone with an Internet connection, this policy is revolutionary in the sense that it returns the assessment of importance from the editorial elite to the people. Kierkegaard has reminded us, “Life must be understood backwards, but…it must be lived forwards” (13), and as we have tried to emphasize, true importance can be fully recognized only in hindsight. It is also worthwhile to note that although authors, reviewers, and editors can disagree on the relative importance of a manuscript, the work was considered of sufficient importance by the investigators to warrant their time and resources. Hence, all manuscript submissions should be treated with respect, for they are important to someone.

The paucity of writings on the topic suggests that importance is considered to be self-evident by most scientists. However, the deconvolution of importance into its SPIN parameters suggests that a quantitative approach to the problem is possible and even worthy of future investigation. A departure from purely subjective assessments toward the establishment of quantitative criteria for importance could eventually provide the basis for a system of manuscript prioritization. Authors might also be tempted to use this approach to put the most favorable “SPIN” on their work. However, they should first recall the words of Frank Harold (10): “Every novel idea in science passes through three stages. First people say it isn't true. Then they say it's true but not important. And finally they say it's true and important—but not new.”