Scientific Utopia: II. Restructuring Incentives and Practices to Promote Truth Over Publishability

Abstract

An academic scientist’s professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results. Prior reports demonstrate how these incentives inflate the rate of false effects in published science. When incentives favor novelty over replication, false results persist in the literature unchallenged, reducing efficiency in knowledge accumulation. Previous suggestions to address this problem are unlikely to be effective. For example, a journal of negative results publishes otherwise unpublishable reports. This enshrines the low status of the journal and its content. The persistence of false findings can be meliorated with strategies that make the fundamental but abstract accuracy motive—getting it right—competitive with the more tangible and concrete incentive—getting it published. This article develops strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.

The chief thing which separates a scientific method of inquiry from other methods of acquiring knowledge is that scientists seek to let reality speak for itself, and contradict their theories about it when those theories are incorrect. … Scientific researchers propose hypotheses as explanations of phenomena, and design experimental studies to test these hypotheses via predictions which can be derived from them. These steps must be repeatable, to guard against mistake or confusion in any particular experimenter. … Scientific inquiry is generally intended to … document, archive and share all data and methodology so they are available for careful scrutiny by other scientists, giving them the opportunity to verify results by attempting to reproduce them.

— From http://en.wikipedia.org/wiki/Scientific_method (Scientific Method, n.d.)

A True Story of What Could Have Been

Two of the present authors, Matt Motyl and Brian A. Nosek, share interests in political ideology. We were inspired by the fast growing literature on embodiment that demonstrates surprising links between body and mind (Markman & Brendl, 2005; Proffitt, 2006) to investigate embodiment of political extremism. Participants from the political left, right, and center (N = 1,979) completed a perceptual judgment task in which words were presented in different shades of gray. Participants had to click along a gradient representing grays from near black to near white to select a shade that matched the shade of the word. We calculated accuracy: How close to the actual shade did participants get? The results were stunning. Moderates perceived the shades of gray more accurately than extremists on the left and right (p = .01). Our conclusion: Political extremists perceive the world in black and white figuratively and literally. Our design and follow-up analyses ruled out obvious alternative explanations such as time spent on task and a tendency to select extreme responses. Enthused about the result, we identified Psychological Science as our fallback journal after we toured the Science, Nature, and PNAS rejection mills. The ultimate publication, Motyl and Nosek (2012), served as one of Motyl’s signature publications as he finished graduate school and entered the job market.

The story is all true, except for the last sentence; we did not publish the finding. Before writing and submitting, we paused. Two recent articles have highlighted the possibility that research practices spuriously inflate the presence of positive results in the published literature (John, Loewenstein, & Prelec, 2012; Simmons, Nelson, & Simonsohn, 2011). Surely ours was not a case to worry about. We had hypothesized it; the effect was reliable. But we had been discussing reproducibility, and we had declared to our lab mates the importance of replication for increasing certainty of research results. We also had an unusual laboratory situation. For studies that could be run through a Web browser, data collection was very easy (Nosek et al., 2007). We could not justify skipping replication on the grounds of feasibility or resource constraints. Finally, the procedure had been created by someone else for another purpose, and we had not laid out our analysis strategy in advance. We could have made analysis decisions that increased the likelihood of obtaining results aligned with our hypothesis. These reasons made it difficult to avoid doing a replication. We conducted a direct replication while we prepared the manuscript. We ran 1,300 participants, giving us .995 power to detect an effect of the original effect size at α = .05. The effect vanished (p = .59).

Our immediate reaction was “why the #&@! did we do a direct replication?” Our failure to replicate does not make definitive the conclusion that the original effect is false, but it raises enough doubt to make reviewers recommend against publishing. Any temptation to ignore the replication and publish the original was squashed only by the fact that our lab mates knew we ran a replication. We were accountable to them. The outcome—a dead or delayed paper—is unfortunate for our career advancement, particularly Motyl’s as he prepared for the job market.

Incentives for surprising, innovative results are strong in science. Science thrives by challenging prevailing assumptions and generating novel ideas and evidence that push the field in new directions. We cannot expect to eliminate the disappointment that we felt by “losing” an exciting result. That is not the problem or at least not one for which the fix would improve scientific progress. The real problem is that the incentives for publishable results can be at odds with the incentives for accurate results. This produces a conflict of interest. The conflict may increase the likelihood of design, analysis, and reporting decisions that inflate the proportion of false results in the published literature.¹ The solution requires making incentives for getting it right competitive with the incentives for getting it published. Without that, the lesson that we could take away from our experience with “Political extremists do not perceive shades of gray, literally” is to never, ever do a direct replication again. The purpose of this article is to make sure that such a lesson does not stick.

How Evaluation Criteria Can Increase the False Result Rate in Published Science

Publishing is “the very heart of modern academic science—at levels ranging from the epistemic certification of scientific thought to the more personal labyrinths of job security, quality of life, and self-esteem” (Mahoney, 1985, p. 30). Publication influences hiring, salary, promotion, tenure, and grant decisions (Miller & Serzan, 1984; Peters & Ceci, 1982). It is also a criterion for the evaluation and ranking of departments and universities (Ball, 2005; Haslam et al., 2008; Nosek et al., 2010; Ostriker, Holland, Kuh, & Voytuk, 2009; Phillips, 2007). Expectations for publishing have expanded to faculty at institutions that do not have graduate research programs, to graduate students seeking employment, and even to undergraduates applying to top programs for graduate study. With an intensely competitive job market, the demands for publication might seem to suggest a specific objective for the early-career scientist: publish as many articles as possible in the most prestigious journals that will accept them (Martin, 1992; Sovacool, 2008).²

Some Things Are More Publishable Than Others

Even if a researcher conducts studies competently, analyzes the data effectively, and writes up the results beautifully, there is no guarantee that the report will be published. Part of the process—peer review—is outside of the researcher’s control. In the social and behavioral sciences, rejection rates of 70%–90% by journals are common (American Psychological Association, 2010; Zuckerman & Merton, 1971). High demand for limited space means that authors must strive to meet all publishing criteria so that an editor will do the unusual act of accepting the manuscript. As such, success in publishing is partly a function of social savvy of knowing what is publishable and empirical savvy in obtaining publishable results.

A Disconnect Between What Is Good for Scientists and What Is Good for Science

On its own, the fact that publishing is essential to success is just a fact of the trade. Running faster defines better sprinters; conducting more high-impact research defines better scientists. The research must be published to have impact. And yet, publishing is also the basis of a conflict of interest between personal interests and the objective of knowledge accumulation. The reason? Published and true are not synonyms. To the extent that publishing itself is rewarded, then it is in scientists’ personal interests to publish, regardless of whether the published findings are true (Hackett, 2005; Martin, 1992; Sovacool, 2008).

The present authors have accuracy motives—to learn and publish true things about human nature. We also have professional motives—to succeed and thrive professionally. Our incentives for professional success can be at odds with scientific practices that improve confidence in the truth of findings. Strong professional motives produce motivated reasoning to arrive at the conclusion we desire, even at the expense of accuracy (Kunda, 1990).

At the extreme, the present authors could lie: make up findings or deliberately alter results. However, detection of such behavior destroys the scientist’s reputation. This is a strong incentive against it, and—regardless of incentives—most resist such behavior because it is easy to identify as wrong (Fanelli, 2009). We have enough faith in our values to believe that we would rather fail than fake our way to success. Less simple to put aside are ordinary practices that can increase the likelihood of publishing false results, particularly those practices that are common, accepted, and even appropriate in some circumstances. Because we have directional goals for success, we are likely to bring to bear motivated reasoning to justify research decisions in the name of accuracy, when they are actually in service of career advancement (Fanelli, 2010a). Motivated reasoning is particularly influential when the situation is complex, the available information is ambiguous, and legitimate reasons can be generated for multiple courses of action (Bersoff, 1999; Boiney, Kennedy, & Nye, 1997; Kunda, 1990).

Motivated reasoning can occur without intention. We are more likely to be convinced that our hypothesis is true, accepting uncritically when it is confirmed and scrutinizing heavily when it is not (Bastardi, Uhlmann, & Ross, 2011; Ditto & Lopez, 1992; Lord, Ross, & Lepper, 1979; Pyszczynski & Greenberg, 1987; Trope & Bassok, 1982). With flexible analysis options, we are more likely to find the one that produces a more publishable pattern of results to be more reasonable and defensible than others (Simmons et al., 2011; Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011). Once we obtain an unexpected result, we are likely to reconstruct our histories and perceive the outcome as something that we could have, even did, anticipate all along—converting a discovery into a confirmatory result (Fischoff, 1977; Fischoff & Beyth, 1975). And even if we resist those reasoning biases in the moment, after a few months, we might simply forget the details, whether we had hypothesized the moderator, had good justification for one set of exclusion criteria compared with another, and had really thought that the one dependent variable that showed a significant effect was the key outcome. Instead, we might remember the gist of what the study was and what we found (Reyna & Brainerd, 1995). Forgetting the details provides an opportunity for reimagining the study purpose and results to recall and understand them in their best (i.e., most publishable) light. The reader may, as we do, recall personal examples of such motivated decisions—they are entirely ordinary products of human cognition.

Novelty and Positive Results Are Vital for Publishability but Not for Truth

The primary objective of science as a discipline is to accumulate knowledge about nature. Learning something new advances that goal; reaffirming something known does not. As Schmidt (2009) noted, “within the social sciences, only the discovery of a new fact is credited” (Schmidt, 2009, p. 95; see also Lindsay & Ehrenberg, 1993). Innovation—producing new ideas, methods, and evidence—is the basis for scientific advancement. As such, successful scientists are those with a productive program of research that reveals new insights about nature.

Related to this, direct replication of another’s study procedures to confirm the results is uncommon in the social sciences (Collins, 1985; Mahoney, 1985; Schmidt, 2009). Neuliep and Crandall (1990; see also Madden, Easley, & Dunn, 1995; Neuliep & Crandall, 1993) found that 94% of journal editors agreed that “replication studies were not included as examples of research encouraged for submission in the editorial policy” (p. 87), and a large majority preferred to publish new findings rather than replications because the latter were deemed “not newsworthy” and a “waste of space.”

These comments are revealing about the relative valuation of new ideas versus truth. Publishing a result does not make it true. Many published results have uncertain truth value. Dismissing a direct replication as “we already knew that” is misleading; the actual criticism is “someone has already claimed that.” The former indicates that the truth value is known; the latter indicates that someone has had the idea and perhaps provided some evidence. Replication is a means of increasing the confidence in the truth value of a claim. Its dismissal as a waste of space incentivizes novelty over truth. As a consequence, when a false result gets into the published literature, it is difficult to expel. There is little reinforcement for conducting replications to affirm or reject the validity of prior evidence and few consequences for getting it wrong. The principal incentive is publication.

Further, in the dominant model of null hypothesis significance testing (Cohen, 1994; Nickerson, 2000; Rosnow & Rosenthal, 2009; Rozeboom, 1960), the nominal false-positive rate of α = .05 has become a de facto criterion for publishing. Like publishing a result, achieving a positive result does not mean that the effect is true, nor does it indicate the probability of its truth (Bakan, 1966; Cohen, 1994; Frick, 1996; Oakes, 1986). Moreover, most published results across scientific disciplines, and over 90% for psychology in particular, are positive effects (Fanelli, 2010b, 2012). This appears to have been true for more than 50 years (Sterling, 1959; Sterling, Rosenbaum, & Weinkam, 1995). If anything, the rate of positive results is increasing (Fanelli, 2012). Greenwald (1975) showed that psychologists perceive a bias against negative (null) results and are less likely to continue pursuing or report negative results and that journals are less likely to publish negative as compared with positive results. As a consequence, negative results are less likely to appear in the literature.

In summary, the demands for novelty and positive results create incentives for (a) generating new ideas rather than pursuing additional evidence for or against ideas suggested previously; (b) reporting positive results and ignoring negative results (Fanelli, 2012; Greenwald, 1975; Ioannidis & Trikalinos, 2007; Rosenthal, 1979); and (c) pursuing design, reporting, and analysis strategies that increase the likelihood of obtaining a positive result in order to achieve publishability (Fanelli, 2010a; Ioannidis, 2005; John et al., 2012; Simmons et al., 2011; Wicherts, Bakker, & Molenaar, 2011; Wong, 1981; Young, Ioannidis, & Al-Ubaydli, 2008). This paints a bleak picture of the incentive structures in science. Simultaneously, we believe that a lot of very good science gets done and does so with incentives and practices that facilitate knowledge accumulation. We believe that “good behaviors” can be promoted further with some adaptations to normative scientific culture and practices and incentives structures that promote and sustain those practices. Before discussing solutions, we briefly review some of the practices that may interfere with efficiency in knowledge accumulation.

Practices That Can Increase the Proportion of False Results in the Published Literature

Other contributions have detailed a variety of practices that can increase publishability but might simultaneously decrease validity (Fanelli, 2010a; Giner-Sorolla, 2012; Greenwald, 1975; Ioannidis, 2005; John et al., 2012; Kerr, 1998; Martinson, Anderson, & Devries, 2005; Rosenthal, 1979; Simmons et al., 2011; Sovacool, 2008; Young et al., 2008). The following are practices that are justifiable sometimes but can also increase the proportion of published false results: (a) leveraging chance by running many low-powered studies, rather than a few high-powered ones³ (Ioannidis, 2005); (b) uncritically dismissing “failed” studies as pilot tests or because of methodological flaws but uncritically accepting “successful” studies as methodologically sound (Bastardi et al., 2011; Lord, Ross, & Lepper, 1979); (c) selectively reporting studies with positive results and not studies with negative results (Greenwald, 1975; John et al., 2012; Rosenthal, 1979) or selectively reporting “clean” results (Begley & Ellis, 2012; Giner-Sorolla, 2012); (d) stopping data collection as soon as a reliable effect is obtained (John et al., 2012; Simmons et al., 2011); (e) continuing data collection until a reliable effect is obtained (John et al., 2012; Simmons et al., 2011); (f) including multiple independent or dependent variables and reporting the subset that “worked” (Ioannidis, 2005; John et al., 2012; Simmons et al., 2011); (g) maintaining flexibility in design and analytic models, including the attempt of a variety of data exclusion or transformation methods, and reporting a subset (Gardner, Lidz, & Hartwig, 2005; Ioannidis, 2005; Martinson et al., 2005; Simmons et al., 2011); (h) reporting a discovery as if it had been the result of a confirmatory test (Bem, 2003; John et al., 2012; Kerr, 1998); and, (i) once a reliable effect is obtained, not doing a direct replication (Collins, 1985; Schmidt, 2009; in an alternate timeline, see Motyl & Nosek, 2012).

The lack of interest in replication is striking given its centrality to science. The scientific method differentiates itself from other approaches by publicly disclosing the basis of evidence for a claim. In Roger Bacon’s cycle of scientific research—observation, hypothesis, experimentation, and verification—disclosure is critical for the fourth step (Bacon, 1267/1859). Other scientists must be able to independently replicate and verify, qualify, or disconfirm the original scientist’s results. This allows scientists to work independently toward a shared objective—accumulating knowledge—without relying on accuracy or trust in any single source. In principle, open sharing of methodology means that the entire body of scientific knowledge can be reproduced by anyone. This democratizing function for acquiring knowledge made replication a central principle of the scientific method from before Bacon to the present (e.g., al Haytham, 1021, as translated by Sabra, 1989; Jasny, Chin, Chong, & Vignieri, 2011; Kuhn, 1962; Lakatos, 1978; Popper, 1934; Rosenthal, 1991; Schmidt, 2009).⁴ Replication is so central to science that it may serve as a “demarcation criterion between science and nonscience” (Braude, 1979, p. 2). Even so, direct replications are seldom pursued in the behavioral sciences perhaps because they are deemed unpublishable as failures of novelty (Collins, 1985; Reid, Soley, & Wimmer, 1981; Schmidt, 2009).

Many of the behaviors that can increase the rate of false results are common because there are good reasons to do them sometimes. The key challenge, however, is that there are good reasons to do them sometimes. For example, when wading into a new phenomenon, having multiple dependent variables can be a more efficient use of resources because there is little existing knowledge for informing which one will be influenced by the independent variable. Simultaneously, this practice inflates the likelihood of false positives. This fact merely increases the importance of replicating the initial finding and disclosing that the initial study included multiple dependent variables and why. The multiple dependent variable design decision is sensible, not embarrassing. The disclosure just provides evaluators with an accurate basis for computing their confidence in the results (Simmons et al., 2011).

The consequences of the publishability-improving practices listed above can be severe. Ioannidis (2005) gave his review of reproducibility a dire title: “Why Most Published Research Results Are False.” Greenwald (1975) estimated the false positive (Type I error) rate at 30% based only on whether researchers submit and editors accept negative findings. Bayer HealthCare reported that of 67 attempts to reproduce published findings in fields of oncology, women’s health, and cardiovascular disease, about 25% of the replications clearly reproduced the published evidence (Prinz, Schlange, & Asadullah, 2011). This low rate was not attributable to publishing journal prestige, closeness of the replication attempt, or the subdiscipline of investigation. Likewise, Begley and Ellis (2012) reported an effort by Amgen to replicate 53 landmark studies of basic research for cancer treatment. Just 6 (11%) of the replications confirmed the original, published result. They noted, “Some non-reproducible preclinical papers had spawned an entire field, with hundreds of secondary publications that expanded on elements of the original observation, but did not actually seek to confirm or falsify its fundamental basis” (p. 532). Finally, an informal assumption among venture capital firms for biomedical research is that more than 50% of published studies from academic laboratories cannot be replicated in industrial laboratories (Osherovich, 2011). In these latter cases, the industrial laboratories pursued replications of academic research because there are considerable incentives for doing so. Investing hundreds of thousands of dollars on a new treatment that is ineffective is a waste of resources and an enormous burden to patients in experimental trials. By contrast, for academic researchers, there are few consequences for being wrong. If replications get done and the original result is irreproducible, nothing happens.

Strategies That Are Not Sufficient to Stop the Proliferation of False Results

False effects interfere with knowledge accumulation. If common scientific practices are increasing the rate of false effects, then changing some practices could improve efficiency in scientific progress. Of course, science is progressing and knowledge is accumulating. Revisions to scientific practices ought not to disrupt those practices that are working well. They should selectively target dysfunctional incentives and practices. Before turning toward our suggested improvements, we briefly review innovations that have been suggested or attempted that are not, in our view, sufficient to address the proliferation of false effects.

Conceptual replication

Although direct replication is rare in the social and behavioral sciences, conceptual replication is more common (Collins, 1985; Schmidt, 2009). As opposed to direct replication, which reproduces the conditions of the original demonstration as faithfully as possible, conceptual replication involves deliberately changing the operationalization of the key elements of the design such as the independent variable, dependent variable, or both. Conceptual replications allow abstraction of the explanation for an effect from the particulars of a given operationalization to the theoretical variable that operationalizations attempt to manipulate or assess (Schmidt, 2009). This is vitally important in science when the constructs of interest are unobservable (Edge, 1985). Demonstrating the same effect with multiple operationalizations provides confidence in its conceptual interpretation.

Often essential for theoretical understanding, conceptual replication is not an effective replacement for direct replication. As Schmidt (2009, p. 95) put it, “Whereas a direct replication is able to produce facts, a conceptual replication may produce understanding.” Because features of the original design are changed deliberately, conceptual replication is used only to confirm (and abstract) the original result, not to disconfirm it. A successful conceptual replication is used as evidence for the original result; a failed conceptual replication is dismissed as not testing the original phenomenon (Braude, 1979).⁵ As such, using conceptual replication as a replacement for direct replication is the scientific embodiment of confirmation bias (Nickerson, 1998).

The mythology of science as self-correcting

Science is self-correcting (Merton, 1942, 1973). If a claim is wrong, eventually new evidence will accumulate to show that it is wrong and scientific understanding of the phenomenon will change. This is part of the promise of science—following the evidence where it leads, even if it is counter to present beliefs. We do believe that self-correction occurs. Our problem is with the word “eventually.” The myth of self-correction is recognition that once published, there is no systemic ethic of confirming or disconfirming the validity of an effect. False effects can remain for decades, slowly fading or continuing to inspire and influence new research (Prinz et al., 2011). Further, even when it becomes known that an effect is false, retraction of the original result is very rare (Budd, Sievert, & Schultz, 1998; Redman, Yarandi, & Merz, 2008). Researchers who do not discover the corrective knowledge may continue to be influenced by the original, false result. We can agree that the truth will win eventually, but we are not content to wait.

Journals devoted to publishing replications or negative results

An obvious strategy for addressing the lack of interest in publishing negative results or replications is to start journals devoted to publishing them (e.g., http://www.jasnh.com/; http://www.jnr-eeb.org/; http://www.journalofnullresults.com/). Unfortunately, we believe this model is doomed to fail. Defining a journal based on negative results or replications is self-defining it as a low importance outlet. For example, the Journal of Personality and Social Psychology has an explicit policy against publishing direct replications, communicating their lower status (Aldhous, 2011). It is not in authors’ interest to publish in a journal that is defined as publishing articles that no other journal will publish.

Education campaigns emphasizing the importance of replication and reporting negative results

If editors, reviewers, and authors are not interested in publishing negative results or replications, then perhaps they could be educated that negative results and replications are important and should be valued like other contributions. This will not work either. Or, more accurately, it has not worked yet. These issues have been prominent topics of discussion of methodology for more than three decades with little change in daily practices (Greenwald, 1975; Rosenthal, 1979). There is, for example, little disagreement that the file drawer effect is a bad thing.

Publishing practices are hard to change because innovative research is more important than replication research and negative results. Innovation is the key driver of scientific progress. Publishing has significant resource constraints. Most journals have page limitations for the number of articles they can publish each year, and they receive many more submissions than can be published even if they wished to do so (Nosek & Bar-Anan, 2012). When faced with the choice between accepting an article reporting a new innovation versus an article reporting a replication or negative result, editors and reviewers will usually select the former. Further, editors have the luxury of demanding aesthetically “clean” results rather than tolerating the reality of many research enterprises as untidy affairs (Giner-Sorolla, 2012; Kaiser, 2012). And because demand for the limited space is so high, there is no shortage of articles reporting innovative ideas in neat packages to choose from.

Increasing expectations of reviewers to catch motivated reasoning and other signs of false results

Reviewers and editors are gatekeepers for scientific publishing. If they are not convinced that the manuscript meets the standards for the journal, the paper is rejected. It is conceivable, then, to ask reviewers to scrutinize research more carefully for signs of false results (Schroter et al., 2008). Of the suggestions in this section, we believe that this one is the most reasonable for two reasons. First, reviewers are already very discriminating. It is likely that editors and reviewers are effective to some extent already at recognizing signals of false results. And, second, some evaluation suggestions would be easy to implement, such as the checklist suggested by Simmons and colleagues (2011).

Given the existing system, the peer review process offers the best method presently for identifying potentially false results, other than the diligence of the authors themselves. Nonetheless, we perceive this as a partial solution for three reasons. First, peer reviewers are volunteers. They already work hard for little to no reward. Requiring more than the simple suggestions is asking a lot of people who have already done more than their due. Second, reviewing is hard work. Even diligent reviewers miss lots of errors (Schroter et al., 2008; Wicherts et al., 2011). And, third, peer reviewers review only the summary report of the research, not the research itself. The report is the authors’ perspective on how the research happened and what it means. Most of the actual research process never makes it into the pages of the report; measures, methods, and analysis strategies are summarized. Further, standard practice is to present a strong narrative of what the reader should learn from the research rather than describing how the research and learning occurred (Bem, 2003). The latter would likely be more revealing of potential problems than the former.

Raising the barrier for publication

A related alternative for gatekeepers is to raise the standards for publication by requiring reports to include replications of new findings (Begley & Ellis, 2012; Hewitt, 2012). In one sense, this solution would be effective. If editors did not publish articles without replications, then published articles would have replications. There are some cases, such as the opening anecdote, in which requiring a replication is reasonable—the effect is provocative and data collection is straightforward. However, we believe that requiring replication as a blanket editorial policy could be an impediment for scientific progress. For one, the standards for publication are already extremely high, especially in the social and behavioral sciences. Manuscripts take years to get published, and they are often reviewed at multiple journals by multiple review teams (Nosek & Bar-Anan, 2012). Increasing expectations would further extend this process and put more demand on editors and reviewers—particularly because the pressure to publish will remain so strong for authors.

Moreover, requiring replication of everything could stifle risk taking and innovation. In some domains, requiring replication is not an onerous requirement because data collection is easy. But in other domains, it is difficult or even impossible to conduct a direct replication because of the resource demands or unique opportunities for data collection (Elms, 1975; Lykken, 1968). If replication were essential for every new phenomenon, then researchers might be disinclined to pursue new and challenging ideas to ensure publishability of what they produce. Innovation requires taking risks. That means that innovators can be wrong, perhaps frequently. The problem is not that false results get into the literature. The problem is that they stay in the literature. The best solutions would encourage innovation and risk taking but simultaneously reward confirmation of existing claims.

Strategies That Will Accelerate the Accumulation of Knowledge

In our view, the key for improving the efficiency of knowledge accumulation is to capitalize on existing motivations to be accurate and to reduce the emphasis on publication itself as the mechanism of achievement and advancement. Scientists have strong accuracy motivations. And in the long run, getting it right has a higher payoff than getting it published. However, the goal to publish is immediate, palpable, and concrete; the goal to be accurate is distal and abstract. As a consequence, the short-term payoffs of publishing can be inordinately influential (Liberman & Trope, 1998; Trope & Liberman, 2003), particularly for early-career scientists for whom there is relative urgency for markers of achievement. To address this, the conditions of daily practice must elevate the importance of the more abstract, longer-term goals in comparison to the persisting importance of the concrete, shorter-term goals. In this section, we suggest new or altered practices to meet these objectives.

Promoting and rewarding paradigm-driven research

Whereas conceptual replication is used only to confirm prior results, another relatively common research strategy, paradigm-driven research, can be used for both confirming and disconfirming prior results. Paradigm-driven research accumulates knowledge by systematically altering a procedure to investigate a question or theory, rather than varying many features of the methodology—by design or by accident. This offers an opportunity to incorporate replication and extension into a single experimental design (Roediger, 2012). Paradigm-driven research balances novelty and replication by building new knowledge using existing procedures. Effective use of this approach requires development of standards, sharing and reuse of materials, and deliberate alteration of design rather than wholesale reinvention. For example, the Deese-Roediger-McDermott paradigm for studying false memories (Roediger & McDermott, 1995) has been adapted to examine how aging (Butler, McDaniel, Dornburg, Price, & Roediger, 2004), mood (Storbeck & Clore, 2005), and expectations (Schacter, Israel, & Racine, 1999) influence the frequency of false memories (see Gallo, 2010, for a review). The subsequent findings reinforce the original results through direct replication and extend those findings by identifying moderating influences, mechanisms, and boundary conditions. A paradigm-driven approach provides confidence in the validity of an effect (or doubt if it fails to replicate) while simultaneously extending knowledge in new directions.

It is easy to do more paradigm-driven research if authors make their paradigms available to others. The primary risk of paradigm-driven research is that research questions can evolve to being about the method itself rather than the theory that the method is intended to address. Using a single methodology for a theoretical question can reify idiosyncratic features of that methodology as being the phenomenon. This is where conceptual replication provides substantial added value. Paradigm-driven research provides confidence in the accuracy of findings. Conceptual replication ensures that the findings are theoretically general, not methodologically idiosyncratic.

Author, reviewer, and editor checklists

Earlier we expressed some doubt about raising expectations of reviewers for catching errors, with one exception—easy to implement checklists such as that suggested by Simmons and colleagues (Simmons et al., 2011). Checklists are an effective means of improving the likelihood that particular behaviors are performed and performed accurately (Gawande, 2009). Authors already follow some checklist-like requirements, such as the formatting prescribed by the style manuals of the American Psychological Association (APA) or the Modern Language Association. It is easy to conceive of statistical and disclosure checklists for authors and editorial teams. For example, CONSORT has a 25-item checklist describing minimum standards for reporting randomized controlled trials (http://www.consort-statement.org/). Checklists can ensure disclosure of obvious items that are sometimes forgotten: sample sizes, statistical tests, effect sizes, and covariates included in analysis. They can also define best practices and methodological standards for domain-specific applications.

Why are checklists needed? The most straightforward reason is that key information is left out with stunning frequency, and advisable methodological practices are not identified “naturally” or systematically in the review process. For example, the value of reporting effect sizes has been widely disseminated (Cohen, 1962, 1969, 1992; Wilkinson and Task Force on Statistical Inference, 1999). Nonetheless, reporting effect sizes has become common only in recent history and is still not standard practice. A checklist requiring their inclusion before publication would change this. Further, Bouwmeester and colleagues examined 71 prediction studies from high-impact medical journals and found pervasive methodological shortcomings in design, reporting, and analysis decisions, such as clear specification of predictor and outcome variables, description of participant exclusion criteria, and handling of missing values (Bouwmeester et al., 2012). They concluded that “The majority of prediction studies in high impact journals do not follow current methodological recommendations, limiting their reliability and applicability.” High standards for publication do not translate into specific standards for reporting. Authors’, reviewers’, and editors’ examination of each article is almost entirely ad hoc. Societies, journals, and individuals could maintain simple checklists of standard requirements to prevent errors and improve disclosure.

Challenging mindsets that sustain the dysfunctional incentives

Earlier we stated: “With an intensely competitive job market, the demands for publication might seem to suggest a specific objective for the early-career scientist: publish as many articles as possible in the most prestigious journals that will accept them.” Although this is a common perception, particularly among early-career scientists, we also believe that there are good reasons—though not yet sufficient evidence—to challenge it. For example, the first author regularly presents to graduate students summary data of the short list from a past search for an assistant professor in psychology at the University of Virginia. For this particular search, more than 100 applications were received. Table 1 presents the 11 applicants that made it to the short list. All short-list candidates had at least four publications and at least one first-authored publication. On the basis of publication numbers, there are clear standouts from this group, such as the postdoc with 35 publications, an assistant professor with 21 publications, and a graduate student with 10 publications. Further, these candidates published in prestigious outlets. However, none of these three were selected as a finalist. In fact, two of the three interviewed candidates were among the least productive on the short list.

Table 1.

Short List From an Assistant Professor Job Search at the University of Virginia.

Current status	Publications	First author
Graduate student 1	8	6
Graduate student 2	10	4
Graduate student 3	5	1
Graduate student 4	4	2
Postdoc 1	35	20
Postdoc 2	7	3
Postdoc 3	8	2
Postdoc 4	6	2
Asst prof 1 (4 years post-PhD)	8	3
Asst prof 2 (4 years post-PhD)	21	12
Asst prof 3 (4 years post-PhD)	16	13
Summary
Min	4	1
Max	35	20
Graduate student mean	7	3
Postdoc mean	14	7
Asst prof mean	15	9

Note: The job search occurred in the 2000s. The original pool contained more than 100 applications. Asst prof = assistant professors.

This anecdote suggests that some degree of publishing productivity is essential to get into the pool of competitive candidates, but after that, other factors are more important for getting the job. Without sufficient evidence, we speculate that publication numbers and journal prestige heuristics do play a role in initial selection from a large hiring pool and then play a much smaller role when the pool is narrowed and the hiring committees can look deeply at each candidate. At that point, the committees can invest time to examine quality, potential impact, and direction of the research agenda. In tenure and promotion cases, the depth of processing ought to be even more acute as it is a detailed review of a single candidate’s record.

This conclusion is based on anecdotal data. Early-career scientists would get useful information from a systematic review of the degree to which publication numbers and journal prestige predict hiring and promotion. Multiple departments could pool and share evidence. The aggregate data might confirm the prevailing perception that publication numbers and journal prestige are the key drivers for professional success, or as we believe, they would illustrate notably weaker predictive validity when the evaluation committee has resources to examine each record in detail.

Metrics to identify what is worth replicating

Even if valuation of replication increased, it is not feasible— or advisable—to replicate everything. The resources required would undermine innovation. A solution to this is to develop metrics for identifying replication value (RV)—what effects are more worthwhile to replicate than others? The Open Science Collaboration (2012b) is developing an RV metric based on the citation impact of a finding and the precision of the existing evidence of the effect. It is more important to replicate findings with a high RV because they are becoming highly influential, and yet their truth value is still not precisely determined. Other metrics might be developed as well. Such metrics could provide guidance to researchers for research priorities, to reviewers for gauging the “importance” of the replication attempt, and to editors who could, for example, establish an RV threshold that their journal would consider as sufficiently important to publish in its pages.

Crowd sourcing replication efforts

Individual scientists and laboratories may be interested in conducting replications but not have sufficient resources available for them. It may be easier to conduct replications by crowd sourcing them with multiple contributors. For example, in 2011, the Open Science Collaboration began investigating the reproducibility of psychological science by identifying a target sample of studies from published articles from 2008 in three prominent journals: the Journal of Personality and Social Psychology, the Journal of Experimental Psychology: Learning, Memory, and Cognition, and Psychological Science (Carpenter, 2012; Yong, 2012). Individuals and teams selected a study from the eligible sample and followed a standardized protocol. In the aggregate, the results were intended to facilitate understanding of the reproducibility rate and factors that predict reproducibility. Further, as an open project, many collaborators could join and make small contributions that accumulate into a large-scale investigation. The same concept can be incorporated into replications of singular findings. Some important findings are difficult to replicate because of resource constraints. Feasibility could be enhanced by spreading the data collection effort across multiple laboratories.

Journals with peer review standards focused on the soundness, not importance, of research

The basis of rejection for much research is that it does not meet the criterion of being sufficiently “important” for the journal considering it. Many manuscripts are rejected on this criterion, even if the reviewers identify the research as sound and reported effectively. Despite evidence of the unreliability of the review process for evaluation and identifying importance (Bornmann, Mutz, & Daniel, 2010; Cicchetti, 1991; Gottfredson, 1978; Marsh & Ball, 1989; Marsh, Jayasinghe, & Bond, 2008; Peters & Ceci, 1982; Petty, Fleming, & Fabrigar, 1999; Whitehurst, 1984), this is a reasonable criterion given that journals have limited space and desires to be prestigious outlets. However, in the digital age, page limits are an anachronism (Nosek & Bar-Anan, 2012). Digital journal PLoS ONE (http://plosone.org/) publishes research from any area of scientific inquiry. Peer review at PLoS ONE is explicitly an evaluation of research soundness and not its perceived importance. Since its introduction in 2006, PLoS ONE’s growth has been astronomical. In 2011, 13,798 articles were published (70% acceptance rate), making it the largest journal in the world. Given the disregard for importance in the review process, one might surmise that PLoS ONE’s impact factor would be quite low. In fact, its 2011 impact factor was an impressive 4.41. This put it in the top 25% of general biological science journals and nearly as high as Psychological Science (4.7). This casts further doubt on reviewers’ ability to predict importance (Gottfredson, 1978) or at least one indicator of importance: citation impact. With a publishing model focused on soundness, negative results and replications are more publishable, and the journal identity is not defined as publishing research that is otherwise unpublishable.

Lowering or removing the barrier for publication

A more radical fix than the PLoS ONE model is to discard publishing as a meaningful incentive. How? Make it trivial to publish. The peer review process presently serves as both gatekeeper and evaluator. Postpublication peer review can separate these concepts by letting the author decide when to publish. Then, peer review operates solely as an evaluation mechanism (Armstrong, 1997; Nosek & Bar-Anan, 2012; Smith, 1999). Nosek and Bar-Anan (2012) provide in-depth discussion for how this is achievable by embracing digital journals and public repositories and by restructuring the review process. Successful models already exist, such as arXiv, the public repository for physics and other fields (http://arxiv.org; see also http://ssrn.com/ and http://repec.org/). By submitting their manuscripts to arXiv, authors make their work publicly available to the physics community. Peer review—through the “typical” journals—occurs independently of disseminating manuscripts through the repository. If physicists want to wait for peer review to determine everything they read, they can still do so. But most physicists use arXiv to keep up to date on what other laboratories are doing in their specialty.

By making it trivial to publish, the act itself is no longer much of an incentive. Anyone can publish. The incentives would then shift to evaluation of the research and its impact on future research (i.e., its contribution to cumulating knowledge). Also, the priorities in the peer review process would shift from assessing whether the manuscript should be published to whether the ideas should be taken seriously and how they can be improved (Nosek & Bar-Anan, 2012). Further, this would remove a major barrier to publishing replications and negative results if and when they occur. The only barrier left would be the authors’ decision of whether it is worthwhile to write up a report at all.

Finally, this change would alter the mindset that publication is the end of the research process. In the present system, it is easy to perceive the final step in research occurring when the published article is added to one’s vita. That is the incentive of publication but not of knowledge building. Knowledge building incentives are satisfied when the research has impact on new investigations. By reducing the value of publication, the comparative value of having impact on other research increases (see Nosek & Bar-Anan, 2012, for a detailed discussion and addressing of common concerns about the impact of moving to a postpublication peer review model).

The Ultimate Solution: Opening Data, Materials, and Workflow

Implementing the strategies in the previous section will shift the incentives toward more efficient knowledge accumulation. They do not, however, address the core factor that led Motyl and Nosek to conduct a replication in the opening anecdote—accountability. Science is a distributed, nonhierarchical system. As noted by Nosek and Bar-Anan (2012),

Open communication among scientists makes it possible to accumulate a shared body of knowledge. … Individual scientists or groups make claims and provide evidence for those claims. The claims and evidence are shared publicly so that others can evaluate, challenge, adapt, and reuse the methods or ideas for additional investigation. … Science makes progress through the open, free exchange of ideas and evidence. (p. 217)

Openness provides scientists with confidence in the claims and evidence provided by other scientists. Further, reputation enhancement is a primary mechanism for reward in unstructured contribution systems. Scientists gain and lose status by their public contributions to scientific progress. As such, public reputation management is the primary lever for promoting accountability in academic science.

In present research practice, openness occurs almost entirely through a single mechanism—the journal article. Buckheit and Donoho (1995) suggested that “a scientific publication is not the scholarship itself, it is merely advertising of the scholarship” to emphasize how much of the actual research is opaque to readers. For the objective of knowledge accumulation, the benefits of openness are substantial. Openness increases accountability (Lerner & Tetlock, 1999); makes it easier to share, adapt, extend, and critique methods, materials, analysis scripts, and data; can eliminate the file-drawer effect; and can improve the potential for identifying and correcting errors (Ioannidis & Khoury, 2011; Ioannidis & Panagiotou, 2011; Schooler, 2011; Stodden, 2011).

Three areas of scientific practice—data, methods and tools, and workflow—are largely closed in present scientific practices. Increasing openness in each of them would substantially improve scientific progress.

Open data

With the massive growth in data and increased ease of making it available, calls for open data as a standard practice are occurring across all of the sciences (Freese, 2007; King, 2006, 2007; Schofield et al., 2009; Stodden, 2011; Wicherts, 2011; Wicherts & Bakker, 2012). For example, the Human Genome Project acknowledges its principle of rapid, unrestricted release of prepublication data as a major factor for its enormous success in spurring scientific publication and progress (Lander et al., 2001). Arguments for open data cite the ability to confirm, critique, or extend prior research (Smith, Budzieka, Edwards, Johnson, & Bearse, 1986; Wicherts, Borsboom, Kats, & Molenaar, 2006; Wolins, 1962), the opportunity to reanalyze prior data with new techniques (Bryant & Wortman, 1978; Hedrick, Boruch, & Ross, 1978; Nosek & Bar-Anan, 2012; Poldrack et al., 2011; Stock & Kulhavy, 1989), increased ability to aggregate data across multiple investigations for improved confidence in research findings (Hrynaszkiewicz, 2010; Rothstein, Sutton, & Borenstein, 2006; Yarkoni, Poldrack, Van Essen, & Wager, 2010), the opportunity for novel methodologies and insights through aggregation and big data (Poldrack et al., 2011), and that openness and transparency increase credibility of science and the findings (Vision, 2010).

The concerns about credibility may be well founded. In one study, only 27% of psychologists shared at least some of their data upon request for confirming the original results even though APA ethics policies required data sharing for such circumstances (Wicherts et al., 2006; see also Pienta, Gutmann, & Lyle, 2009). Further, Wicherts et al. (2011) found that reluctance to share published data was associated with weaker evidence against the null hypothesis and more apparent errors in statistical analysis—particularly those that made a difference for statistical significance. This illustrates the conflict between personal interests and scientific progress—the short-term benefit of avoiding identification of one’s errors dominated the long-term cost of those errors remaining in the scientific literature.

The rate of errors in published research is unknown, but a study by Bakker and Wicherts (2011) is breathtaking. They reviewed 281 articles and found that 15% contained statistical conclusions that were incorrect—reporting a significant result (p < .05) that was not, or vice versa. Their investigation could only catch statistical errors that were detectable in the articles themselves. Errors can also occur in data coding, data cleaning, data analysis, and result reporting. None of those can be detected with only the summary report. For example, a study looking at sample mix-ups in genome-wide association studies found evidence that every single original data set examined had at least one sample mix-up error, that the total error rate was 3%, and that the worst performing paper—published in a highly prestigious outlet—had 23% of its samples categorized erroneously (Westra et al., 2011). Further, correcting these errors had a substantial impact on improving the sensitivity of identifying markers in the data sets.

Making data openly available increases the likelihood of finding and correcting errors and ultimately improving reported results. Simultaneously, it improves the potential for aggregation of raw data for research synthesis (Cooper, Hedges, & Valentine, 2009), it presents opportunities for applications with the same data that may not have been pursued by the original authors, and it creates a new opportunity for citation credit and reputation building (Piwowar, 2011; Piwowar, Day, & Fridsma, 2007). Researchers who create useful data sets can be credited for the contribution beyond their own uses of the data.

Movement toward open data is occurring rapidly. Many infrastructure projects are making it easier to share data. There are field-specific options such as OpenfMRI (http://www.openfmri.org/; Poldrack et al., 2011), INDI (http://fcon_1000.projects.nitrc.org/), and OASIS (http://www.oasis-brains.org/) for neuroimaging data. And, there are field-general options, such as the Dataverse Network Project (http://thedata.org/) and Dryad (http://datadryad.org/). Some journals are beginning to require data deposit as a condition of publication (Alsheikh-Ali, Qureshi, Al-Mallah, & Ioannidis, 2011). Likewise, funding agencies and professional societies are encouraging or requiring data availability postpublication (National Institutes of Health, 2003; National Science Foundation, 2011; PLoS ONE, n.d.).

Of course, although some barriers to sharing are difficult to justify—such as concerns that others might identify errors—others are reasonable (Smith et al., 1986; Stodden, 2010; Wicherts & Bakker, 2012). Researchers may not have a strong ethic of data archiving for past research; the data may simply not be available anymore. Many times data that are available are not formatted for easy comprehension and sharing. Preparing data takes additional time (though much less so if the researcher plans to share the data from the outset of the project). Further, there are exceptions for blanket openness, such as inability to ensure confidentiality of participant identities, legal barriers (e.g., copyright), and occasions in which it is reasonable to delay openness—such as when data collection effort is intense and the data set is to be the basis for multiple research projects (American Psychological Association, 2010; National Institutes of Health, 2003; National Science Foundation, 2011). The key point is that these are exceptions. Default practice can shift to openness while guidelines are developed for the justification to keep data closed or delay their release (Stodden, 2010).

Open methods and tools

Open data allow confirmation, extension, critique, and improvement of research already conducted. Open methods have the same effect and also facilitate progress in reuse, adaptation, and extension for new research (Schofield et al., 2009). In particular, open methodology facilitates replication and paradigm-driven research. Published reports of methodologies often lack sufficient detail to conduct a replication (Donoho, Maleki, Rahman, Shahram, & Stodden, 2009; Stodden, 2011). At best, the written report is the authors’ understanding of what is critical for the methodology. However, there are many factors that could be important but go unmentioned—for example, the temperature of the room for data collection, the identities of the experimenters, the time of day for data collection, or whether instructions were delivered verbally or in written form. Moreover, in paradigm-driven research, changes to the methodology are ideally done by design, not by accident. The likelihood of replicating and extending a result is stronger if the original materials are reused and adapted rather than reinvented on the basis of the new researchers’ understanding of the original researchers’ written description.

Authors cannot identify and report every detail that may be important in a method, but many more parts of the methodology can be shared outside of the report itself. For example, it is easy to create a video of the experimental setting and conduct a simulation of the procedure for posting on the Internet. Figshare (http://figshare.com/) offers a repository for data and methods or materials for private archiving or public sharing. Further, the Open Science Framework (http://openscienceframework.org/) is a Web-based project management framework for documenting and archiving research materials, analysis scripts, or data, and it empowers the user to keep the materials private or make them public.

Presently, only the scientific report is cited and valued. Openness with data, methods, and tools makes them citable contributions (Mooney, 2011; Piwowar et al., 2007; http://www.data-pass.org/citations.html). Contributing data or methods that are the basis for multiple investigations provides reputation enhancement for the originator of the resources. Vitas can include citations to the articles, data sets, methods, scripts, and tools that are each independently contributing to knowledge accumulation (Altman & King, 2007). Also, the ready availability of these materials will accelerate productivity by eliminating the need to recreate or reinvent them. Further, reinvention based on another’s description of methods is a risk factor for introducing unintended differences between the original and replicated methodology.

Open workflow

Given that academic science is a largely public institution funded by public money, it is surprising that there is so little transparency and accountability for the research process. Beyond the published reports, science operates as a “trust me” model that would be seen as laughably quaint for ensuring responsibility and accountability in state or corporate governance.

In some areas of science, however, it is understood that transparency in the scientific workflow underlies credibility and accuracy. For example, clinicaltrials.gov is a National Institutes of Health–sponsored study registry for clinical trials. In 2005, the International Committee of Medical Journal Editors started requiring authors to register their randomized controlled trials prior to data collection as a condition for publication. Companies sponsoring trials have an obvious financial conflict of interest for the outcome of the research. A registry makes it more difficult to hide undesired outcomes. Indeed, using registry data, Mathieu, Boutron, Moher, Altman, and Ravaud (2009) found that 31% of adequately registered trials showed discrepancies between the registered and published outcomes. For those in which the nature of the discrepancies could be assessed, 82% of them favored reporting statistically significant results.

Of course, money is not the only source of conflict of interest. Scientists are invested in their research outcomes via their interests, beliefs, ego, and reputation. Some outcomes may be more desirable than others—particularly when personal beliefs or prior claims are at stake. Those desires may translate into design, analysis, and reporting decisions that systematically bias the accuracy of what is reported, even without realizing that it is occurring (Kunda, 1990; Mullen, Bauman, & Skitka, 2003). Public documentation of a laboratory’s research process makes these practices easier to detect and could reduce the likelihood that they will occur at all (Bourne, 2010). Further, registration of studies prior to their completion solves one aspect of the file-drawer effect—knowing what research was done even if it does not get published (Schooler, 2011).

An obvious concern about transparency of workflow is that researchers are not interested in most of the details of what goes on in other laboratories. Indeed, though advocating this strongly, we do not expect that we would routinely look at the details of other laboratory operations. However, there are occasions for which access would be useful. For example, when we are inspired by another researcher’s work and aim to adapt it for our research purposes, we often need more detail than is provided in the summary reports. Access to the materials and workflow will be very useful in those cases. Further, although we do not care to look at the public data about U.S. government expenditures ourselves (http://www.data.gov/), we are pleased with the transparency and the fact that someone can look. Indeed, much as investigative journalism provides accountability for government practice, with open workflow, new contributors to science might emerge who evaluate the knowledge accumulation process rather than produce it and are valued as such.

Finally, using a registry in an open workflow can clarify whether a finding resulted from a confirmatory test of a strong a priori prediction or was a discovery in the course of conducting the research. The current default practice is to tell a good story by reporting findings as if the research had been planned that way (Bem, 2003). However, even if we intend to disclose confirmation versus discovery, our recollection of the project purpose may not be the same as the project purpose when it began. People reconstruct the past through the lens of their present (Schacter, 2001). People are more likely to presume that what they know now was how they conceived it at the beginning (Christensen-Szalanski & Willham, 1991; Fischoff, 1977; Fischoff & Beyth, 1975). Without a registry for accountability, findings may be genuinely and confidently espoused as confirmatory tests of prior predictions when they are written for publication. However, discoveries are more likely to leverage chance than are confirmatory tests. What appears to be “what we learned” could be “what chance told us.” The point of making a registry available is not to have a priori hypotheses for all projects and findings; it is to clarify when there was one and when there was not. When it is a discovery, acknowledge it as a discovery. As Tukey (1977) summarized in support of discovery:

Once upon a time statisticians only explored. Then they learned … to confirm a few things exactly, each under very specific circumstances. As they emphasized exact confirmation, their techniques inevitably became less flexible. The connection of the most used techniques with past insights was weakened. Anything to which a confirmatory procedure was not explicitly attached was decried as “mere descriptive statistics,” no matter how much we had learned from it. (p. vii)

Discovery is critical for science because learning occurs by having assumptions violated. Strong narratives focusing on what was learned are useful communication devices, and simple disclosures of how it was learned are useful accuracy devices.

Conclusion

We titled this article “Scientific Utopia” self-consciously. The suggested revisions to scientific practice are presented idealistically. The realities of implementation and execution are messier than their conceptualization. Science is the best available method for cumulating knowledge about nature. Even so, scientific practices can be improved to enhance the efficiency of knowledge building. The present article outlined changes to address a conflict of interest for practicing scientists—the rewards of getting published that are independent of the accuracy of the findings that are published. Some of these changes are systemic and require cultural, institutional, or collective change. But others can emerge “bottom-up” by scientists altering their own practices.

We, the present authors, would like to believe that our motivation to do good science would overwhelm any decisions that prioritize publishability over accuracy. However, publishing is a central, immediate, and concrete objective for our career success. This makes it likely that we will be influenced by self-serving reasoning biases despite our intentions. The most effective remedy available for immediate implementation is to make our scientific practices transparent. Transparency can improve our practices even if no one actually looks, simply because we know that someone could look.

Existing technologies allow us to translate some of this ideal into practice. We make our unpublished manuscripts available at personal Web pages (e.g., http://briannosek.com/) and public repositories (http://ssrn.com/). We make our study materials and tools available at personal Web pages (e.g., http://people.virginia.edu/~msm6sw/materials.html; http://people.virginia.edu/~js6ew/). We make data available through the Dataverse Network (e.g., http://dvn.iq.harvard.edu/dvn/dv/bnosek), and we are contributing to the design and construction of the Open Science Framework for comprehensive management and disclosure of our scientific workflow (http://openscienceframework.org/). Opening our research process will make us feel accountable to do our best to get it right and, if we do not get it right, to increase the opportunities for others to detect the problems and correct them. Openness is not needed because we are untrustworthy; it is needed because we are human.