Rewarding Research Transparency

Abstract

Cognitive scientists are increasingly enthusiastic about research transparency. However, their enthusiasm could be tempered if the research reward system fails to acknowledge and compensate these efforts. This article suggests ways to reward greater research transparency during academic job searches, academic promotion and tenure evaluations, and society and national award selections.

The active ingredient of any science, including cognitive science, is that its results reproduce. To ensure greater reproducibility, cognitive scientists are increasingly taking steps toward greater research transparency. Such steps, illustrated in Figure 1, include preregistering their studies’ goals and analysis plans; making their studies’ research materials available to everyone; making their studies’ data available to everyone; and making their resulting research reports available to everyone.

Figure 1. Steps toward Greater Research Transparency.

Steps researchers can take toward greater research transparency include preregistering the goals and aims of their study, making their research materials and study data available to everyone, and making their resulting research reports publicly available.

However, taking steps toward research transparency takes time (and occasionally resources), and these steps might not be rewarded. As the US National Institutes of Health (NIH) have noted, ‘the current incentive system may be a major barrier for achieving transparency in research’ [1],

Similarly, the European Commission has warned that ‘it is critical that researchers, who are the key agents of change’ toward greater research transparency be ‘encouraged and incentivised’ [2], Therefore, in this article I suggest ways to better reward research transparency during three phases: when hiring researchers for academic jobs, when evaluating researchers for academic promotion and tenure, and when selecting researchers for society and national awards.

Rewarding Research Transparency during Academic Hiring

If we value research transparency, we should reward it during one of the most incentivized phases of academic life: hiring. Some departments have already taken the lead in this area by explicitly stating in their hiring announcements that they value transparent research practices, and therefore that they will value job candidates who ascribe to transparent research practices.

For example, at the University of Oregon, all its Department of Psychology recent job advertisements contained the following statement: ‘Our Department embraces the values of open and reproducible science’ [3]. Similarly, at Ludwig Maximilian University of Munich, their Psychology Department job advertisements state: ‘We support transparent research with open data, open material, and preregistrations’ [4].

Another way to reward research transparency during academic hiring is for those of us who serve on search committees to evaluate job candidates in light of our commitment to research transparency. Indeed, the commitment ‘As members of committees (e.g., tenure track, appointment committees, teaching, professional societies) or editorial boards, we will promote the values of open science’ is an actual pledge one can make on ResearchTransparency.org.

Turning to job candidates, they can illustrate their commitment to research transparency by describing their commitment in their cover letters; by creating a Research Transparency section in their Research Statements; and by annotating their vita to indicate which of their studies are based on preregistration, open materials, open data, and open-access research reports [5]. In fact, the Stanford Center for Reproducible Neuroscience recently recommended to the NIH that it should require annotations of research transparency practices such as data sharing, as a standard component of all NIH biosketches [6].

Job candidates can also illustrate their commitment to research transparency by asking their letter writers to address the candidates’ research transparency activities in their letters of recommendation. If departments and candidates both articulate their commitment, and if departments then evaluate candidates according to this commitment, academic hiring will incentivize research transparency.

Rewarding Research Transparency during Academic Promotion and Tenure Evaluation

If we value research transparency, we should also reward it during evaluation for academic promotion and tenure. As the G7 Science Ministers of Canada, France, Germany, Italy, Japan, the UK, and the US advise: ‘evaluation of research careers should better recognize and reward Open Science activities’ [7].

Some academics are already reporting that they are being tasked with assessing research transparency while evaluating academics for promotion and tenure [8,9]. Some candidates for promotion and tenure are also reporting that their research transparency activities are figuring prominently in positive promotion and tenure decisions [10].

For example, at my home institution, the University of Wisconsin–Madison, our Policies and Procedures for Post-Tenure Review of Faculty include, in the section for evaluating Research and Scholarly Productivity, the typical fare of peer-reviewed publications. However, my university also lists ‘open-source databases, online tools … and other networked, digital resources related to scholarship’ as evidence of scholarly productivity. And rightfully so: open-access resources, including research materials, data, and codes that facilitate research transparency, are valuable products that should be acknowledged during faculty review.

Just months before his death, the late Steven Hawking made his dissertation open access because he believed that ‘anyone, anywhere in the world should have free, unhindered access to not just my research’, but to everyone’s research [11].

Of course, not all our open-access materials will be as influential as Stephen Hawking’s dissertation. However, such materials deserve not only attribution, including proper citation for datasets, as recommended by the Joint Declaration of Data Citation Principles and the Austin Principles of Data Citation [12], but also acknowledgement during evaluation.

Thus, I agree with Christopher Long, Dean of the College of Arts and Letters at Michigan State University, who argues that universities need to develop ‘an array of values-based metrics capable of empowering our faculty to tell more textured stories about the impact of their work’ [13]. Rewarding the steps taken toward greater research transparency can, and should be, part of our values-based metric.

Rewarding Research Transparency with Honors and Awards

If we value research transparency, we should honor and award it. A handful of honors and awards specifically do just that.

For example, the Open Science Prize is awarded annually by the Wellcome Trust, the NIH, and the Howard Hughes Medical Institute to ‘unleash the power of open content and data’ (https://www.openscienceprize.org). The Leamer-Rosenthal prizes, named for economist Edward Leamer and psychologist Robert Rosenthal because they ‘helped lay the foundations of research transparency’, are awarded every year by the Berkeley Initiative for Transparency in the Social Sciences (https://www.bitss.org/lr-prizes).

We should also create new awards to specifically honor research that embodies transparency. For example, the Cognitive Science Society could add to its important array of awards a prize for researchers who preach and practice preregistration, sharing materials and data, and providing open-access reports. Similarly, the American Psychological Association, Association for Psychological Science, American Educational Research Association, Linguistic Society of America, Cognitive Neuroscience Society, and other organizations could create awards that specifically honor transparency at different levels of researcher experience and in different disciplines and subdisciplines.

In addition to these specific awards, we can reward research transparency in the following way. Whenever we are asked to nominate researchers for mainstream honors and awards, we can remind ourselves to think of candidates who not only have achieved the classic research accomplishments but who also, as noted in a recent editorial in Science, value and promote quality control activities in research, train young scientists to avoid pitfalls, make their data publicly accessible, and influence ‘research integrity and [the] quality and reproducibility of results’. Because ‘to not do so risks the reputation of the scientific enterprise’ ([14], p. 531).

Concluding Remarks

When a thousand psychologists were recently asked why they do not take more steps toward research transparency (such as preregistering their studies), the most common answer (beyond their research being solely exploratory) was the lack of incentives and rewards [15]. We can and should better reward research transparency. We can do so in ways listed in Box 1.

Box 1. Ways to Reward Research Transparency.

Ways to Reward Research Transparency during the Hiring Process

• (i)Departments explicitly state their commitment to research transparency in their job advertisements and announcements.
• (ii)Search committees evaluate job candidates according to their commitment to research transparency.
• (iii)Job candidates illustrate their commitment to research transparency in their cover letters, and ask letter writers to address their commitment in their letters of recommendation.
• (iv)Job candidates annotate their vitae to indicate which of their articles are based on preregistration, open materials, open data, and open-access research reports.

Ways to Reward Research Transparency during Promotion and Tenure Evaluations

• (i)Departments explicitly consider candidates’ research transparency practices.
• (ii)Outside evaluators instructed to explicitly comment on candidates’ research transparency practices.

Ways to Reward Research Transparency through Society and Organization Honors and Awards

• (i)Societies and organizations create awards and honors to specifically recognize research transparency.
• (ii)When asked to nominate candidates for mainstream honors and awards, nominators consider researchers who not only have achieved classical metrics of research accomplishment but also have done so by employing transparent research practices.

Of course, the proof will be in the pudding. Departments could explicitly state their commitment to research transparency in their job advertisements and promotion criteria, and candidates could aptly illustrate their commitment in their job applications and promotion dossiers. However, if adherence to the value of research transparency is not manifested in attitudes and actions, as well as verbiage, the incentives will evaporate.

Changing scientific culture requires top-down leadership in concert with bottom-up enthusiasm, institutional commitment in support of departmental agreement, publication and funding gatekeepers in sync with publication and funding gate knockers, and actions as well as words. The increasing efforts that cognitive scientists are making toward greater research transparency are exciting, but they need to be rewarded.