The Role of Publons in the Context of Open Peer Review

Abstract

Publons was a peer reviewer rewards platform that aimed to recognize the contribution that academics made during peer review to a journal. For about 10 years of its existence, Publons became the most popular service among peer reviewers. Having gained traction and popularity, Publons was purchased in 2017 by Clarivate Analytics (now Clarivate), and many academics, journals and publishers invested time and effort to participate in Publons. Using Publons, various peer review-related experiments or pilot programs were initiated by some academic publishers regarding the introduction of open peer review into their journals’ editorial processes. In this paper, we examine pertinent literature related to Publons, and reflect on its benefits and flaws during its short-lived history. In mid-August 2022, Clarivate fused Publons into the Web of Science platform. Publons, as a brand peer review service, has now ceased to exist but some of the functionality remains in Web of Science while other aspects that used to be open and free at Publons are now paid-for services. We reflect on the effect of such experiments, which initially had bold and ambitious academic objectives to fortify peer review, on academics’ trust, especially when such projects become commercialized.

Publons has been Fused into Web of Science

Publons1 was a peer review recognition website and service that was owned by Clarivate (previously Clarivate Analytics), which also owns the Web of Science (WoS) and the journal impact factor (JIF), a popular journal-based metric. To the surprise of academics, and quite suddenly, on August 18, 2022 Publons was fused into WoS.2 Publons had operated for 10 years and was, according to some, the most popular peer review platform [1], serving as a platform to recognize and reward mostly voluntary peer review services [2]. All peer review summaries and open peer reviews from Publons have now been amalgamated into WoS article records,3 and are available without a WoS subscription. However, some functionality can only be access with a paid subscription, i.e., functionality that was once open and free at Publons has now been commercialized by Clarivate.

Publons also used to have Publons Academy,4 a highly advertised and praised service [3–8] where academics were trained with regards to several aspects of peer review, but that service ended sometime in 2021, and has been rebranded by Clarivate as the WoS Academy.5 Martínez-López et al. [9] characterized Publons as a “free online social media service that record, verify, and highlight contributions of scholars that act as reviewers without compromising anonymity”, while López-Hermoso et al. [10] also classified it as a “social research network”, even equating Publons to ORCID and ResearchGate, a characterization that we do not entirely agree with because ORCID is a registry of scientists, ResearchGate is a social network for scientists, and Publons was a service for showcasing peer review and editorial contributions, while providing a bibliometric background of authors’ most popular indicators.

In the light of this historical event in the field of academic peer review, as a fundamental basis of the global scholarly publishing industry, we decided to examine—in more detail—the literature that pertains to Publons, focusing on the past few years, when Publons gained popularity and use, and became the premier peer review rewards service of academic publishing. To identify relevant literature, a search was conducted (August 26, 2022) in the WoS Core Collection for “Publons” in the title or keywords. The search was limited to 2015–2022. From 58 records that were initially found, after excluding false positives, irrelevant entries, and non-English papers, the remaining 44 papers were carefully screened for inclusion in this paper. Finally, 40 papers were cited for thematic relevance.

Publons’ Role in Peer Review Rewards: A Critical Perspective

Traditionally, peer reviewers (and editors) work voluntarily (i.e., freely) for journals and publishers to conduct pre-publication peer review, as “useful slaves”, as Fernández-Cano [11] describes them, or as part of science’s “gift economy” [12]. The rationale for free labor (peer reviewing, which is actually equivalent to a professional consultation service) is that a reviewer serves the community (i.e., has communal obligations) and provides support to academia (i.e., reciprocity) [13]. Other motivations to conduct pre-publication peer review include career advancement, recognition (by peers) as an expert, and peer, editor and journal networking [14]. However, there may be an element of exploitation, especially by for-profit publishers, some of which reap massive economic returns from peer-reviewed papers [15]. In part, this is because “labor” costs associated with peer reviewing are zero, essentially giving nothing (at least, financially) in return [16]. In contrast, the annual savings for publishers may be in the billions of US$. For example, using Publons-derived data, Aczel et al. [17] estimated that 100 million hours were invested in peer review globally, translating to US$1.5 billion in 2020 alone, a value that they believed is an underestimate. However, the burden of peer review responsibility is highly skewed in terms of the number of contributors to the process [18, 19].

In that sense, Publons emerged as a platform to “reward” academics for their time and efforts, and as a form of public recognition.6 However, such recognition can be perceived by some academics as a pennies-on-the-dollar scheme because even if demand for reviewers is high, the pool of available academics to complete the task is large, although this sink-and-demand dynamic might change if more academic journals enter the market, placing additional pressure on the same peer pool [20]. Pomponi et al. [21] found that Publons’ marketing approach was promotional, appealing to academics’ vanity and sense of self-worth rather than on peer review-related novelty, also noting that most reviewers emerged from the US, India, then China. In 2015, using Publons-derived data, and depending on the authors involved, the supply of reviewers exceeded demand by 15–249%, while about 20% of reviewers completed 69–94% of all reviews, i.e., peer review tasks were highly skewed [19]. What may result with this imbalance is reviewer fatigue as the number of invitations increases, placing greater pressure on academics who are repeatedly invited as reviewers, while greater pressure is also placed on editors to identify suitable reviewers, creating an unhealthy (exhausting and unsustainable) sink-and-demand dynamic [22]. Reviewer fatigue may then lead to reviewer refusal, especially if they are overwhelmed with invitations, lacking motivation, or finding that papers for which they are invited to review lack originality [23]. The available pool of peer reviewers may be further pressurized by invitations to peer review in journals that are not indexed, or possibly even predatory journals [24, 25], thinning the available labor and their time, energy and expertise available for indexed and supposedly more scholarly journals [26]. Peer review may become, as a result, biased because editors might then consistently recruit the same “reliable” (i.e., productive) peer reviewers [27] or those that are known to the editors [28]. However, placing the burden on a productive sector of the academic community may result in greater refusals to peer review [29, 30]. To compound this issue, a positive agreement to review a paper might not necessarily translate into a “gain” for the journal, i.e., a positive outcome of peer review [31].

Using Publons-derived data, Ahmed and Yessirkepov [32] attempted to assess the peer review productivity of researchers in five countries in Central Asia (Kazakhstan, Kyrgyzstan, Tajikistan, Turkmenistan, Uzbekistan), noting that only 11.7% of 15,764 researchers had verified peer-reviewing records, eight being Publons awards winners, with > 90% by researchers in Kazakhstan and Uzbekistan. Among nine Central Asian countries, in 1 year, China contributed the highest number of peer reviews (12,071 reviews from Zhejiang University alone), followed by Iran [33]. Nojavan et al. [34] found that most reviewers from Iranian research institutes rewarded at Publons were from the University of Tehran. Sadly, these three studies only discussed the quantitative output of these researchers, but no attention was paid to the content and quality of the peer reports that they produced. For example, it is not uncommon to find academic papers (as two examples: [35, 36] written by editors of journals who laud their own academic achievements and the achievements of their journals, including aspects related to peer review. Yet, close examination of these self-lauding papers often reveals one serious gap, namely the absence of evidence—in the form of open peer review reports—that would substantiate the positive aspects of peer review. There is also no guarantee that these peer reports are available at WoS in order to verify their content, an analysis that is urgently needed in the future. Blind faith in editors’ self-appreciation of their own journal’s successes or achievements, absent quantifiable evidence, as well as peer reviewers’ boastful output of peer review reports, as was previously show-cased at Publons and now at WoS, absent the existence of tangible evidence in the form of open peer reports, should induce academia to reflect on whether it is still viable, and deontologically correct, to continue to operate the academic publishing industry on the premise of “trust me” [37]. Moreover, since many reviewers are academics themselves, they are also under pressure to complete their own scholarly tasks and publish academic papers, leaving them little free time to serve as voluntary reviewers. To the authors’ knowledge, there is limited quantitative evidence to prove that Publons and Clarivate, in the Publons-based venture, have actually or tangibly improved peer review performance across the global academic publishing industry, other than increasing adherence to the platform [38]. For example, already by 2019, Publons had already attracted the adherence of two million academics who had created profiles [21].

Participation at Publons was also highly skewed, in terms of disciplines, and with some journals or publishers being more comprehensively covered by Publons than others, most likely as a result of agreements between those journals and/or publishers and Clarivate [39], with some reviewers being highly—perhaps excessively—productive [21, 40], and the majority being male [41]. Rice et al. [42] confirmed this latter finding by examining the peer reports of Publons-registered “mega peer reviewers”, i.e., reviewers that had reviewed at least 100 papers in a 1-year period (in 2018), finding that 74% of them were male, compared to 58% of “standard” peer reviewers, i.e., those who had reviewed 18 or fewer papers per year. It is also unclear how many of the claimed peer reviews at Publons might also involve “ghost” peer reviewers, such as early career researchers who secretly (or opaquely) conduct reviews on behalf of their mentors or principal investigators, with the latter receiving the formal credit at Publons [43]. However, the volume of peer review is not necessarily associated with its quality, and inducing a quantity-based culture of peer review, as occurred at Publons, and now also at WoS, with awards offered to those who review the most, as an “elite” subset of all peer reviewers who are “rewarded” at Publons, may inculcate a culture of unhealthy competition as there was a desire to achieve a Publons reviewer award, amplifying the “publish or perish” culture by adding a new layer of pressure, namely “peer review or perish” [44]. The fact that Publons users used to be able to activate or deactivate their availability as reviewers at Publons, and the existence of inappropriate requests [45], suggests that there were more flaws with the Publons platform than was made to believe. Was this a possible reason why Publons suddenly ceased to exist and was subsequently rebranded and fused into the WoS platform? Smith [46] referred to highly productive peer reviewers, i.e., those that pump out thousands of peer reviews, without having assessed the actual content and quality of those reviews, as “highly-ranked”, suggesting by what was not stated, that those with few peer reviews might be poorly ranked and thus, by association, inferior. Smith [46] further encouraged participation in this unhealthy competition: “Sign up now to see how much better or worse you are at peer review than the colleague next to you” (p. 265). Such unhealthy competition may ultimately engender peer reviewer burnout [47].

Unlike such subjectivity, in the context of funding applications, Gallo et al. [48] proposed a “scientific merit score” from 1 to 5 that offered a more objective appreciation of the content, and thus quality, of a reviewer’s report. Similarly, Mavrogenis et al. [49] suggested a points-based rewards system in which a reviewers’ comments and observations are assigned points based on characteristics divided among three groups of characteristics (instructional and informative, organized and objective, and responsible and reliable). Chakraborty et al. [50] proposed a sentiment-based evaluation of peer report content, and when each of several parameters was ranked as positive, neutral or negative, it would be possible to more objectively arrive at an accept or reject decision. Also, within the context of reviewers involved in the evaluation of projects for funding, reviewer performance plateaued when reviewers assessed two proposals, and not more [51]. According to Ortega [39], Publons used to have a “scoring system”7 for reviews, involving a mixture of metrics, namely (1) quality (ranked from 1 to 10); (2) significance (ranked from 1 to 10); (3) an overall Publons score, or aggregate scores with a maximum value of 10; (4) number of reviews; (5) scores (reflecting individual opinions about a paper); (6) WoS-based citations. Publons had indicated that the Publons Score was “completely anonymous”, with only one score allowed per paper per Publons account. It is now extremely difficult to analyze how individual papers, which were claimed to have been peer reviewed, were scored, since all Publons pages have now been deleted.

If the quality of peer reports is not clearly indicated at Publons, and if peer reviewers are unaware of the community-perceived quality of their reports, either through editorial or peer feedback, then this might also impact their motivation to peer review, since quality assurance and enhancement-related parameters that define a good reviewer and/or peer report might not be clear [52]. There is also no evidence, from our assessment, that actual or perceived conflicts of interest by reviewers were indicated at Publons on reviewers’ reports, and since it is realistically inconceivable that 100% of reviewers would have zero conflicts of interest, financial or non-financial [53], this suggests that some peer reviewers, their reports, and/or Publons may have been hiding conflicts of interest.

Using Google Scholar-based citations, Ortega [54] found that peer review quality was weakly correlated with reviewers’ bibliometric indicators and, quite surprisingly, that “the ratio of manuscript acceptance is inverse to the number of reviews” (p. 960). Ortega [39] further noted a very weak correlation between Publons metrics (e.g., Publons Score) and bibliometric and altmetric indicators, namely PlumX, Altmetric.com and Crossref Event Data. Examining 45,819 papers that were indexed in Publons in 2018, Ortega [39] noted that 83.5% of them had pre-publication peer reviews, only 4.9% had post-publication peer reviews, while 54.5% had Publons Scores. Ortega [39] also found that, in terms of pre-publication peer reviews, that 37.7% were by one reviewer, 49.4% were by two reviewers, 10.1% were by three reviewers, while 2.9% were by more than three reviewers. Garcia et al. [55] noted that, as the rejection rate of a journal increased, in order to maintain a high “quality” journal standard, greater pressure is placed on peer reviewers to equally produce high “quality” peer reviews, ultimately placing pressure on peer reviewers to perform at high standards, even though they are not financially compensated for providing such high-level professional services. Ultimately, the “cost” of peer review increases as journal motivation to strive for higher journal “quality” increases. Although Garcia et al. [55] did not discuss this particular outcome, it is possible that reviewers of papers that are ultimately rejected might feel that their work is under-appreciated, especially if they recommended acceptance and publication, lowering their motivation to either peer review again for the same or another journal. The motivation not to review might also be associated with a mismatch in expertise and the paper’s topic, the fame, level or rank of the journal, or lack of identification with the scientific community [56]. Under-appreciation of peer reviewers’ voluntary work is certainly one logical reason why reviewers might be motivated not to peer review, but there are also costs, namely “having to deal with low-quality papers, taking responsibility for errors, and editors and authors disregarding the review suggestions” (p. 220) [52].

Not many of these papers focused on the skill-sets of peer reviewers as being a key factor in the success of the peer review process, and how such skills might have factored into Publons-based recognition, select aspects of which were more recently assessed by Huang and Zong [57], Lei [58] and Zhang et al. [59]. For example, many librarians and information professionals have a very high set of organizational and analytical skills that would make them ideal peer reviewers of systematic reviews, with a survey of 291 such academics indicating that the vast majority (95%) recommending rejection or revisions of papers based on methodological flaws [60].

This suggests that Publons was also a marketing gimmick related to inconsistent peer review rewards, as is also debated in this paper, than a de facto ethics- or peer review integrity-reinforcing tool. The greater risk is that Publons was amplifying the further commodification of peer review, but without just rewards, thereby not necessarily serving the greater good of academia, but rather the greater good of for-profit publishers [61]. Furthermore, Clarivate used to offer academic publishers additional services based on Publons, such as Publons Transparent Peer Review, the Publons Reviewer Recognition Service, or Reviewer Locator.8 It is not immediately clear from those websites which of those services were free, and which were paid for. This issue is now irrelevant given that Publons has now been completely phased out.9 A lingering question remains, however, namely if Publons was merely another vanity-based tool that gave superficial rewards while serving as a front for the deeper exploitation of academics and their peer-reviewing skills?

This paper does not pretend to try and answer this question because an answer would require a side-by-side analysis of peer reviewer reports at Publons and the corresponding published papers so that the academic validity of the product (published paper) and veracity of peer review, i.e., proof that peer review has actually been conducted and is not fake [62], can be independently and publicly verified. It is now also highly likely that this question might never be answered, now that Clarivate has scrubbed clean the entire Publons website, and thus evidence that could assist academics in answering important unanswered questions.

The Intersection Between Publons and Open Peer Review

Another problem with Publons, in the authors’ view, is that not all peer reports were open to the public, i.e., the rewards scheme promoted by Publons (and thus Clarivate) continues to be a predominantly closed peer review model and not an open peer review (OPR) model, sensu stricto [63, 64]. In theory, if a review received a DOI from Crossref, then it would be possible to assess the total number from the Crossref database. However, Crossref does not specify the type of peer review, and since OPR is scattered across various platforms, it is currently difficult to envision how to gather all this information. The greater risk of rewarding something blindly (i.e., a product that cannot be seen) is that bad peer reviews (i.e., poor quality, unprofessional behavior, etc.), superficial peer reviews, predatory and unethical peer reviews (including self-citation requests) may be hidden, and if such cases exist, then rewards on Publons (and now at WoS) might be unfair and thus unscholarly [25, 65]. Other risks of not openly displaying peer reports include hidden biases, conflicts of interest, rude language, failure of authors to respond, disregarded peer reviewers’ opinions by editors, and ultimately the commercialization of knowledge based on improperly conducted peer review [66]. As demand for reviewers grows, absent proper educational mechanisms to train them, including early career researchers, about how to avoid such issues, a negative feedback loop ensues, and a peer review crisis develops [67]. One possible reason why the OPR model has not been more widely embraced by the academic community, including authors, peer reviewers, editors, journals and publishers, is an element of fear, fear of public criticism, challenges, insults, attacks and undesired scrutiny [68]. It is not uncommon to observe journals in which the OPR model is offered as a choice, i.e., authors can voluntarily opt in or opt out of this scheme (see select cases in Table 1). By December 2019, Wolfram et al. [64] had found that 617 journals from 38 publishers had adopted some form of OPR.

Table 1.

OPR policies in select OA indexed journals

Title	TPR/OPR	Reviewer’s open identities	Open reports	Author’s responses	Editor’s decision letter
Business Ethics, the Environment and Responsibility	Necessary	Optional	Yes	Yes	Yes
eLife	Necessary	Optional	No	Yes	Yes
Environmental Research Letters	Optional	Optional	Yes	Yes	Yes
F1000Research	Necessary	Necessary	No	No	No
Nature Communications	Optional	Optional	Yes	Yes	Yes
PeerJ	Optional	Optional	Optional	Optional	Optional
PLOS ONE	Optional	Optional	Yes	Yes	Yes
Publications	Optional	Optional	Yes	Yes	Yes
Royal Society Open Science	Necessary	Optional	Yes	Yes	Yes

Despite open identities being the most prevalent trait of OPR [69], reviewers are sensitive to questions about the possibility of protecting their anonymity [70]. The ability of reviewers to remain anonymous ensures that they can provide critical reviews without fear of reprisals from their academic colleagues [71]. A theoretical simulation model of scientists’ behavior during OPR testified the importance of anonymity and confidentiality of the process, but was found to “possibly undermine the quality and efficiency of the process”, i.e., of peer review [72]. However, the results of research on real OPR in MDPI journals [73] and Taylor & Francis’ publishing platform F1000Research [74] showed that despite certain disciplinary features, open identities did not directly affect the reviewers’ decisions nor did they pose a threat to the reliability of the entire review process.

Of note, Ross-Hellauer [69], who conducted an assessment of definitions of OPR in the literature, found as many as 22 separate definitions, although ultimately, “optimal” OPR involved the following seven characteristics: (1) open identities in which “authors and reviewers are aware of each other’s identity”, (2) open reports, which “are published alongside the relevant article”; (3) open participation, where the “wider community are able to contribute to the review process”; (4) open interaction, which involves “direct reciprocal discussion between author(s) and reviewers, and/or between reviewers, is allowed and encouraged”; (5) open pre-review manuscripts, which “are made immediately available (e.g., via pre-print servers like arXiv) in advance of any formal peer review procedures”; (6) open final-version commenting, allowing the “review or commenting on final “version of record” publications”, a practice that is comely referred to as post-publication peer review (PPPR); (7) open platforms (“decoupled review”) that allow review to be “facilitated by a different organizational entity than the venue of publication” (quotes from p. 7).

However, authors and peer reviewers are two critical elements of a peer review, so for OPR to work, there has to be unity and consensus between authors and peer reviewers to make their peer reports, and responses, open and public. If one of these parties fails to provide approval, or does not agree to this arrangement, then OPR fails. We describe next some of the pilot projects that had been implemented by select publishers, and related to OPR that were based on Publons.

The “pitfalls” of Transparent Peer Review Pilots

In journals that request authors to anonymize manuscripts, peer review may be double-blind. In such a case, the identities of peer reviewers is unknown to authors, and vice versa, a process that is usually confidential. However, if a journal’s Publons policy allowed (or mandated) peer reports and/or peer reviewers’ identities to be revealed at Publons (and now at WoS), even after papers are reviewed or published, does this not defeat the purpose of anonymity, blinding or confidentiality during peer review? The same challenge is being faced with preprints, with open author identities, and the clash with mandatory requirements of anonymized papers by a body of peer-reviewed journals [75].

It should be noted that in addition to, or as a replacement of, the review model adopted in a journal, Publons invited editors to experiment with Publons Transparent Peer Review, which is now linked with Clarivate’s online submission system ScholarOne,10 claimed to offer a comprehensive technical solution for introducing open procedures into the review process on a voluntary basis:

• Authors could opt-out of publicly displaying the article’s peer review content;

• Reviewers could opt-out of publicly displaying the article’s peer review content;

• Reviewers could choose to sign their reviewer reports with their name or as anonymous.11

A permanent DOI was assigned to any peer reviewer report. We should also emphasize, as shown in Table 2 for a sample of several open access journals, that the degree of openness of the review process depended primarily on the journal’s editors, and not on the journal’s participation in the Publons Transparent Peer Review program.

Table 2.

The policy of presentation select OA journals on Publons

Title	Publons parters	Allows reviews to be publicly displayed on Publons	Allows reviewers to display the title of the article they reviewed on Publons	Reviewer recognition	Publons transparent peer review
Business Ethics, the Environment and Responsibility	No	No	No	No	No
eLife	Yes	Yes	Yes	Yes	No
Environmental Research Letters	Yes	No	No	Yes	Yes
F1000Research	No	Yes	Yes	No	No
Nature Communications	No	No	No	No	No
PeerJ	Yes	Yes	Yes	Yes	No
PLOS ONE	Yes	Required an author’s permission	Yes	Yes	No
Publications	Yes	Yes	Yes	Yes	No
Royal Society Open Science	Yes	Yes	Yes*	Yes	Yes

Information was valid and openly/publicly verifiable at Pubones until August 17, 2022, prior to the sudden closure of all Publons-related websites

*Does not allow reviewers to sign or publicly display reviews of rejected publications subsequently published in another journal

For example, in a Wiley-published journal, Business Ethics, the Environment and Responsibility (BEER), a Committee on Publication Ethics (COPE) member publisher and journal, there is a clause12 that still indicates a pilot or experimental program that involves Publons: “This journal is participating in a pilot on Peer Review Transparency and you have the choice to opt-out during the submission process. By submitting to this journal, you agree that the reviewer reports, their responses, and the editor’s decision letter will be linked from the published article to where they appear on Publons in the case that the article is accepted. Reviewers can choose to remain anonymous unless they would like to sign their report.” It is not clear what happens to the peer review and the integrity of the Publons entry for the paper if the author(s) opt-in for this process, and its associated clauses, but if the peer reviewer wishes to opt-out. BEER publicly rewarded only the top peer reviewer and three runners-up in 2021, but there is no indication of the precise criteria that the BEER Associate Editors used to decide on the awards.13 It is also unclear if these awards winners received any monetary compensation for their hard work, considering that they were likely responsible for bringing in considerable funding for the publisher, as OA APCs and/or subscription fees for the papers that they reviewed. Finally, BEER claims that the peer reports were “consistently high-quality”, but no links to any Publons webpages exist that would allow members of academia to independently judge the content of those reports and thereby independently assess their quality. On that same BEER peer reviewer rewards page, eight academics received the award in 2019, while no reviewers were listed for 2020. Since Wiley did not cease to publish papers in 2020, despite COVID-19, it is unclear why no peer reviewers received an award in that year. Wiley and BEER provide no explanation for that gap. What those awards suggest, however, is that peer reviewers are rewarded (including at Publons) for primarily quantity [61]. Kendall [40] noted that the top three reviewers at Publons, with 1.7 million researchers and 10.8 million registered reviews, had reviewed 7.69, 5.08 and 4.71 papers a day over several years, which seems to be unrealistic, and humanly impossible. Those reports should be made open to public scrutiny to appreciate their quality. The recent mass resignation of editors from a Wiley journal in protest of unsustainable editorial practices14 alerts academia that there continues to be a schism between editorial exploitation by for-profit publishers, and under the surface, the continued unremunerated exploitation of peer reviewers, despite now outdated superficial Publons-based rewards schemes.

The Wiley experiment is not new and individual Wiley journals had been experimenting and working with Publons since 2016 [76–79]. Using editorials, SAGE also enthusiastically announced its partnership with Publons [80, 81], as did two Wolters Kluwer journals [82, 83], and an IOP Publishing journal [84], although they15 provided no quantitative analysis indicating the number of participating authors and/or peer reviewers, or whether participation was voluntary or mandatory. A similar Publons-linked pilot that took into account the opt-in or opt-out options of authors and peer reviewers was attempted by three IOP Publishing journals, JPhys Materials, Journal of Neural Engineering, and Environmental Research Letters over the period of 1 year, although only 6 months’ data was presented [85]. In that pilot, peer reviewers had the option of openly publishing the content of their review reports, while staying anonymous, a concept that they coined transparent peer review (TPR), i.e., a Clarivate service.16 TPR is thus a half-open measure. Papers in which authors and/or peer reviewers participated, a Publons Badge was assigned to each article, for which there were three categories: (1) TPR with reviewer reports hosted at Publons (but unnamed peers); (2) one or more reviewers claimed to have reviewed the paper, but there are no peer reports; (3) no peer review content for that paper, but there is for other papers in that journal over the space of 1 year. Papers where neither authors nor reviewers participated did not have a badge (category). We are of the opinion that except for category 1, which is minimally transparent because peer identities are not revealed, so this TPR model can never be referred to as fully “open”, like OPR,17 the remaining categories are standard opaque categories of participation. Domingo and Harris [85] revealed that in 6 months of this pilot, authors’ agreement to participate was 54–59% while that of reviewers was 41–49% for the three journals. Unfortunately, no data was provided for the overlap of both participants, i.e., the number of authors that agreed to participate and the number of reviewers that agreed to participate. Moreover, no data was provided regarding how many papers’ peer reviews were OPR, i.e., with peers’ identities disclosed. Despite these gaps in data and relative weak levels of adhesion and participation, Domingo and Harris [85] overhyped the success of the program, using terms like “highly efficient”, “elegant”, “a successful pilot” and “relatively high” uptake, claims that the data of that paper did not support, i.e., i.e., spin [86]. The hype can be put into context by noting that the first and second authors are Clarivate and IOP Publishing employees, respectively. The authors concluded that “reviewers who opted in for TPR, most (over 80%) chose to remain anonymous”, suggesting that this pilot did not manage to achieve transparency or openness, unlike what was claimed. Even though the primary focus was on the opt-in of authors and reviewers, no insight was provided why such high percentages of authors and peer reviewers did not opt for TPR, even less for OPR. Despite this, the publisher planned “to roll out the TPR option on all of its fully open access journals in the near future” (p. 76).

Prior to that, in a 1-year pilot at Nature Communications involving 787 papers, 60% of authors opted-in for having their peer reports open [87]. In a 2020 update of a PLOS TPR pilot program involving more than 3500 papers, only 39% of authors opted-in while 60% of papers had signed peer review reports, i.e., OPR (PLOS, 2020). However, those details were presented as a blog post and there was no serious discussion of the failure of the TPR and OPR models, nor was there any discussion about an opt-out option for authors or peer reviewers. Neither the Nature Communications nor PLOS pilot programs referred to Publons, and it is unclear if any of those reports that are OPR are in fact currently linked to WoS (previously at Publons).

Regarding PLOS ONE, we note a very curious peculiarity. Authors were allowed to post reviews on their website, but not on Publons,18 while on Publons, a warning used to exist: “The reviewer has opted to show the content of this review, but Plos One's privacy policy prevents it from being shown here”.19 In the latter case, we also note that the review carried a CC BY 4.0 license, which should allow for unrestricted use and reproduction in any medium or platform provided that the source is properly cited. It is unclear what licenses peer reviewer reports at WoS now carry, but an in-depth analysis is merited. Malcom [88] had briefly touched on the issue of privacy and ownership of peer reports, and if clauses or confidentiality-based limitations might have clashed with Publons’ objectives.

None of these pilot programs indicated how they would manage to increase opt-in rates to 100%, how they would convince authors and peer reviewers to opt-in, or if they would revert to mandatory options in order to achieve full adherence to TPR and/or OPR models.

Four Royal Society journals (Proceedings A, B, Royal Society Open Science and Open Biology) also continue to claim to be participating in the Publons TPR program and using OPR by publishing reviewers’ reports, decision letters and responses alongside published articles.20 The Royal Society encourages reviewers to disclose their names to authors, but this is a voluntary choice. Moreover, the publisher does not permit the publication of peer review information for rejected papers. The publisher’s blog notes that this greater openness has resulted in an increase in the number of non-Western authors, editors and reviewers in Royal Society journals, and a more representative gender distribution [89]. However, these benefits of openness are simply asserted without providing evidence, and it is not clear, for example, how OPR can better promote the observance of the principle of gender equality compared to, for example, double-blind peer review?

Now that the Publons experiment has been culled, answers to many pressing questions might never be known. Even though Publons was abruptly terminated almost 2 weeks ago, Wiley and this journal (BEER), as well as the Royal Society journals continue to advertise Publons as part of their peer reviewer rewards schemes. A future analysis should assess if journals and publishers continue to publicly advertise Publons-related services on their websites or in emails to academics.

Mandatory Versus Voluntary/Optional Clauses Related to Publons, and Opt-In and Opt-Out Options

Authors who wish to voluntarily peer review, but who might not necessarily want any peer reviewer rewards at Publons, for personal and/or professional reasons, might find themselves in an odd predicament. In such a case, where an author does not want their peer reviewer report alongside the published paper, will their request and choice be respected by the journal, publisher and WoS, and will it clash with any OPR objectives of Clarivate?21 Very importantly, is the use of the term “transparent peer review” being used by publishers or their journals to mandate peer reviewers to make their reports open, i.e., mandatory OPR? How can peer reviewers’ rights of opting in versus opting out of OPR be respected without unfairly applying pressure? Publishers’ mandatory policies for OPR, with an obligatory link to WoS (previously Publons), merits very careful scrutiny, in order to appreciate if reviewers’ rights of choice are being respected, and if there are suitable opt-in and opt-out clauses [61]. With such a clause, a peer reviewer cannot merely peer review simply because they are willing to do so, out of a vocational desire or for some other personal or professional reasons, they are forced to be associated with the WoS entry for the paper, i.e., almost oddly, recognition becomes mandated.

The same debate applies to ownership of reviewers’ reports. The authors are of the opinion that a reviewer is the copyright holder of a report that they create, as an original document. However, this ownership claim might clash with the publisher’s opinion, which might be that since peer reviewers are “recruited” by the publisher, or its journal, to complete a task—even if voluntarily—that they (i.e., the peer reviewers) are “hired” by the journal/publisher to complete this task (i.e., peer review), and so the peer review report is a “work for hire”. In such a case, copyright lies with the publisher, and with not the reviewer. The issue of ownership of peer reviewer reports is often not discussed by publishers, whereas confidentiality is over-emphasized, but it is a necessary discussion, especially if peer reviewers’ reports are added as open reports on journals’ websites or at WoS, as part of OPR, with or without a Creative Commons CC BY license.

To try and better appreciate if these contradictions were apparent, or real, and if they pose any deontological challenges, or suppress the rights of authors or peer reviewers, we decided to examine several of the websites linked to these statements and policies. The first website related to Publons, and previously hosted at Publons,22 used to indicate (August 3, 2022) that “410,396 Wiley reviewers have already added 2,477,655 of their reviews to Publons”, noting further that 1242 journals were using this “integrated” service. That web page indicated that peer reviewers could “get recognition even if your reviews are anonymous and the manuscript is never published”, but because this was a voluntary opt-in option, this suggests that peer review output may be skewed, i.e., some peer reviewers may be pumping out many peer reviews with the purpose of collecting credit, even if the content of those reports is neither open, publicly verifiable, nor good (i.e., superficial). In other words, how can the quality of peer reports that received “credit” at Publons be independently judged and quantified if they are not open [65]? Secondly, whereas one peer reviewer might be driven by vanity, seeking public recognition, even in the light of an opaque recognition platform (i.e., Publons, now possibly WoS), another peer reviewer might take a more modest approach, and merely seek to offer assistance, without seeking, or wanting, Publons-based recognition. In such a case, recognizing that the latter “class” of peer reviewers does not wish to be recognized voluntarily, how can both “classes” of peer reviewers be treated as “equals”? Will they be able to “win awards”23 if they have done the same amount of reviewing as the former “class” of peer reviewers? Curiously, that hyperlink from the Wiley Publons page leads to a 2018 (and not a 2019–2022) “Peer Review Awards” page, all of which are now defunct. This terminology also invokes a culture of gambling in academic publishing, which is not considered to be a scholarly principle [90]. The Publons page ironically then turns readers’ attention back to a Wiley page related to Publons.24 Both the Publons and Wiley websites continue to refer to the Wiley-Publons collaboration as a “pilot” program, labelled as the “Transparent Peer Review pilot”, most likely following the name assigned by the Clarivate service, Publons TPR.25 We emphasize that TPR here is not necessarily the same as OPR because not all peer reviewers’ reports are open. We thus challenge the claim that this program is/was transparent if peer reports’ content and peer reviewers’ identities are/were not openly, and thus transparently, indicated.

Conclusion

New services and practices, which are now commonly combined under the umbrella term “open science”, tend to declare (or assert) that they aim to make the process of scientific communication more transparent and fair. In particular, various approaches to conducting OPR are being actively tested today in order to make this process free from subjective prejudices, fake evaluations, and also to reward reviewers’ work. However, peer review standards in a wide range of journals and publishers still varies widely [91], even if many of them fall under the same umbrellas of ethics-promoting organizations such as COPE or the ICMJE. There is also doubt whether purely reputational gains will be a sustainable model for OPR to become the main or predominant peer review model because there are many aspects that may hinder the incentivization of academics to openly participate in such schemes [92]. We believe that the sudden disappearance of all Publons-related websites, even if the information was integrated into WoS, may demotivate some academics that may have invested heavily (personally and professionally) into the Publons brand. We also feel that the opaque handling of this transition, and poor public communication displayed by Clarivate throughout and about this process, suggests that peer review has taken one step back in its efforts to democratize and popularize the process.

In this study, we addressed the challenge of achieving consensus among authors and reviewers for conducting OPR using the now-defunct Publons platform, which was actively promoted by Clarivate along with other solutions built on that platform, including Publons TPR. We examined examples of open initiatives of individual journals, and found that the complete openness of the review process did not depend on the journal’s participation in pilot projects with Publons. At the same time, the lack of non-committed consideration of the results of OPR pilot projects by publishers was monitored, in particular, the lack of thorough consideration of feedback from authors and reviewers, as well as how the pilot projects affected the quality of peer review. Thus, despite the technological convenience, such good intentions to improve the quality of reviewing can have an opposite effect, and new services can increase the exploitation of free work of reviewers for the purpose of obtaining additional profits by commercial players. In addition, authors and reviewers may feel some psychological pressure due to their refusal to be as open as possible in the review process and, despite their declared voluntary participation, new services were pushing reviewers to seek to increase their quantitative metrics on Publons. It remains to be seen if the same culture, methodologies and campaigns will be employed by Clarivate as it seeks to fortify its peer rewards platform at WoS.

Finally, and also importantly, given that science, and its replication, may be in a state of crisis, including the core principle of industry-wide trust, it would be important to evaluate the post-publication peer reviewer (PPPR) reports that existed at Publons [93], and to appreciate their content in WoS to assess whether positive PPPR reports (praising research) outweigh negative ones (criticizing research or calling out errors or possible fraud). Goldstein [94] noted a deficit of PPPR reports relative to pre-publication reports at Publons, while also noting bias towards, or in favor of, life sciences (health and medical sciences), while Publons metrics were weakly and insignificantly correlated with bibliometric and altmetric indicators. Mondal and Mondal [95] were also surprised to find zero PPPR reports at Publons by Indian peer reviewers in the field of dermatology. Also of great importance, there will be a need to assess peer reviewers’ reports that are associated with retracted papers, especially those associated with fake peer review, paper mills, and other instances of publishing fraud [62], in order to determine whether those peer reviewers who have been rewarded at Publons (and now at WoS) may have been associated with such ethically and integrity-compromised papers, and what their role or responsibility was in approving the publication of such flawed literature, or whether they had detected any of the issues that led to the papers’ retractions during peer review.

Finally, some authors have sought to define the parameters that would allow an “excellent” Publons-associated peer reviewer to be distinguished, focusing on their productivity and bibliometric metrics and performance associated with their own citations, number of articles in WoS, number of words in a peer review report, and some other factors [57]. Similarly, Zhang et al. [59] searched for associations between review length and several factors linked to the peer reviewers themselves (gender, cultural background, disciplines, English proficiency, publications, and verified reviews). Lei [58] attempted to associate the H-index of reviewers with the JIF of journals, concluding that “reviewers of higher impact factor/average journal impact factor (IF/JIF) percentile ranked journals had better citation metrics than those of lower ranked journals” (p. 149). While those papers were certainly interesting from a bibliometric perspective, they failed to focus on the most important factor, namely the actual content of peer review reports in order to assess quality or to make associations with metrics or other bibliometric indicators, most likely because the vast majority of peer review reports at Publons were not open, i.e., those journals are not OPR. We are of the opinion that when attempting to discuss the excellence of a peer reviewer or of a journal’s peer review, that the greatest weighting should always be on the content of reports. The quality of that content may, in turn, be related to the guidance, motivation and supervision provided by a journal’s editors to its oft-volunteer peer reviewers [96]. And in order for that to be objectively judged and accurately assessed, journals must obligatorily revert to an OPR system.

Author Contributions

The authors contributed equally to all aspects of the paper.

Funding

No funding was received by either author for this research.

Declarations

Conflict of interest

The authors declare no relevant conflicts of interest.