COVID-19 and the research scholarship ecosystem: help!

Abstract

Objectives

Data sharing practices remain elusive in biomedicine. The COVID-19 pandemic has highlighted the problems associated with the lack of data sharing. The objective of this article is to draw attention to the problem and possible ways to address it.

Study Design and Setting

This article examines some of the current open access and data sharing practices at biomedical journals and funders. In the context of COVID-19 the consequences of these practices is also examined.

Results

Despite the best of intentions on the part of funders and journals, COVID-19 biomedical research is not open. Academic institutions need to incentivize and reward data sharing practices as part of researcher assessment. Journals and funders need to implement strong polices to ensure that data sharing becomes a reality. Patients support sharing of their data.

Conclusion

Biomedical journals, funders and academic institutions should act to require stronger adherence to data sharing policies.

“Access to information is a human right, but it is often treated as a privilege. This has to change – and it will take all of us to make it happen” Erin McKiernan

Coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has had a catastrophic mortality and morbidity impact on the world. The Institute for Health Metrics and Evaluation, at the University of Washington, report (January 15^th 2021) more than 2 million deaths world-wide due to the virus (in less than a year since it was declared a pandemic) and estimate 3 million COVID-19 infections, daily [1]. The Spanish flu, approximately 100 years ago, is the closest public health comparator. As I write this paper, Health Canada and regulators in other jurisdictions have granted approval for several vaccines to combat the virus. The vaccines have started to be given, albeit at a much slower pace than anticipated, to people in many parts of the world. This is really good news and might have been even better had governments contracted out the delivery of the vaccines to groups that know the distribution business well. Should another pandemic happen, engaging entities whose daily routine is distribution and delivery might facilitate making the process more efficient. What is less encouraging is the impact that COVID-19 research is having on the research ecosystem, specifically open science and research integrity.

1. Open science practices

There is a recognition that open science accelerates knowledge discovery. COVID-19 provides an exuberant example of this. Within 12 days of sharing the severe acute respiratory syndrome coronavirus 2 genome, it was sequenced. Subsequent research has created an unspecified number of jobs, globally, thought to be in the hundreds of thousands. What is concerning, and insufficiently discussed in the literature, is the lack of direct access to data underpinning much of the COVID-19 research output, in general, and the all-important results of the vaccine trials, in particular. Setting aside the problems of reporting research by press release [2], problematic regardless of the public health importance, none of the data underlying the vaccine trials is directly and easily available to the scientific community, patients or the broader community.

Lack of access to the trial data might explain why none of the vaccine trials were published in BMJ or PLOS Medicine. Both journals have the highest data sharing standards of journals, to date [3]. Had more journals followed these commendable standards, data from the trials would already be available to readers. Not sharing trial data also appears to go against the wishes of patients [4] without who's altruism randomized trials could not be conducted.

For countries that have spent countless billions of dollars either helping to develop and/or procure the vaccines, it is odd that no measures were put in place to ensure easy access to the data. The necessity of data sharing is not simply an esoteric need of a fringe element of the scientific community. It is built on evidence. Regardless of the severity of language one wishes to use, there is general agreement that reproducibility is a problem in research [5]. The psychology community, for example led by the efforts of Nosek and colleagues through multiple studies, have clearly shown that a worrying fraction of research results cannot be replicated [6]. Given these results coupled with the consequential public health magnitude of the pandemic, it is unfortunate that health policy experts did not recommend setting aside a small budget (perhaps < 0.0001% of the global COVID expenditure) to ensure that the vaccine trials, and other COVID related research, could be replicated. Wouldn't such replication provide added confidence about the results and conclusions of this research?

While there have been valiant efforts to ensure easy access to COVID research and sharing of its underlying data – the Wellcome initiative early on in the pandemic endorsed by hundreds of organizations [7] – this has not resulted in meaningful change. An analysis of 535 COVID articles on preprint servers found that “only 21% of authors included data availability statements, and only 11% of those made their data available in external repositories” [8]. It is time for journals and funders to up their credibility and invoke higher data sharing standards?

What's really important is to encourage the development and implementation of new tools into the scholarly communication ecosystem, such as a digital dashboard to enable users, be they funders or universities, to benchmark their open science practices and think about how to improve them. Critical to any change in behavior or practice is providing the feedback loop. For example, there is reasonable evidence that data sharing does not really happen in biomedicine [9]. If we can provide feedback to institutions about this and engage them in discussions – from leadership to a bottom-up approach with faculty and staff – ascertaining what might be a reasonable data sharing threshold to aim for immediately and in the future. Isn't this more likely to be a societal benefit rather than the usual journal impact factor madness?

2. Research waste and quality

At least one estimate suggests close to 100,000 COVID-19 research articles have been published [10]. This is an enormous addition to the compendium of knowledge about the pandemic. Is this knowledge useful? Is it wasteful? It's likely too early to truly know if it is wasteful although conceptually it is reasonable to think so. There has been little effort to think about COVID research in a global unified manner. Instead, individual jurisdictions have mandated COVID research often specific to their interests. Were these interests duplicated across countries? Is there an unnecessary duplication of effort? There are many multiples of tens of hydroxychorlinequire trials funded through public funding. Any reasonable inspection of the inception research ‘proclaiming’ the virtues of the hydroxychorlinequire would quickly and easily ascertain the foundation of the claims were highly questionable [11]. The extent to which researchers registered their COVID-19 protocols is unclear. This might have helped reduce some waste [12]. As London and Kimmelman ask, are “we against pandemic research exceptionalism” [13]. We likely need more time to more thoroughly investigate the level of waste in COVID-19 research. Is the emerging knowledge useful?

3. Incentivizing and rewarding data sharing as part of researcher assessment

We don't yet know the exact curve that adequately describes the growth of COVID-19 research in our ecosystem. Whether it is exponential or something else, there is little doubt that researchers will be adding these outputs to their CVs. Given the traditional standards of academic promotion and tenure [14], it is all but certain that the currency of publications will result in a subsequent increase in promotion and tenure on the back of COVID research. Academic institutions should take this opportunity and demote counting the number of publications and journal impact factors, which provide no information about the quality of research being counted during COVID and use more meaningful open science practices. The Declaration on Research Assessment group helped lead the charge, strongly advocating for the discontinuation of journal impact factors for researcher assessment [15]. The Hong Kong Principles, and others, advocate for using open science practices as a meaningful way to assess researchers [16]. Why should researchers who publish COVID-19 research without making their data available be able to advance their careers? Does such research output have a societal benefit?

There is no doubt that COVID-19 research has brought tremendous pressure to bear on journals. While there is no public data regarding the increase in the volume of submissions, privately I've been told it has more than doubled in several journals. This likely also increased pressure of potential peer reviewers. I now typically receive 10 to 20 requests to peer review, daily. I've given up formally responding to journals given the time it takes (only to be asked for names or other potential reviewers). Now I simply delete almost all of the requests. Unfortunately, journals do not require peer reviewers to be trained resulting in a shaky system even without the pandemic. There has been an alarming number of retractions of COVID-19 articles [17] some of which would have been easy to identify on submission had there been greater oversight. Two recent high-profile publications (and subsequent retractions) of pharmacoepidemiology studies reporting the effectiveness and risk of hydroxychloroquine in COVID-19 patients received international media attention [18,19]. Transparent and complete reporting of these studies could have provided peer reviewers and editors with sufficient information to question the methods used and the validity of results. Since these studies used routinely collected health data, the guidelines for the REporting of studies Conducted using Observational Routinely collected health Data should have been applied to ensure complete reporting of the research [20]. Unfortunately, editors do not routinely ask their peer reviewers to use reporting guidelines as part of their review process [21].

It may be too early to assess the trustworthiness of the COVID-19 research output. What is more evident is that research integrity has likely been compromised between journals and their readers. The international committee of medical journal editors state “In return for the altruism and trust that make clinical research possible, the research enterprise has an obligation to conduct research ethically and to report it honestly”. Trustworthy research also means that research reports must be useful to readers. The increased retractions and other concerns about COVID-19 research output means that journals and funders will need to work hard to ensure research integrity is improved.

4. Assessing open science practices

Notwithstanding the tremendous effect and speed with which the trials were planned, conducted and reported, there are methodological concerns about them. Peter Doshi has discussed some of these [22], such as the absolute risk reduction rate not being reported and possible accidental unblinding, Other concerns include, were the outcomes for some of the trials correct and patient centric? Access to the trial data might help resolve some of the questions and concerns.

This is also the case for the Transparency and Openness Promotions (TOP) guidance [23] created by scientists representing journals, funders, and professional societies to better align publishing and funding practices with best practice in the conduct of research. The TOP Factor [24] is a metric that reports the policies that a journal is taking to implement policies that comply with the TOP standards. This journal is a TOP signatory but an examination of their TOP factor (a means to replace the very shaky journal impact factor) reveals a score of 1 for data citation (out of a possible total of 30). The journal is silent on the several aspects concerning data sharing.

Many paywalled journals have unlocked their doors to COVID content. This is a positive move and provides readers, particularly health care providers and patients, globally, with access to content regardless of their resources. Open science is definitely an equalizer and an important enabler of equity, diversity and inclusiveness. Why stop at COVID content? Aren't there other diseases worthy of open access? In late 1971, President Nixon declared war on Cancer. More than 40 years later President Obama asked his then vice present Joe Biden to lead a similar effort. My suggestion is to declare war on paywalls. Such a policy might enable all journals to be open rather than for the privileged few who have the resources to buy subscriptions. Plan S [25], an international effort to ensure that publicly funded research is disseminated via open access outlets, might be a starting point.

5. Audit and feedback

The scientific ecosystem cannot change without feedback. I'm unaware of any funder or publisher who regularly provides audits and feedback as to whether they are meeting their respective missions. Such audits should now be mandatory. There are examples of how audits can enhance the entire system. A few years ago, Goldacre and colleagues developed a trial tracker monitoring system [26] providing feedback on the proportion of registered trial protocols that had a subsequently completed trial published. Even in my own institution this data proved effective. The hospital leadership put in place a monitoring system to help reach a much higher threshold of published clinical trials. Why don't funders and publishers set aside a small fraction of their revenues and do likewise for a broad spectrum of open science practices concerning COVID research?

6. Actions to promote data sharing

There is no vaccine to block the impact of questionable COVID researchers and COVID research outputs. The plume of questionable open science practices and research integrity continues and there are few controls in place as to how the research might be used. Everybody in the research ecosystem needs to work more closely together on ways to maximize open science practices. Society will be the beneficiary. Perhaps it is time for this journal and others to push academic institutions and funders to implement policies that incentivize and reward data sharing as part of researcher assessments. At the same time journals themselves should commit to doubling down on their effects to ensure data sharing is an integral part of journal submission criteria.