RQ1: What benefits and risks do journalists consider in deciding whether to cover preprints?
|
A risk-benefit equation
|
Journalists weighed the risks of covering preprints against potential benefits for the public; audience needs were central this calculation |
The calculation is: ‘Do we think that the audience needs to hear the story now or can they wait six to eight weeks… for the story to be peer reviewed?’ And most of the time we think the wait is important and we tend to for the vast majority to avoid picking preprint [J1] |
Accessibility (Benefit)
|
Preprints were valued because they were free to access. This benefit was both practical (i.e. easier for journalists) and ethical (i.e. a belief that knowledge should be free) |
Preprints, it’s easy, because they’re freely available… Luckily, for Covid, a lot of things are open access. Maybe this is the future of science. It should be. But for now, we have to manage as we can. [J12] |
Timeliness (Benefit)
|
Preprints were valued because they allowed more timely access to relevant research than was possible through peer reviewed research |
[As journalists,] our allegiance is to our readers, and getting accurate but timely information to readers. . . When people are dying, you gotta get things going a little bit. And so that’s, I think, what we’ve seen in the last year, in this argument over preprints. [J5] |
Potential to misinform (Risk)
|
Covering preprints was seen as risky because unreviewed results could turn out to be false or flawed, contributing to misinformation |
There was a study that was bad about running, how doing more running caused you to expel the virus. That was totally bunk. It was one of the things that like after a week of it making the rounds everywhere, everyone realized, ‘Oh, wait. That wasn’t true at all.’ [J2] |
Difficult to verify (Risk)
|
Lacking the expertise, resources, or skills needed to verify preprints was described as a major challenge |
Just because I have a Ph.D. in [the field] doesn’t mean that I have in most cases the right expertise to look at the paper and perfectly judge it. This is why we tend to favor covering that at peer reviewed. [J1] |
RQ2: What practices do journalists use to find, verify, and communicate the preprints they cover?
|
Active strategies (Finding preprints)
|
Journalists actively searched for new preprints, most often direct from the servers themselves |
With preprints, you tend to have to just go on, like, the preprint websites and kind of just sift through it and, like, see. [J11] |
Passive strategies (Finding preprints)
|
Journalists received preprints from other sources, such as PR services and from authors themselves |
[Preprint research] only comes into my life usually when I’m already sort of interviewing someone and they say, well, we have this piece that’s out for peer review. [J11] |
Extra skepticism (Verifying preprints)
|
Journalists felt that an added layer of skepticism was needed when verifying preprints; often linked to a trust in peer review as a quality control mechanism |
There’s another level of skepticism that should go into reporting on preprints, because there’s one less safety net, basically, that the research has gone through [J8] |
Critical reading (Verifying preprints)
|
Journalists verified preprints by reading studies with a critical eye, asking critical questions of the methods, sample, analyses, and findings; this practice was seen as a standard aspect of any science journalism |
We’re using the same toolkit, our toolkit for looking at a paper and evaluating its newsworthiness. We’re like, ‘Okay. Well, is this a good paper? Is the science good? Do the statistics make sense to us? Do the results actually answer the question that it says it answers, and what’s left out?” We ask those things of formally-published—you know, if it comes out in Science. We ask those questions, too, because sometimes the answer is ‘No’ and sometimes the answer is like, ‘Actually, this does seem dicey’ [J16] |
Triangulation (Verifying preprints)
|
Journalists verified preprints by comparing findings to information from other trusted sources, such as peer reviewed papers, experts, or other preprints |
If we can find some article, in that case, we look more at similar work in the literature to back up some of the claims [J1] |
Do your own peer review (Verifying preprints)
|
Journalists used outside scientists to verify preprints—to comment on methods, results, and significance in a process resembling scholarly peer review |
In my opinion you can’t do an unsourced preprint coverage. Like you need to ask like 10 doctors or 10 you know epidemiologists or 10 whatever relevant you know specialists there are like, "What did you think of this" and then you need to include the back-and-forth that naturally results from that to do it responsibly. [J3] |
Intuition (Verifying preprints)
|
Journalists relied on trust and intuition as a substitute for, or an addition to, other preprint verification strategies; this gut instinct was viewed critically by some journalists |
I just come back to that idea that it is so much about the individual reporter’s gut feeling about something. That is, I think, a little scary. Fortunately we have a lot of good reporters working on these things, but, but yeah I don’t think that anybody has like a framework that’s agreed upon for how to approach these things. [J4] |
State that research is unreviewed (Communicating preprints)
|
Journalists felt it was important to disclose the unreviewed nature of the preprints they cited (e.g., by labeling it a preprint, stating it had not yet been peer reviewed) |
people can share these articles on social networks and everywhere like they’re peer-reviewed–like they’re something that’s already textbook knowledge, which is far from it. This is something that should also be highlighted in the article, and I try to highlight it: ‘It’s a preprint. It’s not peer reviewed yet’ [J12] |
Contextualize (Communicating preprints)
|
Journalists added context to preprint findings by comparing them with information from other sources |
I think contextualized properly, they’re a really useful and valid source of reporting… where possible, you should try and bolster it with other information [J9] |
Highlight limitations (Communicating preprints)
|
Journalists emphasized the importance of describing caveats, weaknesses, and limitations associated with the preprints they cited |
Something that I find fantastic in a lot of medical articles that are hardly found in any other discipline is discussion on limitations, which we are trying to include more and more in our articles [J1] |
RQ3: How has the COVID-19 pandemic affected journalists’ use of preprints?
|
A new normal
|
COVID-19 preprint coverage was viewed as a complete paradigm shift in science journalism, one likely to continue post-pandemic. |
we didn’t consider them as really newsworthy items before Covid. Now, we consider them. . . like–something that should be covered like a normal peer-reviewed article, which is a complete paradigm shift, maybe, in science covering [J12] |
A moderate shift
|
COVID-19 preprint coverage was viewed as a more temporary change in journalism practice, an exception caused by the pandemic |
I think that among myself and sort of my friends/peer group you know I think we’re pulling back a little bit and I doubt that arXiv is the place a lot of medical reporters are going to eagerly pull reporting from [J4] |
Undecided
|
Journalists unsure whether preprint news coverage would persist post-pandemic. |
It will be interesting to see like, what the implications of that are going forward…are preprints going to be covered more generally even outside such an urgent context? [J3]. |