Reporting health services research to a broader public: An exploration of inconsistencies and reporting inadequacies in societal publications

Reinie G Gerrits; Michael J van den Berg; Anton E Kunst; Niek S Klazinga; Dionne S Kringos

doi:10.1371/journal.pone.0248753

. 2021 Apr 7;16(4):e0248753. doi: 10.1371/journal.pone.0248753

Reporting health services research to a broader public: An exploration of inconsistencies and reporting inadequacies in societal publications

Reinie G Gerrits ¹, Michael J van den Berg ¹, Anton E Kunst ¹, Niek S Klazinga ¹, Dionne S Kringos ^1,^*

Editor: Quinn Grundy²

PMCID: PMC8026015 PMID: 33826619

Abstract

Introduction

Little is known about the accuracy of societal publications (e.g. press releases, internet postings or professional journals) that are based on scientific work. This study investigates a) inconsistencies between scientific peer-reviewed health services research (HSR) publications and non-scientific societal publications and b) replication of reporting inadequacies from these scientific publications to corresponding societal publications.

Methods

A sample of HSR publications was drawn from 116 publications authored in 2016 by thirteen Dutch HSR institutions. Societal publications corresponding to scientific publications were identified through a systematic internet search. We conducted a qualitative, directed content analysis on societal publications derived from the scientific publications to assess both reporting inadequacies and determine inconsistencies. Descriptive frequencies were calculated for all variables. Odds ratios were used to investigate whether inconsistencies in societal publications were less likely when the first scientific author was involved.

Results

We identified 43 scientific and 156 societal publications. 94 societal publications (60.3%), (associated with 32 scientific publications (74.4%)) contained messages that were inconsistent with the scientific work. We found reporting inadequacies in 22 scientific publications (51.2%). In 45 societal publications (28.9%), we found replications of these reporting inadequacies. The likelihood of inconsistencies between scientific and societal publications did not differ when the latter explicitly involved the first scientific author, (OR = 1.44, CI: 0.76–2.74); were published on the institute’s or funder’s website, (OR = 1.32, CI: 0.57–3.06); published with no involvement of a scientific author, (OR = 0.52, CI: 0.25–1.07).

Conclusion

To improve societal publications, one should examine both the consistency with scientific research publications and ways to prevent replication of scientific reporting inadequacies. HSR institutions, funders, and scientific and societal publication platforms should invest in a supportive publication culture to further incentivise the responsible and skilled involvement of researchers in writing both scientific and societal publications.

Background

In academia, scientific research publications are an important source of knowledge, as well as a means of research dissemination [1]. Outside the research community, however, most people take note of research findings through non-scientific, societal publications such as press releases, newspapers, social media, internet postings or professional journals [2–4]. The content of societal publications impacts the thinking, debates and decisions of the general public, as well as those of patients, health professionals and policymakers [4–6]. Consequently, researchers who publish a scientific paper are increasingly incentivised to translate their findings into a corresponding societal publication, in order to reach broader, often non-academic audiences [7].

By necessity, the authors of societal publications simplify scientific messages and conclusions for their lay target group [8]. Although this can be done in a responsible manner, it does present a risk for misrepresentation and misinterpretation of the research findings [9]. Previous studies on biomedical publications concluded that unjustified causal claims are introduced in 20% to 33% of press releases, and that 40% of news articles give more explicit health advice to the readers than was expressed in the underlying scientific publication [10–14].

In health services research (HSR), less is known about the potential misrepresentation or misinterpretation of evidence in societal publications. HSR aims to provide usable evidence for policy and for management of health and health care [5]. This practice-oriented ambition amplifies the importance of accuracy in all messages and conclusions relayed in societal publications [15].

Researchers are often expected to have a societal impact beyond their scientific impact. Funders of HSR increasingly demand strategies to achieve a societal impact. Methods for measuring impact are being developed and refined [16–18]. Researchers, however, may lack the experience or capability to write responsible societal publications that accurately reflect their scientific findings [10, 19]. Some previous research has concluded that a researcher’s involvement is not associated with better societal publications [10]. Researchers may have difficulty working with journalists, or they may lack the ability to explain their findings in simple terms [19]. Moreover, fellow researchers may take a critical view of colleagues who invest considerable time in media attention, and thus discourage them to put significant efforts in writing societal publications [20].

Messages and conclusions are too often poorly reported in the scientific publications [21, 22]. In a previous assessment of peer-reviewed HSR publications written in the Netherlands for an international academic audience, we found per publication a median of 6 out of 35 possible ‘questionable research practices’ in the reporting of messages and conclusions [23]. In the current study, these questionable research practices will be called ‘reporting inadequacies’. They include conclusions that are insufficiently supported by the research results, recommendations that are not justified and limitations that are inadequately explained [23]. Even if a researcher tries to avoid inconsistencies in a subsequent societal publication, such reporting inadequacies in the original work may well find their way to a broader audience. As the scientific publication is used as the standard, reporting inadequacies will likely be copied or ‘replicated’ to societal publications.

Given the potential impact of societal publications on policy and practice, knowledge of responsible reporting in societal publications, and how researchers can achieve it, is important for the HSR community [24]. Such knowledge is currently inadequate [10, 14]. Whereas the previous studies in the field of biomedicine focused largely on press releases and newspapers, broader insights are needed into the full scope of societal HSR publications, including information sources such as fact sheets, web pages and articles in professional journals.

The aims of this study are to explore

whether societal publications on HSR are consistent with the messages reported in the underlying research papers
whether reporting inadequacies in scientific HSR publications are replicated in societal publications
whether fewer inconsistencies occur in societal publications if the first scientific author is involved in writing them.

Methods

In a collaboration funded by the Netherlands Organisation for Health Research and Development (ZonMw), thirteen Dutch academic and non-academic HSR institutions (see Acknowledgements section for the listing) took part in several studies designed to promote responsible reporting. The present study builds on the results of a previous study that identified reporting inadequacies in scientific publications [23].

No patients or human participants were involved in this study. A waiver for ethical approval was obtained for this study from the Medical Ethics Review Committee at Amsterdam UMC.

To investigate inadequacies in research reporting and inconsistencies between scientific and societal publications, we employed a mixed-methods approach. We first conducted a directed qualitative content analysis of scientific HSR publications and related societal publications that derived from them, followed by a quantitative description of the results.

Subsample of scientific publications

We based our selection of scientific publications on a random sample of 116 such publications authored in 2016 by researchers from the thirteen participating HSR institutions.

In short, complete publications lists were obtained from all institutions, from which a total of 717 scientific HSR publications were identified, applying commonly used definitions of HSR from Plochg and colleagues [25] and Lohr and Steinwachs [26]. Two researchers independently selected publications based on these definitions. Publications that were selected by both researchers were included. Remaining publications were included or excluded after review by the full research group. A sample of 116 publications was included. Two researchers independently assessed those publications for inadequacies in the reporting of messages and conclusions, using a validated checklist of 35 possible inadequacies. Each inadequacy was recorded on an assessment form. During periodic consensus meetings, the reviewers compared their assessment of all items. Inconsistencies between the individually assessed reporting inadequacies were identified, discussed and adapted. Any remaining disagreements (n = 2) were resolved by a third, senior, researcher. The list of assessed reporting inadequacies is added to S1 Appendix. An extensive description of the sampling, development of the assessment tool and the assessment of these publications has been published elsewhere [23].

The current study confined itself to scientific writings that had one or more associated societal publications. Quantitative, qualitative and mixed-methods studies were included. We aimed to include publications with the most and the least reporting inadequacies, to allow for detection of differences in the replication reporting inadequacies. We sampled scientific publications with relatively high and low numbers of reporting inadequacies based on the median of inadequacies per publication (‘high’ being more than 6 and ‘low’ fewer than 6 reporting inadequacies). We did not include scientific publications with the median of 6 reporting inadequacies.

Sample of societal publications

Societal publications corresponding to scientific publications were identified through a systematic internet search. We included societal publications that (1) were in the public domain, (2) contained messages on the same research as the corresponding scientific publication (including statements on the results, conclusions, discussion, recommendations or implications) and (3) were written in Dutch (aside from social media messages communicating the title of the publication). Societal publications corresponding to the specific scientific publication were identified based on the content of the publication, containing mention of the author, research program, specific study title or aim.

For each scientific publication, a variety of internet sources were consulted, following a systematic search strategy. We searched or consulted (1) specific institute websites, funders’ websites and Altmetrics; (2) document databases of Dutch government and parliament (including www.rijksoverheid.nl, www.tweedekamer.nl, and https://zoek.officielebekendmakingen.nl/); (3) databases of Dutch popular science periodicals (https://www.skipr.nl/zoeken?q, https://www.medischcontact.nl) and a periodical aimed at medical professionals https://www.ntvg.nl/zoeken); (4) an existing database of Dutch newspaper articles (www.lexisnexis.nl); (5) public social media platforms (LinkedIn and Twitter) of the authors and the institutes; (6) the Google search engine, to identify publications from further sources. In the Google search, we entered search terms (see next section) and followed all links provided in the first 30 results, as we did not expect to find relevant societal publications beyond that ranking. To ensure that earlier searches did not affect the Google search, our browser history data, including cookies, were deleted beforehand. All internet sources were accessed in the month of August 2018.

Search terms and filters

For each scientific publication, specific search terms were derived from Dutch translations of key terms in the title and abstract; also included were the name of the first author’s institution, the authors’ names and the funder(s). Any new key terms found during the search were added.

Search strings were used if the database enabled the use of logical operators. Because results of a study may be reported prior to the appearance of the scientific publication, we included societal publications appearing up to two years beforehand and one year afterwards (presuming that all societal publications would appear within a year). All search terms were discussed and approved by two members of the project team (RG and NK).

Analyses of messages in societal publications

Societal publications were analysed using a directed content analysis approach [27]. A directed content analyses is a more structured approach than traditional content analyses. It starts with an initial coding scheme based upon prior knowledge [27]. First, we identified distinct messages and conclusions in the societal publications that related to the corresponding scientific publication. Messages could be a single sentence or a section of the text elaborating on the same topic; a single research result or a concluding statement was marked as a distinct message. Multiple messages might be identified in a single societal publication.

Second, we assessed whether the message in the societal publication was consistent with that in the corresponding scientific work. A message was considered consistent if it conveyed the same meaning as the scientific results, discussion or conclusion and if no changes, additions or subtractions had been made with respect to the content of the scientific assertion. In line with our directed content analyses approach, an initial coding scheme based on the possible inconsistencies was prepared, informed by other checklists for public reporting [12, 28]. To discover other types of inconsistencies not included in those checklists, we iteratively improved the coding scheme during the first stages of coding, adding new aspects that emerged during the coding.

Third, we determined whether a message in the societal publication replicated a reporting inadequacy in the scientific publication. Messages in the societal writings were compared to any reporting inadequacies recorded during the previous assessment of the corresponding scientific paper [23]. A message that identically reproduced the reporting inadequacy was marked as a ‘replicated reporting inadequacy’.

Finally, for each societal publication we gauged the likelihood of the first scientific author’s involvement (‘named as author’, ‘published on institute or funder web page’ or ‘no involvement’).

One coder (RG) performed the analysis. The identification of the messages and the coding method of the first ten publications were checked by project members (NK, DK and MB) and thoroughly discussed until the coding method and scheme had been agreed. To ensure consistency of analyses, we had 10 per cent (n = 16) of the analysed societal publications randomly checked by DK. Prompted by the check, we decided to revisit the final 15 societal publications to improve possible inconsistencies with earlier codes, and to correct one identified replicated reporting inadequacy. Analyses were conducted in MAXQDA.

All messages were coded as categorical variables. Counts of inconsistencies and reporting inadequacies were calculated within the program MAXQDA. S2 Appendix provides the coding scheme.

Statistical analyses

The codes in MaxQDA were transferred to a SPSS dataset. Odds ratios were calculated to compare the frequencies of inconsistencies in societal publications (1) authored by the first scientific author, (2) published on the institute’s or funder’s web page or (3) published with no involvement of a scientific author who was part of the scientific publication. Three tests were performed, each comparing one category to the other two categories combined. A societal publication was deemed ‘inconsistent’ if at least one message in it was identified as inconsistent.

Results

Characteristics of analysed publications

We conducted the structured internet search until we identified 46 scientific publications (23 with high and 23 with low inadequacies) that had associated societal publications. We identified the included 46 publications after examining 84 scientific publications in our sample (46 with the highest number of inadequacies and 38 with the lowest number of inadequacies). We examined 188 societal publications obtained in our internet search and excluded 32 of them from further analysis because they only described the applied methodology without reference to study results. That left three further scientific publications without corresponding societal ones, so that 43 scientific publications and 156 societal publications remained in the final sample. Fig 1 provides a flow diagram describing the inclusion and exclusion process.

Table 1 shows the characteristics of the included societal publications.

Table 1. Characteristics of the analysed societal publications.

Type of societal publication	Societal publications, n (%)
News message	37 (23.7)
Institute web page	21 (13.5)
Magazine	20 (12.8)
Social media	19 (12.2)
Professional journal	19 (12.2)
Report	13 (8.3)
Thesis summary	9 (5.8)
Funder web page	8 (5.1)
Fact sheet	7 (4.5)
Video	2 (1.3)
PowerPoint slides	1 (0.6)
Linkage to scientific reporting inadequacies	Societal publications, n (%)
Linked to high (>6) reporting inadequacies in scientific publications (n = 20)	62 (39.7)
Linked to low (<6) reporting inadequacies in scientific publications (n = 23)	94 (60.3)
Publication time of the societal publication
Before publication of scientific publication	30 (19.2)
After publication of scientific publication	126 (80.8)
Total societal publications	156 (100.0)

Open in a new tab

Scientific publications had a maximum of 14 associated societal publications, a minimum of 1 and a median of 3. The majority (n = 27) had 1 to 3 societal publications, 11 had 4 to 6 societal publications, and 5 had more than 6. A total of 60.3% of the societal publications corresponded to a scientific publication with low reporting inadequacies; 39.7% were linked to a publication with high inadequacies.

Consistency of reporting between societal and scientific publications

In the 156 analysed societal publications, we identified 577 distinct messages, 342 (59.3%) of which were consistent with the corresponding message in the underlying scientific publication. Multiple types of inconsistencies were sometimes identified within a single societal publication, such as altered conclusions or differing interpretations of outcomes.

Inconsistencies in conclusions

We identified inconsistencies in conclusions in 64 out of 156 societal publications (41.0%). The majority of inconsistencies in societal publications concerned the scientific research conclusions. Conclusions might be altered entirely (in 13 publications) or partially, where some words or sentences were added (in 14) (e.g. a study concluding that patients have trouble speaking up, concludes patients ‘never’ speak up.) Moreover, conclusions were presented that were not underpinned by the scientific results or discussion. Some publications gave interpretations of the study results that were not included in the scientific work (17 publications) or added strong rhetoric to conclusions that was inconsistent with the scientific verdict (e.g. implying that a problem was worse; 12 publications). Some societal publications put forward conclusions that could not have resulted from the scientific study (21 publications) or that were derived from the introduction (4 publications) (e.g. the societal publication states that the study concludes smoking is harmful, while the scientific publication concludes on measures that would reduce harm). Some societal publications even contradicted the conclusions from the corresponding scientific publications (10 societal publications) (e.g. the societal publication states improvement in attitudes, while the scientific publication states attitudes showed no change).

Inconsistencies in results

We identified inconsistences in 38 out of 156 societal publications (24.4%). In various societal publications, new results were introduced that were not reported in the corresponding scientific publication (22 societal publications) (e.g. mention of additional secondary analyses not included in the scientific publication, or mention of an additional theme not included in a qualitative analyses). Results were reported in different combinations that changed the interpretation of the results (in 7 publications) (e.g. from a survey study the societal publication picks a different combination of results than the main results mentioned in the conclusion of the scientific publications). Some quantitative results were altered with respect to the figures given in the scientific publication (e.g. differing exact percentages) or qualitative results were worded differently, causing the core meaning of the scientific findings to change (13 publications). Non-significant results were presented as significant (in 1 publication), such as referring to a ‘lesser effect from this intervention’, whereas no effect had been indicated or argued in the scientific publication.

Inconsistencies in recommendations

We identified inconsistencies in recommendations in 25 out of 156 societal publications (16.0%). Recommendations differed from those made in scientific publications in three ways: (1) entirely new recommendations for policy or practice were put forward in the societal publication, whilst not mentioned in the scientific publication (in 21 societal publications) (e.g. the scientific publication mentions no concrete recommendation, but the societal publication concretely recommends the implementation of a national policy); (2) relevant limitations of the recommendations given in the scientific publication were omitted in the societal publication (1 publication) (e.g. the societal publications recommends implementation of the program. However, the scientific publication mentions this is only effective if another party takes action, which was considered unlikely); (3) elements of recommendations given in societal publications were omitted in the scientific publication (6 societal publications) (e.g. the societal publication states what the results cannot be used for, while the scientific publication does not).

Inconsistencies in the reporting of conditions in the conclusion

We identified inconsistencies in the reporting of conditions in the conclusion in 4 out of 156 societal publications (2.6%). In these four societal publications, conditions affecting the study conclusions were left out, although the scientific publication explicitly made the conclusions subject to those conditions (e.g. with an ‘if’ or ‘when’ statement as part of the conclusion).

Inconsistencies in the reporting of implications for policy and practice

Implications for policy and practice must be differentiated from recommendations: implications describe the importance of the findings for policy and practice, while recommendations are specific measures that could improve policy and practice. In four societal publications out of 156 societal publications (2.6%), implications for policy and practice were reported that were not mentioned in the scientific publication (e.g. the societal publication mentions the possible implication that the findings might aid in saving time in healthcare provision, although this was not mentioned in the scientific publication).

Inconsistencies in the reporting of causality

Four societal publications out of 156 publications (2.6%) contained statements on potential causal relationships that were not mentioned in the scientific publication, and causality was implied without mention of mediating influences (e.g. one societal publication mentioned a correlation, as opposed to the scientific publication, where a causal relationship was claimed).

Inconsistencies in reporting generalisations

Three societal publications out of 156 (1.7%) generalised findings beyond the setting described in the scientific publication–to a different time period or geographical location, as from an urban to a rural setting; to different population characteristics such as gender, ethnicity or age; or to settings or institutions not included in the research.

Objectives not included in the scientific publication

In one societal publication out of 156 societal publications (0.6%), a study objective was added and discussed that was not included in the scientific publication (nor in any related research project).

Replication of reporting inadequacies from scientific to societal publications

Reporting inadequacies found in 22 of the 43 (51.2%) included scientific publications were reproduced in corresponding societal publications. From our checklists of inadequacies in scientific reporting, we identified nine types of inadequacies that were replicated in societal publications:

‘Conclusions do not adequately reflect the findings as presented in the results section’ (from 23.3%, n = 10 of the 43 scientific publications)
‘Recommendations do not adequately reflect the results in the context of the referenced literature’ (26.3%, n = 7 of the 43 scientific publications)
‘The title does not adequately reflect the main findings’ (9.3%, n = 4); that is, the inadequate title of the scientific publication was replicated in a societal publication.
‘The sampling methodology does not allow the type of generalisation provided’ (7%, n = 3 of the 43 scientific publications)
‘The conclusions in the abstract do not adequately reflect the conclusions in the main text [of the scientific publication]’ (4.7%, n = 2 of the 43 scientific publications); that is, inadequately reported conclusions from the abstract were replicated in a societal publication.
‘A potential causal relationship claimed in the discussion paragraph is not justified’ (4.7%, n = 2 of the 43 scientific publications)
‘Implications for policy and practice do not adequately reflect the results in the context of the referenced literature’ (2.3%, n = 1 of the 43 scientific publications)
‘The abstract does not adequately reflect the main findings’ (2.3%, n = 1 of the 43 scientific publications)
‘Generalising findings to geographical locations not included in the original study is not justified’ (2.3%, n = 1 of the 43 scientific publications)

The role of the first scientific author in inconsistencies appearing in societal publications

From our sample of 43 scientific publications, 26 first authors were named as authors of a societal publication (60.5%). Some 34 scientific publications were linked to at least one societal publication that did not explicitly state involvement of the first author (79.1%). Research from 20 scientific publications was summarised on the website of a research institute or funder without explicit mention of the involvement of the author (46.5%).

Odds ratios were calculated comparing the frequencies of inconsistencies in societal publications (1) authored by the first author of the scientific publication, (2) published on the institute’s or funder’s website, and (3) published elsewhere without explicit involvement of the scientific author. No associations were found between the number of inconsistencies in societal publications and any of those three conditions (Table 2).

Table 2. The role of the first scientific author in the occurrence of inconsistencies between a societal and a scientific publication; odds ratio for each category (row) compared to both others (N = 156).

Type of societal publication	Author involvement, n (%)	At least one inconsistency found, N (%)		Odss ratios and confidence intervals
		Yes	No
No involvement of scientific author	84 (53.9)	54 (64)	30 (36)	OR^a 1.44
No involvement of scientific author	84 (53.9)	54 (64)	30 (36)	CI^b 0.76–2.74
Authored by first scientific author	43 (27.6)	21 (49)	22 (51)	OR^a 0.52
Authored by first scientific author	43 (27.6)	21 (49)	22 (51)	CI^b 0.25–1.07
Published on institute or funder website	29 (18.6)	19 (66)	10 (34)	OR^a 1.32
Published on institute or funder website	29 (18.6)	19 (66)	10 (34)	CI^b 0.57–3.06
Total	156 (100)	94 (60)	62 (40)

Open in a new tab

^a Odds Ratio

^b Confidence Interval.

Consistencies and replicated reporting inadequacies across scientific publications

Following our analyses, the sample of scientific publications (N = 43) could be broken down into four unique groups in relation to the associated societal publications:

All corresponding societal publications were consistent and did not replicate any reporting inadequacies (n = 7 scientific publications associated with 45 societal publications).
Corresponding societal publications replicated reporting inadequacies, but were fully consistent with the scientific publication (n = 4 scientific publications associated with 17 societal publications).
Corresponding societal publications were inconsistent, but did not replicate reporting inadequacies (n = 15 scientific publications associated with 67 societal publications).
Corresponding societal publications were inconsistent with the scientific publication and replicated reporting inadequacies (n = 17 scientific publications associated with 27 societal publications).

Discussion

The aims of this study were to explore (1) whether societal publications on health services research are consistent with the messages communicated in the original scientific research paper, (2) whether apparent reporting inadequacies in scientific HSR publications are replicated in societal publications, and (3) whether fewer inconsistencies occur in societal publications if they are authored by the first author of the scientific work. 60.3% of the 156 societal publications (associated with 74.4% of the scientific publications) contained messages that were inconsistent with the scientific work. Reporting inadequacies in 51.2% (n = 22) of the scientific publications were replicated in corresponding societal publications (n = 45, 28.9%). The involvement of the first author was not associated with more consistent societal publications.

Our results indicate that, as previously shown for biomedicine, the field of HSR faces issues with (mis)representation and (mis)interpretation of the research findings, as reported in societal publications [10–14]. Such issues arise not only in news articles or press releases, but also in societal publications such as professional journal articles aimed directly at policy and practice.

Limitations

As our coding scheme was not specifically designed to identify causality, we have likely underestimated the occurrence of causal claims. The coding schemes used in previous studies, though very extensive, would not have been adequate for detecting many types of inconsistencies, such as rhetorical formulations of conclusions or diverging interpretations of results, as we have done in this study. In addition, the existing coding schemes would not have been suitable for HSR, as different types of systematic research were addressed here, including qualitative and mixed methods studies, and different types of societal publications were included in our analyses, such as tweets and fact sheets.

We analysed whether assertions in a societal publication were consistent with those in the corresponding scientific publication. We did not assess omitted messages; that is, we did not identify scientific reporting inadequacies attributable to the absence of common elements such as limitations, recommendations or contradictory evidence. Consequently, we also did not take a normative stand on whether those items should have been included in a societal publication. Such would not have been feasible considering the variety of societal publications studied, ranging from tweets to professional journals.

The numbers of associated societal publications were not equally distributed over the included scientific publications; one scientific author of multiple societal publications could have skewed our results. We therefore recommend further research on the roles of individual researchers in writing responsible societal publications.

Our sample of scientific publications was small and insufficiently wide-ranging to determine the prevalence of reporting inadequacies and inconsistencies across the field of HSR internationally.

Interpretation

Our results indicate that most societal publications contain inconsistencies or replicated reporting inadequacies. Inconsistencies are not necessarily negative, as they may correct an inadequacy in the scientific publication. Moreover, reporting inadequacies we identified in this study were not necessarily ‘bad’. There is no straightforward rule for what is allowed in terms of rhetorical wordings or simplifications of scientific results in either scientific or societal publications. However, the current discussion on public reporting is focused too narrowly on exaggeration and causality [10, 11]. There is little debate on questions such as whether conclusions and recommendations are adequately reported in scientific literature, the extent to which messages in societal publications may justifiably be simplified, how much detail needs to be provided, and whether a researcher or journalist may add interpretations in societal publications that would not be accepted in scientific literature.

Reporting inadequacies in scientific publications are commonly replicated in societal publications. Most frequently this involves inadequately reported conclusions, policy and practice recommendations, and titles. It is therefore insufficient to focus merely on preventing inconsistencies in societal publications. We recommend that future studies that assess quality in societal publications should extend their research questions to analyse this interplay between the reporting in scientific publications and societal publications.

No substantial differences emerged overall between societal publications produced by research institutes or funders and ones written by outsiders. Moreover, none of the included societal publications were written by other secondary authors of a scientific publication. A stronger relation between the involvement of researchers in writing societal publications and consistency with their scientific publications may be desirable. Media pressures, relationships with funders, and journal demands may cause researchers to consciously or unconsciously introduce reporting inadequacies into a scientific publication [29–31].

Implications and recommendations for policy and practice

The current COVID-19 pandemic shows the impact of disinformation and misinformation (e.g. on trust in government and their measures). Researchers, research institutes and journalists should be attentive to the effects that the rewriting of research results and conclusions in societal publications might have on policy and practice. Additionally, researchers should be aware that reporting inadequacies in their scientific publications may get replicated in societal publications and subsequently affect policy and practice. Routines such as peer feedback in the final stages of publication could prevent such reporting inadequacies from occurring in scientific publications. Such peer feedback is equally relevant to apply to societal publications. Further training and time dedicated to societal reporting and to communicating about scientific work in lay language would better equip researchers to take active roles in the writing of societal publications.

Conclusion

To improve societal publications on health services research, we should examine not only how consistency with scientific research publications can be achieved, but also how to prevent scientific reporting inadequacies from being replicated in societal publications. HSR institutions, funders, and scientific and societal publication platforms should invest in a supportive publication culture in order to further incentivise the responsible and skilled involvement of researchers in writing both scientific and societal publications.

Supporting information

S1 Appendix

(DOCX)

Click here for additional data file.^{(15.2KB, docx)}

S2 Appendix

(DOCX)

Click here for additional data file.^{(15.8KB, docx)}

Acknowledgments

We thank the Dutch HSR institutions that participated in this study: Erasmus MC, Department of Public Health; Erasmus University, Erasmus School of Health Policy and Management; Leiden University Medical Centre, Departments of Medical Decision Making and Public Health and Primary Care; Maastricht University, Health Services Research; Netherlands Institute for Health Services Research (NIVEL); Radboud UMC, IQ Healthcare; National Institute for Public Health and the Environment (RIVM); University of Groningen, Faculty of Economics and Business; Tilburg University, Social and Behavioural Sciences, Tranzo; Trimbos Institute; University Medical Centre Utrecht, Julius Centre for Health Sciences and Primary Care; and Amsterdam UMC, locations VU Medical Centre and Academic Medical Centre.

Data Availability

All relevant data are available within the project files in the Figshare public repository: https://doi.org/10.21942/uva.9255335.

Funding Statement

This study was funded by grant number 445001003 from the Netherlands Organisation for Health Research and Development (ZonMw). The funder had no role in the study design, the collection, analysis and interpretation of data, the writing of the manuscript or the decision to submit the paper for publication. All authors had full access to the data during the conduct of the study and they take responsibility for the integrity of the data and the analysis.

References

1.Weingart P. Science and the media. Res Policy. 1998;27(8):869–79. [Google Scholar]
2.Ritter A. How do drug policy makers access research evidence? Int J Drug Policy. 2009;20(1):70–5. 10.1016/j.drugpo.2007.11.017 [DOI] [PubMed] [Google Scholar]
3.Zardo P, Collie A. Type, frequency and purpose of information used to inform public health policy and program decision-making. Bmc Public Health. 2015;15. 10.1186/s12889-015-1581-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
4.Gerrits RG, van den Berg MJ, Klazinga NS, Kringos DS. Statistics in Dutch policy debates on health and healthcare. Health Res Policy Syst. 2019;17(1):55. 10.1186/s12961-019-0461-y [DOI] [PMC free article] [PubMed] [Google Scholar]
5.Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, et al. Examining the role of health services research in public policymaking. Milbank Q. 2002;80(1):125–54. 10.1111/1468-0009.00005 [DOI] [PMC free article] [PubMed] [Google Scholar]
6.Kuruvilla S, Mays N, Walt G. Describing the impact of health services and policy research. J Health Serv Res Policy. 2007;12 Suppl 1:S1-23–31. 10.1258/135581907780318374 [DOI] [PubMed] [Google Scholar]
7.Bornmann L. Measuring the societal impact of research: research is less and less assessed on scientific impact alone—we should aim to quantify the increasingly important contributions of science to society. EMBO reports. 2012;13(8):673–6. 10.1038/embor.2012.99 [DOI] [PMC free article] [PubMed] [Google Scholar]
8.Greenhalgh T, Russell J. Evidence-based policymaking: a critique. Perspect Biol Med. 2009;52(2):304–18. 10.1353/pbm.0.0085 [DOI] [PubMed] [Google Scholar]
9.Smith R. Measuring the social impact of research. BMJ. 2001;323(7312):528. 10.1136/bmj.323.7312.528 [DOI] [PMC free article] [PubMed] [Google Scholar]
10.Sumner P, Vivian-Griffiths S, Boivin J, Williams A, Bott L, Adams R, et al. Exaggerations and Caveats in Press Releases and Health-Related Science News. PLoS One. 2016;11(12):e0168217. 10.1371/journal.pone.0168217 [DOI] [PMC free article] [PubMed] [Google Scholar]
11.Schat J, Bossema FG, Numans ME, Smeets I, Burger P. Exaggerated health news. Nederlands Tijdschrift voor Geneeskunde. 2018;162(1). [PubMed] [Google Scholar]
12.Sumner P, Vivian-Griffiths S, Boivin J, Williams A, Venetis CA, Davies A, et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ: British Medical Journal. 2014;349. 10.1136/bmj.g7015 [DOI] [PMC free article] [PubMed] [Google Scholar]
13.Buhse S, Rahn AC, Bock M, Muhlhauser I. Causal interpretation of correlational studies—Analysis of medical news on the website of the official journal for German physicians. PLoS One. 2018;13(5):e0196833. 10.1371/journal.pone.0196833 [DOI] [PMC free article] [PubMed] [Google Scholar]
14.Haber N, Smith ER, Moscoe E, Andrews K, Audy R, Bell W, et al. Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review. PLoS ONE. 2018;13(5). 10.1371/journal.pone.0196346 [DOI] [PMC free article] [PubMed] [Google Scholar]
15.Alla K, Hall WD, Whiteford HA, Head BW, Meurk CS. How do we define the policy impact of public health research? A systematic review. Health Res Policy Sy. 2017;15(1):84–. 10.1186/s12961-017-0247-z [DOI] [PMC free article] [PubMed] [Google Scholar]
16.Dinsmore A, Allen L, Dolby K. Alternative Perspectives on Impact: The Potential of ALMs and Altmetrics to Inform Funders about Research Impact. PLOS Biology. 2014;12(11):e1002003. 10.1371/journal.pbio.1002003 [DOI] [PMC free article] [PubMed] [Google Scholar]
17.Council for Medical Sciences. The societal impact of applied health research: Towards a quality assessment system. Amsterdam: Royal Netherlands Academy of Arts and Sciences; 2002. [Google Scholar]
18.Canadian Institutes of Health Research. Developing a CIHR Framework to Measure The Impact of Health Research. Ottawa; 2005.
19.Waddell C, Lomas J, Lavis JN, Abelson J, Shepherd CA, Bird-Gayson T. Joining the conversation: newspaper journalists’ views on working with researchers. Healthcare policy = Politiques de sante. 2005;1(1):123–39. [PMC free article] [PubMed] [Google Scholar]
20.Chapman S, Haynes A, Derrick G, Sturk H, Hall WD, St George A. Reaching "an audience that you would never dream of speaking to": influential public health researchers’ views on the role of news media in influencing policy and public understanding. J Health Commun. 2014;19(2):260–73. 10.1080/10810730.2013.811327 [DOI] [PubMed] [Google Scholar]
21.Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005;365. 10.1016/S0140-6736(05)71879-1 [DOI] [PubMed] [Google Scholar]
22.Dechartres A, Trinquart L, Atal I, Moher D, Dickersin K, Boutron I, et al. Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ. 2017;357:j2490. 10.1136/bmj.j2490 [DOI] [PubMed] [Google Scholar]
23.Gerrits RG, Jansen T, Mulyanto J, van den Berg MJ, Klazinga NS, Kringos DS. Occurrence and nature of questionable research practices in the reporting of messages and conclusions in international scientific Health Services Research publications: a structured assessment of publications authored by researchers in the Netherlands. BMJ Open. 2019;9(5):e027903. 10.1136/bmjopen-2018-027903 [DOI] [PMC free article] [PubMed] [Google Scholar]
24.McVay AB, Stamatakis KA, Jacobs JA, Tabak RG, Brownson RC. The role of researchers in disseminating evidence to public health practice settings: a cross-sectional study. Health Res Policy Syst. 2016;14(1):42. 10.1186/s12961-016-0113-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
25.Juttmann RE. Het domein van gezondheidszorgonderzoek. In: Plochg T, Juttmann RE, Klazinga NS, Mackenbach JP, editors. Handboek gezondheidszorgonderzoek. Houten: Bohn Stafleu van Loghum; 2007. p. 22–3. [Google Scholar]
26.Lohr KN, Steinwachs DM. Health services research: an evolving definition of the field. Health Serv Res. 2002;37(1):7–9. [PubMed] [Google Scholar]
27.Hsieh H-F, Shannon SE. Three Approaches to Qualitative Content Analysis. Qualitative Health Research. 2005;15(9):1277–88. 10.1177/1049732305276687 [DOI] [PubMed] [Google Scholar]
28.Gerrits RG, Kringos DS, van den Berg MJ, Klazinga NS. Improving interpretation of publically reported statistics on health and healthcare: the Figure Interpretation Assessment Tool (FIAT-Health). Health Res Policy Syst. 2018;16(1):20. 10.1186/s12961-018-0279-z [DOI] [PMC free article] [PubMed] [Google Scholar]
29.Ioannidis JPA, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. The Lancet. 2014;383(9912):166–75. [DOI] [PMC free article] [PubMed] [Google Scholar]
30.Tijdink JK, Schipper K, Bouter LM, Maclaine Pont P, de Jonge J, Smulders YM. How do scientists perceive the current publication culture? A qualitative focus group interview study among Dutch biomedical researchers. BMJ Open. 2016;6(2):e008681. 10.1136/bmjopen-2015-008681 [DOI] [PMC free article] [PubMed] [Google Scholar]
31.Bouter LM. Commentary: Perverse Incentives or Rotten Apples? Accountability in Research. 2015;22(3):148–61. 10.1080/08989621.2014.950253 [DOI] [PubMed] [Google Scholar]

PLoS One. doi: 10.1371/journal.pone.0248753.r001

Decision Letter 0

Quinn Grundy

4 Sep 2020

PONE-D-20-23277

Reporting Health Services Research to a broader public: An exploration of inconsistencies and reporting inadequacies in societal publications

PLOS ONE

Dear Dr. Kringos,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

==============================

The reviewers noted the importance and novelty of the topic. However, to further assess the manuscript, they requested much more comprehensive reporting of the Methods in place of referencing a past publication. Particularly, they perceived major inconsistencies between the purported study design (a qualitative content analysis) and the analysis strategy and presentation of results (which largely included statistics). You will note that Reviewers 1 and 3 offer conflicting suggestions as to how to remedy the incongruence between the design, methods, analysis, and reporting, however, they were in agreement that the incongruence needs addressing. The authorship team thus has some important methodological and analytic choices to make and I look forward to receiving a revised version of the manuscript.

==============================

Please submit your revised manuscript by Oct 19 2020 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

We look forward to receiving your revised manuscript.

Kind regards,

Quinn Grundy, PhD, RN

Academic Editor

PLOS ONE

Journal Requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

Reviewer #3: Partly

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

Reviewer #3: No

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: Yes

Reviewer #3: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: The authors sought to investigate possible inconsistencies and reporting inadequacies in non-scientific societal publications based on published health services research. This is an important topic of increasing interest in both academic, healthcare and public spheres. The rationale is well-written and clear. I have a number of questions and concerns, that if addressed will strengthen the paper.

Major

1. There is confusion as to the design of this study. It is described as a qualitative study, yet a number of statistical results are reported, which is particularly prominent in the abstract. The statistical results are of little use, especially given questions as to the inclusion criteria related to the studies. I suggest deleting the statistical analysis altogether, as well as the emphasis on the number of publications that authors deemed as engaging in specific inaccuracies. The number of publications with each type of identified inaccuracy are not the purpose of the study. I would recommend authors focus on the types of inaccuracies found and describing those. The descriptions are really limited.

2. The description of a result as a “statistical trend” is problematic and should be deleted (lines 318-319). Wood et al. Trap of trends to statistical significance: likelihood of near significant P value becoming more significant with extra data. BMJ 2014; 348 doi: https://doi.org/10.1136/bmj.g2215

3. More information on the inclusion criteria of scientific publications is required for this paper to stand alone. Are qualitative and quantitative studies included? Why were 46 publications included (23 for each group of reporting adequacies)? Later this is unclear on pg 10, lines 203-205. A flow diagram may be helpful here. Later in the discussion, authors mention that qualitative and quantitative studies were included.

4. It’s not clear if societal publications were limited to Dutch or English, or both? Same for scientific publications. Please clarify. If there were different languages used in societal vs. scientific publications, this needs to be explained and examined in greater detail.

5. It is not clear why societal publications that only included results were excluded?

6. It is unclear why the dataset cannot be shared if all documents are publicly available.

7. I suggest providing quotes to provide examples of types of inconsistencies in reporting.

8. In the discussion authors could provide a more fulsome discussion of the limitations present in dissemination by societal publications. There is pressure from media to utilize sensational headlines, to focus on recommendations, and to communicate findings with extremely limited text space, etc.

Reviewer #2: This is an interesting paper, which addresses an important topic. The paper is clear and well written. I have highlighted a few areas that requires further consideration/clarification.

Authors have focused on the role of first scientific author in inconsistencies appearing in societal publications. However, I wonder if they have also looked at the involvement of other authors. Any of the authors of the scientific publications (rather than first author) may author some societal publications. As it is, it seems some of these may have been classified as “no involvement of scientific author”. Please clarify. If there are any possibilities of misclassification, it should be discussed in the limitations.

The authors included societal publications up to two years before the scientific publication. Some of the earlier societal publications may be based on interim results. Potential implications of this should also be discussed in the discussion, especially if significant proportion of the societal publications were published about two years before the scientific publication. In addition, Table 1 should show the distribution of the societal publications based on time between the scientific publication and the societal publication. For example, the table can show how many societal publications were published two years before the scientific publication, one year before scientific publication, same year as scientific publication and then one year after the scientific publication.

Some projects may involve various work packages, which may be reported in different scientific publications? Therefore, a societal publication may report a different but related work package to the scientific publication. It is not clear if the authors considered such issues. This may account for some of the inconsistencies, such as new results being introduced that were not reported in the corresponding scientific publication. Please clarify.

Reporting inadequacies were categorised into high vs low but it may be good to also have a further insight to the distribution of inadequacies. For example, a range (of the number of inconsistencies reported for the 43 publications) would be useful. Are there any scientific publications with no reporting inadequacies in any of the associating societal publications? What was the maximum number of inconsistencies identified for one publication?

I would also suggest that the authors report the distribution of reporting inadequacies in a table/figure showing the 35 possible inadequacies and the corresponding number of publications. This is similar to what was presented in Figure 2 of their previous publication (reference 21) but should be based on the sample (43 scientific publications) used in this study. Such aggregated data of the distribution of reporting inadequacies would not compromise anonymity. I believe it would be useful for the readers.

Are the authors able to clarify how the sample size of 23 in each group was determined?

In some subheadings (such as line 224, 236, 246 etc), it is not clear why the figures (such as, “64 societal publications, 41.0%”) were presented in the sub-heading rather than the main body of the sections.

Reviewer #3: This manuscript aims to compare articles and their respective media reports, whenever available. Below see my comments. I suggest a major throughout the points I have stressed. I congratulate the authors for addressing this relevant topic and please see my comments as a way to improve your research.

-----

33 “such translated informations”

33 When you introduce you will investigate inconsistencies, I was expecting X between Y. Thus, please re-organize in a way you may allow readers to understand who is the comparator immediately.

37 Please describe in the methods how you did the content analysis and translated it to numerical variables for the chi-squared test. This journal does not limit the word count in the abstract so please be as detailed as you can be.

37 How much of the 43 and the 156 pieces, respectively, did you extract variables? How much missing variables did occur? How many pieces were unable to be extracted in total?

37 The methods seem completely incomplete in the abstract.

45 About the results – and the same is true for the full text: I feel completely lost because of the absence of proportions. Chi-square statistics won’t tell the readers the full info neither the p-values. Please insert counts and proportions with confidence intervals. Suggest: a table in the full text and in the wrote text, only counts/proportions with confidence intervals.

49 Would you have a stronger suggestion for them? How about peer reviewing of societal publications? Do you think illiteracy of readers is the confounder, rather than discrepancies of scientific and societal publications?

60 Please remove “scientific research”

65 Do you really think that policymakers use cross-media to support decisions?

67 This is positive, not negative.

87 Messages and conclusions in scientific publications are too much poor reported. I suggest stressing the phrase and include references (plenty of David Moher, An Wen Chan etc).

103 I suggest collating aims of study in the last paragraph of the Introduction

120 As a spin-off of a paper, I suggest expanding the methods. This habit of to cite “the methods are described elsewhere” is inadequate nowadays and if you insist on this, you are being inconsistent with your own study, that is about reporting.

126 Again

131 From the beginning of the paragraph: it seems to me the sample was chose by a counting until a sufficient number for analysis was achieved. Is that correct?

134 What you considered as inadequacies?

155 How did you ensure societal publications and scientific pieces were matching?

173 Could you provide to the readers the coding?

193 I am really really interest in seeing the risk ratio of the association you did infer. The chi-square test can provide it (and odds ratios, or any ratio in which a contingence can permit) if you set the proper coding for it. In your study. You only showed to the readers if is there association or not, but not the magnitude of it. Thus, here is my recommendation (PS: do not forget of counts and confidence intervals also once relative measures could spin results).

207 Methods are not important to persuade people? Do you, for example, you could be persuaded only by the methods of a given research in a newspaper? I do not agree with such exclusions.

226 How did you consider “partially “?

236 This item alerts me. What is the probability of selective outcome reporting of one of the parts? An Wen Chan has an extensive work about this and Evan Mayo-Wilson about the consistencies of results. Even though I am comparing apples and oranges, maybe an exploratory investigation of such original studies and their registries could be welcome, or an association of publication bias, source of funding and industry, which could reflect in the societal publication. It is even more clear to investigate when you cite that only one negative result (called by non-significant) was found. It goes towards to the literature that deals with scientific pieces.

246 Recommendations by societal pieces are completely useless for me. Thank you for the data.

254 Societal pieces with poorly reported information and spin is also useless for me. Thank you for the data againg.

267 Causality? That’s terrific. Even well conducted RCTs don’t dare to accuse causality. Thank you for the data.

272 Generalizations? The same.

278 A result of what I meant above.

282 Replication of inadequacy: I congratulate you by addressing this very important topic. If you can just clarify for me a point I didn’t catch, I would be more than welcome. You found that barely 50% of your pieces matched in inadequacy by a lot of potential reasons. However, in the description below, the numbers do not reflect the same for a case of interpretation. I know this might be a source of data collection (methods) rather than internal validity – I needed to deal in data set of a study of mine about this situation. Is that the case? If yes, I strongly recommend you inform authors with a glossary of variables and how they were collected. If not, any potential reason for the discrepancies between the data?

307 Societal publications do not necessarily need to involve first authors or any author once the publication is online. However, in my point of view, if the first author (or any author) could be included and work together journalists and persons of communication, the literacy of the general population would improve exponencially by a combination of expertises. So, nice to see you X2. I won’t say you have a negative result – maybe there is an erratic interpretation (in advance, please remove the term trend in all citations) of what a p-value is. A p-value basically tells you what the probability of the data of given distribution is is embedded in the other distribution, and you set the tolerable limit for your inference in terms of dispersion. Do you really think that 2.0% of an increase of a bit of noise in your distribution would blunt your null hypothesis test? Please re-write it accordingly to the American Statistical Association. The same is for any other inferential test in your study. Finally, about stats, don’t claim for efficacy/association based on the p-value. Please interpret the X2 statistics (or the relative risk or any association measure I recommended before).

378 Your results have more than “some” (what is not bad). Just see the numbers. Please re-phrase it accordingly.

407 You can’t say it affects interpretation of general population. You didn’t measure it. As a legacy of COVID-19, I think one of the most important things we as scientists need to catch is how large is the noise in the literature and the attempt to publish whatever you have in hands. Given this, combined with the illiteracy of the population about evidence-based decisions, science, treatments, etc – which will be another legacy of COVID-19, I definitely would conclude this paper you are writing based in what is happening with evidence and media (no politics, please). This could be an important, reasonable recommendation.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

Reviewer #3: Yes: Lucas Helal

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

PLoS One. 2021 Apr 7;16(4):e0248753. doi: 10.1371/journal.pone.0248753.r002

Author response to Decision Letter 0

4 Dec 2020

Revisions “Reporting Health Services Research to a broader public"

Dear editor and reviewers,

Thank you for the opportunity to revise our manuscript “Reporting Health Services Research to a broader public”

We thank the reviewers for their valuable feedback. We describe how we have addressed the comments below.

Sincerely,

R.G. Gerrits

M.J. van den Berg

A.E. Kunst

N.S. Klazinga

D.S. Kringos

Major

Reply: Thank you for your comments regarding the design of the study. We understand we have created some confusion with regard to the qualitative and quantitative aims of the study. Our research questions are best answered through the combination of initial qualitative assessment followed by a quantitative description, as is not uncommon for directed content analyses (e.g. Hsieh H-F, Shannon SE.(200): doi:10.1177/1049732305276687). The numerical insights are important to answer in particular research question three, and further provide insight in what consistencies and reporting inadequacies are replicated, and which seem to occur the most. We have adapted our description and wording in the methods section to better characterize the mixed qualitative/quantitative nature of the study whilst emphasizing the description of types and relative importance of inaccuracies found.

Reply: We have deleted the mention of statistical trends in our paper.

Reply: Both quantitative and qualitative studies are included. We purposely included 46 publications to gain an even distribution of QRPs in the original scientific publications. We aimed to include publications with the most and the least QRPs,so we might detect differences in the replication reporting inadequacies. We have clarified this by adding this description under the heading subsample of scientific publications. Moreover, We have added a flow diagram to show the selection process as appendix A.

Reply: Scientific publications were only published in English language journals. Only Dutch language publications were included as societal publications, aside from twitter where the (English language) title was used as a message. We have clarified this in the manuscript under the heading: Sample of societal publications

5. It is not clear why societal publications that only included results were excluded?

Reply: Thank you for noticing this mistake in wording. We intended to communicate that no societal publications that were solely describing methodology without reference to the results was included. This was adjusted in the manuscript.

6. It is unclear why the dataset cannot be shared if all documents are publicly available.

Reply: To protect the privacy of Dutch authors/researchers whos work was included in the study we complied with existing privacy rules. Although underlying documents are publicly available, making the data set public would unnessesary expose individuals and undermine the trust with which this study was executed in the Dutch HSR community. This procedure was agreed upon with all partner institutes at the start of the study and formalized in a Data Management procedure, and formed an important basis for transparency and trust to allow for collaboration between (often competing) institutes and authors in this study.

7. I suggest providing quotes to provide examples of types of inconsistencies in reporting.

Reply: Thank you for this suggestion. Due to protection of the authors, we use no direct quotes as this will make the publication traceable. We have however, added some elaboration where we considered it helpful to further illustrate the types.

Reply: There is not a strong evidence-base of such pressure executed by media. We can only address the clearly documented expectation by funders and academia that scientists report to non-scientific audiences in lay-terms their results.

Reviewer #2: This is an interesting paper, which addresses an important topic. The paper is clear and well written. I have highlighted a few areas that requires further consideration/clarification.

Reply: The category “no involvement of the scientific author” includes publications where none of the authors were involved. This means that in none of the assessed scientific publications secondary authors were named as writers. We clarified this in the manuscript.

Reply: We noted how many publications were published before and after the scientific publication. Only a small sample was published before (19.2%). We included these numbers in the manuscript. Because only 20 percent was published in advance, we therefore don’t consider this will significantly impact the findings.

Reply: Separate results from related studies and publications were taken into account. We did not mark those results as inconsistent, but excluded them from our analyses. We solely included results that could only have resulted from the same study that was published. These results were presented in the societal publication to be part of the particular published scientific study.

Reporting inadequacies were categorized into high vs low but it may be good to also have a further insight to the distribution of inadequacies. For example, a range (of the number of inconsistencies reported for the 43 publications) would be useful. Are there any scientific publications with no reporting inadequacies in any of the associating societal publications? What was the maximum number of inconsistencies identified for one publication?

Reply: Scientific publications defined as low had 0-5 reporting inadequacies identified. Those ranked as high had 7-18 reporting inadequacies. The maximum number of inconsistencies for one publication gives no particular information. This number is dependent on the number and type of societal publications per scientific publication, which varies widely across the sample. Therefore, no comparison can be made.

Reply: The scatterplot in our previous publication has the primary function of showing the co-occurrence of QRPs in publications. This was possible because scientific publication is standardized and all publications had the same chance of including the same QRPs. This is not the case for the current study.

Co-occurrence of replicated reporting inadequacies is dependent on the scientific publication having this QRP. However, not all 43 scientific publications contain the same QRPs. Moreover, societal publications vary widely in content and description. In a short societal publication, the chance for replicating multiple reporting inadequacies is low. Any direct comparison between studies would therefore provide no insight in co-occurrence. Because we believe such a graph may easily be misinterpreted, we decided not to include it in the study.

Are the authors able to clarify how the sample size of 23 in each group was determined?

Reply: We purposely included 46 publications to gain an even distribution of reporting inadequacies in the original scientific publications. We aimed to include publications the most and the least reporting inadequacies, to allow for detection of differences in the replication reporting inadequacies. This decision was made so we might see the difference in societal publications associated with publications with a high number of inadequacies (>6) and a low number of inadequacies (<6). This has been clarified in the methods section.

Reply: We showed the main result besides category heading (e.g. Inconsistencies in reporting generalisations (3 societal publications, 1.7%) to promote reading efficiency. It will save readers from looking for the full count in the text below. It also immediately shows that it concerns the overall count, rather than the subcounts presented in the text.

-----

33 “such translated informations”

Reply: We adapted this sentence.

33 When you introduce you will investigate inconsistencies, I was expecting X between Y. Thus, please re-organize in a way you may allow readers to understand who is the comparator immediately.

Reply: Thank you for this suggestion. We have clarified this.

Reply: We added a more detailed description of how we translated the content analyses to the methods section in the abstract.

37 How much of the 43 and the 156 pieces, respectively, did you extract variables? How much missing variables did occur? How many pieces were unable to be extracted in total?

Reply: As the results were derived from a qualitative coding, no missing values occurred. Because of this we think it not necessary to mention missing data in the manuscript. We added more descriptive information regarding the included publications in our manuscript.

37 The methods seem completely incomplete in the abstract.

Reply: We added more information to the methods section in the abstract to allow for a better understanding of our approach.

Reply: We have adjusted the table including proportions and confidence intervals. As the current way of reporting follows APA standards, we have decided to maintain the description to avoid confusion.

Reply: We have added to the discussion section that peer feedback in the final stages of publication, is equally relevant to be applied to societal publications.,

60 Please remove “scientific research”

Reply: This was adjusted accordingly.

65 Do you really think that policymakers use cross-media to support decisions?

Reply: Yes. Policy makers use summaries of scientific studies and other published advice to support their decisions. Supporting evidence used in parliament and government often refers to newspaper articles, reports and summaries (as was investigated in: Gerrits, R.G., van den Berg, M.J., Klazinga, N.S. et al. Health Res Policy Sys 17, 55 (2019). https://doi.org/10.1186/s12961-019-0461-y) . But hardly ever original academic research publications. In a recent (perhaps anecdotal) example, in the Netherlands the effect of using masks against corona is summarized and communicated through societal publications (occasionally in the form of direct advice). Citing these societal publications, the Dutch government has decided to not make wearing masks mandatory. Abroad however, based on the same evidence, different conclusions are communicated in societal publications, and consequentially, different measures are taken.

67 This is positive, not negative.

Reply: We did not intent to make this sound negative, but describe a trend neutrally. As the parentheses might have given the impression it was a negative interpretation, we removed them.

87 Messages and conclusions in scientific publications are too much poor reported. I suggest stressing the phrase and include references (plenty of David Moher, An Wen Chan etc).

Reply: We have stressed the phrase and added references.

103 I suggest collating aims of study in the last paragraph of the Introduction

Reply: We included a detailed description of the aims to promote readability. The aims differ in a subtle manner, which makes it difficult to collate them without creating confusion.

Reply: Thank you for this suggestion. We have expanded our description of the assessment of the reporting inadequacies and added a list of inadequacies to the appendix.

126 Again

Reply: We have described the methods in more detail.

131 From the beginning of the paragraph: it seems to me the sample was chose by a counting until a sufficient number for analysis was achieved. Is that correct?

Reply: That is correct. We have clarified this.

134 What you considered as inadequacies?

Reply: We have added the list of reporting inadequacies to Appendix B. This will provide a complete description to the readers. A sentence to explain this was added.

155 How did you ensure societal publications and scientific pieces were matching?

Reply: This was done based on content. Either the authors, scientific publication, research institute and or research programmed were mentioned. Further, the descriptions and the methods of the study show whether the societal and scientific publication described the same study. Any societal publication that seemed to refer to another similar or related study was not included. However, we did not identify any societal publications where the relation was unclear. A sentence to explain this was added under the heading of Sample of societal publications.

173 Could you provide to the readers the coding?

Reply: The coding scheme has been added as Appendix C.

Reply: We added the Odds Ratio to table 2, which indeed is easier to interpret for the reader.

207 Methods are not important to persuade people? Do you, for example, you could be persuaded only by the methods of a given research in a newspaper? I do not agree with such exclusions.

Reply: While methodology is an important tool to aid persuasion, our study was not focused on the description of methodology. The excluded societal publications contained only a mention that the study was being performed and a description through which methods. This was not within the scope of this study.

226 How did you consider “partially “?

Reply: In agreement with the advice provided by reviewer 1, we added further explanation to illustrate this.

Reply: we will take your suggestion into account for further research to explore this topic more in depth, which goes beyond the current scope of this explorative work for the field of health services research.

Reply: The numbers seem to differ because these are respectively the number of identified inadequacies and the number of scientific publications. As multiple inadequacies could be identified in a scientific publication, these figures are slightly different.

Reply: We agree with you.

So, nice to see you X2. I won’t say you have a negative result – maybe there is an erratic interpretation (in advance, please remove the term trend in all citations) of what a p-value is. A p-value basically tells you what the probability of the data of given distribution is is embedded in the other distribution, and you set the tolerable limit for your inference in terms of dispersion.

Reply: We have removed this term.

Do you really think that 2.0% of an increase of a bit of noise in your distribution would blunt your null hypothesis test? Please re-write it accordingly to the American Statistical Association. The same is for any other inferential test in your study. Finally, about stats, don’t claim for efficacy/association based on the p-value. Please interpret the X2 statistics (or the relative risk or any association measure I recommended before).

Reply: We replaced the chi-square with odds ratios, which makes the associations easier to interpret. Indeed, we don’t draw bold conclusions based on this analyses and associations, which are relatively weak.

378 Your results have more than “some” (what is not bad). Just see the numbers. Please re-phrase it accordingly.

Reply: We removed the word ‘some’.

Reply: We contextualized the paper in the current context in the discussion by mentioning that the current COVID-19 pandemic has showed the impact of disinformation and misinformation.

PLoS One. doi: 10.1371/journal.pone.0248753.r003

Decision Letter 1

Quinn Grundy

18 Jan 2021

PONE-D-20-23277R1

Reporting Health Services Research to a broader public: An exploration of inconsistencies and reporting inadequacies in societal publications

PLOS ONE

Dear Dr. Kringos,

==============================

The reviewers again had the opportunity review the manuscript and found the reporting to be greatly strengthened. The introduction and the manuscript are reading really well and are much improved. However, there are a few reporting considerations, a couple points of clarification and revision of the abstract that need to happen before we can reach a final decision. I ask that you address the following minor comments:

Please note in your limitations section that your analysis including odds ratios does not adjust for confounders, so please note that this is a gross analysis.

Many of the changes you made in the main text to improve the clarity of reporting have not been reproduced in the Abstract. Please address the following in the abstract:

- define societal publications in the first sentence as “such as press releases, newspapers, social media, internet postings or professional journals.”

- describe as a content analysis in a way that reflects the Methods in the main text

- a couple of typo “inconsistencies” instead of “inconstancies” also typo with “all” at the end of the line line 42

- reproduce the language from the main text in describing the coding process

- Line 44- replace “After all documents were coded, counts per code were calculated,” with suggested wording, “Descriptive frequencies were calculated for all variables of interest.”

- please round percentages to one decimal place

- First two lines of the abstract results are confusing; please just state how many scientific and societal publications were included.

- Please also rephrase this sentence lines 54-55: “Reporting inadequacies in 51.2% of the scientific publications were replicated in associated societal publications (28.9%).” I don’t understand what the proportions refer to.

In the main text, please address the following:

- Introduction, line 100, typo “poorly” reported

- Please spell out “questionable research practices” throughout the manuscript as it is a non-standard acronym

- What is a “directed” qualitative content analysis? You provide a citation later in the paper, but if this is a particular method, please cite here as well. Also, please define what you mean by “directed.” I think simply stating that this study design is a “content analysis” will address some confusing – content analyses often have both qualitative and quantitative analyses. Please also reflect whatever changes you make in the abstract.

- Please make Appendix B a Figure to be included in the manuscript. This figure should also be referenced not in the Methods, but the first paragraph of the Results. I am still rather confused about the 43 initially included scientific HSR publications and then the 46 included scientific HSR publications and many of the numbers reported in the Methods section, but they are much clearer in the first paragraph of the Results. It will likely add clarity if you simply describe the search and screening strategy in the Methods in general terms, but leave the details about the specific numbers to this paragraph in the Results. Please also reflect any changes to the main text in the abstract.

- I would suggest round proportions to whole numbers throughout the manuscript.

- Wherever you report a proportion, please report also the accompanying (numerator/denominator). Similarly, where you report numerators (e.g. top of page 13, lines 259-272), report the corresponding denominators and proportions (or, simply present the findings qualitatively as I’m not sure these counts really add much). See, for examples, the section beginning line 357 “Role of the first scientific author."

- The sub-headings beginning page 12 (e.g. “Inconsistencies in conclusions) are helpful. However, I would suggest moving the quantification into a sentence in the paragraph following the sub-headings (e.g. 64/X societal publications, 41%).

- Line 332, when you saw that reporting inadequacies were “replicated” or “reproduced” in the societal publication, do you mean that it was copied verbatim? Or that it was also present in some form? The counts in this section might better reflect the scientific/societal publication pairs. Currently, you only describe the proportion of “scientific publications” that had an inadequacy, which is confusing given that you just presented these frequencies. Could you rephrase to describe the matched pairs as the unit of analysis?

==============================

Please submit your revised manuscript by Mar 04 2021 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org. When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.
A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.
An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

We look forward to receiving your revised manuscript.

Kind regards,

Quinn Grundy, PhD, RN

Academic Editor

PLOS ONE

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

Reviewer #3: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

Reviewer #2: Yes

Reviewer #3: Partly

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

Reviewer #3: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

Reviewer #2: Yes

Reviewer #3: No

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

Reviewer #2: Yes

Reviewer #3: Yes

**********

6. Review Comments to the Author

Reviewer #2: (No Response)

Reviewer #3: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: No

Reviewer #3: Yes: Lucas Helal

PLoS One. 2021 Apr 7;16(4):e0248753. doi: 10.1371/journal.pone.0248753.r004

Author response to Decision Letter 1

3 Mar 2021

Revisions “Reporting Health Services Research to a broader public"

Dear editor and reviewers,

Thank you for the opportunity to make further minor revisions to our manuscript “Reporting Health Services Research to a broader public”.

Below, we explain how we have addressed the comments point-by-point.

Sincerely,

R.G. Gerrits

M.J. van den Berg

A.E. Kunst

N.S. Klazinga

D.S. Kringos

Comments regarding Ethics:

Your ethics statement should only appear in the Methods section of your manuscript.

Reply: we have moved our ethics statement to the first paragraph of the methods section.

Comments regarding the abstract:

- Please note in your limitations section that your analysis including odds ratios does not adjust for confounders, so please note that this is a gross analysis.

- Many of the changes you made in the main text to improve the clarity of reporting have not been reproduced in the Abstract. Please address the following in the abstract:

- define societal publications in the first sentence as “such as press releases, newspapers, social media, internet postings or professional journals.”

Reply: we added these suggestions to the abstract, keeping in mind the limit of 300 words.

Comment: describe as a content analysis in a way that reflects the Methods in the main text

Reply: Given the allowed limit of 300 words, we have expanded the description of the methods section, but kept it focused on its essential elements.

Comment: a couple of typo “inconsistencies” instead of “inconstancies” also typo with “all” at the end of the line line 42

Reply: We have closely re-read the text for consistent use of the word “inconsistencies” and corrected typos.

Comment: reproduce the language from the main text in describing the coding process.

Reply: Given the allowed limit of 300 words, we have expanded the description of the methods section, but kept it focused on its essential elements, which does not allow room for such detailed steps.

Comment: Line 44- replace “After all documents were coded, counts per code were calculated,” with suggested wording, “Descriptive frequencies were calculated for all variables of interest.”

Reply: we adapted this.

Comment: please round percentages to one decimal place

Reply: currently all our percentages are rounded to one decimal place, however, percentages in the introduction are rounded as they were presented as such in the referenced literature.

Comment: First two lines of the abstract results are confusing; please just state how many scientific and societal publications were included.

Reply: These two sentences were replaced with a simpler description.

Comment: Please also rephrase this sentence lines 54-55: “Reporting inadequacies in 51.2% of the scientific publications were replicated in associated societal publications (28.9%).” I don’t understand what the proportions refer to.

Reply: We rephrased this sentence.

Comments: In the main text, please address the following:

- Introduction, line 100, typo “poorly” reported

- Please spell out “questionable research practices” throughout the manuscript as it is a non-standard acronym

Reply: We adapted the text in accordance with these suggestions and added a sentence explaining the core concept of a directed content analyses approach.

Comment: Please make Appendix B a Figure to be included in the manuscript. This figure should also be referenced not in the Methods, but the first paragraph of the Results.

Reply: The flow chart was entered as a figure to the results section.

I am still rather confused about the 43 initially included scientific HSR publications and then the 46 included scientific HSR publications and many of the numbers reported in the Methods section, but they are much clearer in the first paragraph of the Results. It will likely add clarity if you simply describe the search and screening strategy in the Methods in general terms, but leave the details about the specific numbers to this paragraph in the Results. Please also reflect any changes to the main text in the abstract.

Reply: All mentioning of numbers of societal publications was left out of the method section, limiting reporting of the selection results to the results section.

Comment: I would suggest round proportions to whole numbers throughout the manuscript.

Reply: In line with the earlier suggestion, we decided to maintain a decimal point reporting.

Comment: Wherever you report a proportion, please report also the accompanying

(numerator/denominator). Similarly, where you report numerators (e.g. top of page 13, lines 259-272), report the corresponding denominators and proportions (or, simply present the findings qualitatively as I’m not sure these counts really add much). See, for examples, the section beginning line 357 “Role of the first scientific author."

Reply: All percentages are now preceded by numerator/denominator.

Comment: The sub-headings beginning page 12 (e.g. “Inconsistencies in conclusions) are helpful. However, I would suggest moving the quantification into a sentence in the paragraph following the sub-headings (e.g. 64/X societal publications, 41%).

Reply: We moved the numbers into the suggested paragraph.

Comment: Line 332, when you saw that reporting inadequacies were “replicated” or “reproduced” in the societal publication, do you mean that it was copied verbatim?

Reply: As stated in the methods “A message that identically reproduced the reporting inadequacy was marked as a ‘replicated reporting inadequacy’. The scientific publication is in English and the societal publications were Dutch. Thus, an identical reproduction refers to a direct translation.

Comment: Or that it was also present in some form? The counts in this section might better reflect the scientific/societal publication pairs. Currently, you only describe the proportion of “scientific publications” that had an inadequacy, which is confusing given that you just presented these frequencies. Could you rephrase to describe the matched pairs as the unit of analysis?

Reply: For clarification we added the number of societal publications matching with the scientific publications.

PLoS One. doi: 10.1371/journal.pone.0248753.r005

Decision Letter 2

Quinn Grundy

5 Mar 2021

Reporting Health Services Research to a broader public: An exploration of inconsistencies and reporting inadequacies in societal publications

PONE-D-20-23277R2

Dear Dr. Kringos,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Quinn Grundy, PhD, RN

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

PLoS One. doi: 10.1371/journal.pone.0248753.r006

Acceptance letter

Quinn Grundy

16 Mar 2021

PONE-D-20-23277R2

Reporting Health Services Research to a broader public: An exploration of inconsistencies and reporting inadequacies in societal publications

Dear Dr. Kringos:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Quinn Grundy

Academic Editor

PLOS ONE

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1 Appendix

(DOCX)

Click here for additional data file.^{(15.2KB, docx)}

S2 Appendix

(DOCX)

Click here for additional data file.^{(15.8KB, docx)}

Data Availability Statement

All relevant data are available within the project files in the Figshare public repository: https://doi.org/10.21942/uva.9255335.

[pone.0248753.ref001] 1.Weingart P. Science and the media. Res Policy. 1998;27(8):869–79. [Google Scholar]

[pone.0248753.ref002] 2.Ritter A. How do drug policy makers access research evidence? Int J Drug Policy. 2009;20(1):70–5. 10.1016/j.drugpo.2007.11.017 [DOI] [PubMed] [Google Scholar]

[pone.0248753.ref003] 3.Zardo P, Collie A. Type, frequency and purpose of information used to inform public health policy and program decision-making. Bmc Public Health. 2015;15. 10.1186/s12889-015-1581-0 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref004] 4.Gerrits RG, van den Berg MJ, Klazinga NS, Kringos DS. Statistics in Dutch policy debates on health and healthcare. Health Res Policy Syst. 2019;17(1):55. 10.1186/s12961-019-0461-y [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref005] 5.Lavis JN, Ross SE, Hurley JE, Hohenadel JM, Stoddart GL, Woodward CA, et al. Examining the role of health services research in public policymaking. Milbank Q. 2002;80(1):125–54. 10.1111/1468-0009.00005 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref006] 6.Kuruvilla S, Mays N, Walt G. Describing the impact of health services and policy research. J Health Serv Res Policy. 2007;12 Suppl 1:S1-23–31. 10.1258/135581907780318374 [DOI] [PubMed] [Google Scholar]

[pone.0248753.ref007] 7.Bornmann L. Measuring the societal impact of research: research is less and less assessed on scientific impact alone—we should aim to quantify the increasingly important contributions of science to society. EMBO reports. 2012;13(8):673–6. 10.1038/embor.2012.99 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref008] 8.Greenhalgh T, Russell J. Evidence-based policymaking: a critique. Perspect Biol Med. 2009;52(2):304–18. 10.1353/pbm.0.0085 [DOI] [PubMed] [Google Scholar]

[pone.0248753.ref009] 9.Smith R. Measuring the social impact of research. BMJ. 2001;323(7312):528. 10.1136/bmj.323.7312.528 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref010] 10.Sumner P, Vivian-Griffiths S, Boivin J, Williams A, Bott L, Adams R, et al. Exaggerations and Caveats in Press Releases and Health-Related Science News. PLoS One. 2016;11(12):e0168217. 10.1371/journal.pone.0168217 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref011] 11.Schat J, Bossema FG, Numans ME, Smeets I, Burger P. Exaggerated health news. Nederlands Tijdschrift voor Geneeskunde. 2018;162(1). [PubMed] [Google Scholar]

[pone.0248753.ref012] 12.Sumner P, Vivian-Griffiths S, Boivin J, Williams A, Venetis CA, Davies A, et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ: British Medical Journal. 2014;349. 10.1136/bmj.g7015 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref013] 13.Buhse S, Rahn AC, Bock M, Muhlhauser I. Causal interpretation of correlational studies—Analysis of medical news on the website of the official journal for German physicians. PLoS One. 2018;13(5):e0196833. 10.1371/journal.pone.0196833 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref014] 14.Haber N, Smith ER, Moscoe E, Andrews K, Audy R, Bell W, et al. Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): A systematic review. PLoS ONE. 2018;13(5). 10.1371/journal.pone.0196346 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref015] 15.Alla K, Hall WD, Whiteford HA, Head BW, Meurk CS. How do we define the policy impact of public health research? A systematic review. Health Res Policy Sy. 2017;15(1):84–. 10.1186/s12961-017-0247-z [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref016] 16.Dinsmore A, Allen L, Dolby K. Alternative Perspectives on Impact: The Potential of ALMs and Altmetrics to Inform Funders about Research Impact. PLOS Biology. 2014;12(11):e1002003. 10.1371/journal.pbio.1002003 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref017] 17.Council for Medical Sciences. The societal impact of applied health research: Towards a quality assessment system. Amsterdam: Royal Netherlands Academy of Arts and Sciences; 2002. [Google Scholar]

[pone.0248753.ref018] 18.Canadian Institutes of Health Research. Developing a CIHR Framework to Measure The Impact of Health Research. Ottawa; 2005.

[pone.0248753.ref019] 19.Waddell C, Lomas J, Lavis JN, Abelson J, Shepherd CA, Bird-Gayson T. Joining the conversation: newspaper journalists’ views on working with researchers. Healthcare policy = Politiques de sante. 2005;1(1):123–39. [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref020] 20.Chapman S, Haynes A, Derrick G, Sturk H, Hall WD, St George A. Reaching "an audience that you would never dream of speaking to": influential public health researchers’ views on the role of news media in influencing policy and public understanding. J Health Commun. 2014;19(2):260–73. 10.1080/10810730.2013.811327 [DOI] [PubMed] [Google Scholar]

[pone.0248753.ref021] 21.Chan AW, Altman DG. Epidemiology and reporting of randomised trials published in PubMed journals. Lancet. 2005;365. 10.1016/S0140-6736(05)71879-1 [DOI] [PubMed] [Google Scholar]

[pone.0248753.ref022] 22.Dechartres A, Trinquart L, Atal I, Moher D, Dickersin K, Boutron I, et al. Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study. BMJ. 2017;357:j2490. 10.1136/bmj.j2490 [DOI] [PubMed] [Google Scholar]

[pone.0248753.ref023] 23.Gerrits RG, Jansen T, Mulyanto J, van den Berg MJ, Klazinga NS, Kringos DS. Occurrence and nature of questionable research practices in the reporting of messages and conclusions in international scientific Health Services Research publications: a structured assessment of publications authored by researchers in the Netherlands. BMJ Open. 2019;9(5):e027903. 10.1136/bmjopen-2018-027903 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref024] 24.McVay AB, Stamatakis KA, Jacobs JA, Tabak RG, Brownson RC. The role of researchers in disseminating evidence to public health practice settings: a cross-sectional study. Health Res Policy Syst. 2016;14(1):42. 10.1186/s12961-016-0113-4 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref025] 25.Juttmann RE. Het domein van gezondheidszorgonderzoek. In: Plochg T, Juttmann RE, Klazinga NS, Mackenbach JP, editors. Handboek gezondheidszorgonderzoek. Houten: Bohn Stafleu van Loghum; 2007. p. 22–3. [Google Scholar]

[pone.0248753.ref026] 26.Lohr KN, Steinwachs DM. Health services research: an evolving definition of the field. Health Serv Res. 2002;37(1):7–9. [PubMed] [Google Scholar]

[pone.0248753.ref027] 27.Hsieh H-F, Shannon SE. Three Approaches to Qualitative Content Analysis. Qualitative Health Research. 2005;15(9):1277–88. 10.1177/1049732305276687 [DOI] [PubMed] [Google Scholar]

[pone.0248753.ref028] 28.Gerrits RG, Kringos DS, van den Berg MJ, Klazinga NS. Improving interpretation of publically reported statistics on health and healthcare: the Figure Interpretation Assessment Tool (FIAT-Health). Health Res Policy Syst. 2018;16(1):20. 10.1186/s12961-018-0279-z [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref029] 29.Ioannidis JPA, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. The Lancet. 2014;383(9912):166–75. [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref030] 30.Tijdink JK, Schipper K, Bouter LM, Maclaine Pont P, de Jonge J, Smulders YM. How do scientists perceive the current publication culture? A qualitative focus group interview study among Dutch biomedical researchers. BMJ Open. 2016;6(2):e008681. 10.1136/bmjopen-2015-008681 [DOI] [PMC free article] [PubMed] [Google Scholar]

[pone.0248753.ref031] 31.Bouter LM. Commentary: Perverse Incentives or Rotten Apples? Accountability in Research. 2015;22(3):148–61. 10.1080/08989621.2014.950253 [DOI] [PubMed] [Google Scholar]

PERMALINK

Reporting health services research to a broader public: An exploration of inconsistencies and reporting inadequacies in societal publications

Reinie G Gerrits

Michael J van den Berg

Anton E Kunst

Niek S Klazinga

Dionne S Kringos

Roles

Abstract

Introduction

Methods

Results

Conclusion

Background

Methods

Subsample of scientific publications

Sample of societal publications

Search terms and filters

Analyses of messages in societal publications

Statistical analyses

Results

Characteristics of analysed publications

Fig 1. Flow chart inclusion and exclusion process.

Table 1. Characteristics of the analysed societal publications.

Consistency of reporting between societal and scientific publications

Inconsistencies in conclusions

Inconsistencies in results

Inconsistencies in recommendations

Inconsistencies in the reporting of conditions in the conclusion

Inconsistencies in the reporting of implications for policy and practice

Inconsistencies in the reporting of causality

Inconsistencies in reporting generalisations

Objectives not included in the scientific publication

Replication of reporting inadequacies from scientific to societal publications

The role of the first scientific author in inconsistencies appearing in societal publications

Table 2. The role of the first scientific author in the occurrence of inconsistencies between a societal and a scientific publication; odds ratio for each category (row) compared to both others (N = 156).

Consistencies and replicated reporting inadequacies across scientific publications

Discussion

Limitations

Interpretation

Implications and recommendations for policy and practice

Conclusion

Supporting information

Acknowledgments

Data Availability

Funding Statement

References

Decision Letter 0

Quinn Grundy

Roles

Author response to Decision Letter 0

Decision Letter 1

Quinn Grundy

Roles

Author response to Decision Letter 1

Decision Letter 2

Quinn Grundy

Roles

Acceptance letter

Quinn Grundy

Roles

Associated Data

Supplementary Materials

Data Availability Statement

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases