A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines

David Blanco; Darko Hren; Jamie J Kirkham; Erik Cobo; Sara Schroter

doi:10.12688/f1000research.20556.3

. 2019 Dec 23;8:1682. Originally published 2019 Sep 24. [Version 3] doi: 10.12688/f1000research.20556.3

A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines

David Blanco ^1,^2,^a, Darko Hren ³, Jamie J Kirkham ⁴, Erik Cobo ¹, Sara Schroter ⁵

PMCID: PMC6880264 PMID: 31824668

Version Changes

Revised. Amendments from Version 2

We are now reporting the results for the ease of implementation/effectiveness in terms of medians and quartiles instead of means and standard deviations. Regarding Figure 1, we have removed the means and we are now displaying the medians as blue horizontal lines. Moreover, we decided to remove the black points representing the individual scores as they do not allow to see the medians in the graphs. In this way, Figure 1 is clearer and cleaner. In the Discussion, we have added further justifications on whether we believed that sample size and response rates were adequate for the purpose of the survey. We have modified the Conclusions both in the abstract and at the end of the paper, linking them to the survey aims (perceptions, barriers, facilitators…). We have also moved to the end of the Discussion section what was reported in the old Conclusions.

Abstract

Background: Improving the completeness of reporting of biomedical research is essential for improving its usability. For this reason, hundreds of reporting guidelines have been created in the last few decades but adherence to these remains suboptimal. This survey aims to inform future evaluations of interventions to improve adherence to reporting guidelines. In particular, it gathers editors’ perceptions of a range of interventions at various stages in the editorial process.

Methods: We surveyed biomedical journal editors that were knowledgeable about this topic. The questionnaire included open and closed questions that explored (i) the current practice of their journals, (ii) their perceptions of the ease of implementation of different interventions and the potential effectiveness of these at improving adherence to reporting guidelines, (iii) the barriers and facilitators associated with these interventions, and (iv) suggestions for future interventions and incentives.

Results: Of the 99 editors invited, 24 (24%) completed the survey. Involving trained editors or administrative staff was deemed the potentially most effective intervention but, at the same time, it was considered moderately difficult to implement due to logistic and resource issues. Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and were concerned that the quality of peer review could be compromised. Reviewers are generally not expected to focus on reporting issues but on providing an expert view on the importance, novelty, and relevance of the manuscript. Journals incentivising adherence, and publishers and medical institutions encouraging journals to take action to boost adherence were two recurrent themes.

Conclusions: Biomedical journal editors generally believed that engaging trained professionals would be the most effective, yet resource intensive, editorial intervention. Also, they thought that peer reviewers should not be asked to check RGs. Future evaluations of interventions can take into account the barriers, facilitators, and incentives described in this survey.

Keywords: Completeness of reporting, Journal policies, Quality of reporting, Reporting guidelines, Survey, Barriers, Facilitators

Abbreviations

RGs: reporting guidelines; CONSORT: Consolidated Standards of Reporting Trials; RCT: Randomised controlled trials; EQUATOR: Enhancing the QUAlity and Transparency Of Health Research; MiRoR: Methods in Research on Research; STROBE: STrengthening the Reporting of OBservational studies in Epidemiology; PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses; APCs: article processing charges; CME: continuing medical education; ICJME: International Committee of Medical Journal Editors.

Introduction

Transparent and accurate reporting of research is essential for increasing the usability of available research evidence. Reporting guidelines (RGs) can be useful tools to help authors report research methods and findings in a way that they can be understood by readers, replicated by researchers, used by health care professionals to make clinical decisions, and included in systematic reviews ¹. Since the inception in 1996 of the Consolidated Standards of Reporting Trials (CONSORT) for the reporting of randomised controlled trials (RCTs) ², more than 400 RGs for different study types, data, and clinical areas have been developed. These RGs can be found in the library of the Enhancing the Quality and Transparency Of Health Research (EQUATOR) Network ³.

Biomedical authors’ adherence to RGs has been observed to be suboptimal ⁴. Consequently, in recent years various stakeholders have proposed, and sometimes evaluated, the impact of different types of interventions to improve this adherence. These interventions were identified and classified in a recently published scoping review ⁵. We found that the strategies most widely used by journals have been shown not to have the desired effect ^6–
9 and this highlighted the need for the implementation and evaluation of the other interventions proposed ⁵.

This paper reports a survey aimed to inform the future evaluation of interventions to improve adherence to RGs. In particular, we focused on interventions that can be implemented at various points in the editorial process. Our specific objectives were to explore the perceived ease of implementation of different interventions and the potential effectiveness of these at improving adherence to RGs; to map the barriers and facilitators associated with these interventions; to determine possible solutions to overcome the barriers described, and to identify further editorial interventions that could be implemented and subsequently evaluated.

Methods

Participants

Purposive sampling was used to recruit biomedical editors that were expected to be knowledgeable and experienced in the topic we aimed to explore. We recruited participants not based on their representativeness of all medical journals but on the fact that they were “information-rich cases” ¹⁰.

Participants were sampled from three sources: (i) editors of journals that had published studies describing interventions to improve adherence to RGs identified in our scoping review ⁵, (ii) members of the Methods in Research on Research (MiRoR) Network with current editorial positions and (iii) editors of the top-10 journals (based on impact factor) of BMJ Publishing Group which, apart from being one of the partner institutions of MiRoR, has published the main RGs ^2,
11–
13) and has traditionally performed research to improve the transparency and quality of biomedical publications ¹⁴. The authors of this survey who met the eligibility criteria were excluded as potential participants.

Recruitment

The survey was only open to editors that we invited to participate. We contacted three editors (including the editors-in-chief) of each of the sampled journals, as well as individual editors from the group (ii) above. By replying to our invitation email, participants could suggest further editors that they considered could contribute to the survey. To contact editors not known to us we sought email addresses in the public domain. The survey was not advertised on any website.

Survey administration

The survey was administered by SurveyMonkey ¹⁵ and was open between 27 November 2018 and 24 February 2019. Participants were sent a personalised email inviting them to complete an online survey investigating their opinions about different editorial interventions to improve author adherence to RGs. Each invitation was tied to a unique email address. Two reminders to complete the survey were sent to non-responders at four and eight weeks after the initial mailing.

Participants could edit their responses while completing the survey. However, they could not re-enter the survey once it was completed as no two entries from the same IP address were allowed. We did not offer any incentives for completing the survey.

Response rates

We recorded the view rate of the invitation email (subjects opening the invitation email/subjects invited), the response rate (subjects completing the survey/subjects invited), and the completion rate (subjects completing the survey/subjects completing the first question of the survey).

Questionnaire development

Our previous scoping review ⁵ identified 31 interventions targeting different stakeholders in the research process. For use in this survey we chose a smaller subset of nine interventions that could be implemented during the editorial process as our focus was on journal editors’ perceptions (see Box 1).

Box 1. Interventions included and their targets.

A.
Interventions targeting authors:
- •
  A requirement for authors to submit a completed RG checklist (using all appropriate extensions, if applicable) indicating the page numbers where each item is addressed ( Intervention 1)
- •
  A requirement for authors to submit a populated RG checklist with text from their manuscript in order to facilitate the peer review process ( Intervention 2)
- •
  A requirement for authors to highlight in the manuscript where each RG item is addressed ( Intervention 3)
- •
  A requirement for authors to include new subheadings within their manuscript corresponding to different RG items within the traditional IMRaD format (Introduction, Methods, Results, and Discussion) ( Intervention 4)
- •
  A requirement for authors on submission to use a freely available writing aid tool that guides authors through the RG checklist items, shows the key elements that need to be reported, and includes examples of adequate reporting (e.g. COBWEB) ( Intervention 5)
B.
Interventions targeting peer reviewers:
- •
  Instruct peer reviewers to use the appropriate RGs when assessing a manuscript ( Intervention 6)
- •
  Instruct peer reviewers to scrutinise the completed RG checklist submitted by the authors and check its consistency with the information reported in the manuscript ( Intervention 7)
C.
Interventions targeting editorial staff:
- •
  An evaluation of the completeness of reporting by a trained editor (or editorial assistant), who would return incomplete manuscripts to authors before considering the manuscript for publication ( Intervention 8)
D.
Interventions targeting authors, peer reviewers, and editors:
- •
  Training for authors, peer reviewers, and editors on the importance, content, and use of RGs (e.g. The EQUATOR Network toolkits) ( Intervention 9)

The survey combined open and closed response questions to seek participants’ perceptions of a series of interventions to improve authors’ adherence to RGs that could potentially be implemented during the editorial process. We pilot tested the draft survey questionnaire with two collaborators of the MiRoR project who currently hold editorial positions. They were asked to review the survey for its clarity and completeness and to provide suggestions on how to improve its structure.

Based on feedback from the pilot we decided not to include the intervention “Implementation of the automatic tool Statreviewer ¹⁶” since participants were not aware of this software and stated that their perceptions would strongly depend on details about how it operates which are not publicly available.

Part 1: Current practice. Participants were asked to describe the measures their journal currently takes to improve adherence to RGs.
Part 2: Perceptions of nine potential interventions. Participants were asked to indicate on 5-point Likert scales (i) how easy it would be (or was) to implement these interventions at their journals (1-very difficult, 2-moderately difficult, 3-neither difficult nor easy, 4-moderately easy, 5-very easy) and (ii) how effective they thought the interventions would be (or was) at improving adherence to RGs if these were implemented at their journals (1-very ineffective, 2-moderately ineffective, 3-neither ineffective nor effective, 4-moderately effective, 5-very effective). We included images to clarify meanings and context to prompt participants to think about the benefits and drawbacks of the interventions. Free text boxes were included so participants could justify their responses.
Part 3: Identifying the barriers and facilitators. Participants were asked to choose which intervention they considered potentially the most effective for their journal at improving adherence to RGs. They were asked to describe (i) why they thought that intervention would be the most effective, (ii) what the main difficulties in implementing that intervention would be, and (iii) how they would try to overcome these difficulties.
Part 4: Further interventions. Participants were asked for further suggestions of possible interventions, including modifications and combinations of the interventions previously discussed.
Part 5: Demographic questions.

The survey was distributed over 18 pages with 1 to 3 items per page. These items were not randomised.

Data analysis

For quantitative data (Part 2 of the questionnaire), we used R version 3.6.0 ¹⁸. As these data were ordinal, we calculated medians together and the 1st and 3rd quartiles. We excluded from the analysis one questionnaire where the participant just opened the survey and left without answering any question. We did not exclude any questionnaire based on the amount of time that the participant needed to complete it.

For qualitative information, the lead investigator (DB) used the software program NVivo 12 ¹⁹. We mapped the barriers and facilitators for each of the interventions explored, as well as other key themes such as the incentives for the use of RG and the implementation of further editorial strategies. The initial mapping made by the lead investigator was discussed with another investigator (SS) and subsequently refined.

For Part 1 of the survey ( Current practice) the unit of measure were the journals and therefore editors of the same journal were grouped. This was due to the fact that participants’ answers represented an overarching policy and not an individual’s opinion. For all other parts of the survey (Part 2 to Part 5), we analysed editors’ responses independently, no matter what their journal was.

Ethics approval & informed consent

The Research Committee of the Governing Council of the Universitat Politècnica de Catalunya (UPC) granted ethical approval for this study (Reference EC 01, Date 2 May 2018).

In the invitation email, we informed survey participants that (i) the completion of the survey indicated consent to participate, (ii) they were free to stop and withdraw from the study at any time without providing a reason, (iii) the estimated time to complete the survey was 15 minutes, (iv) any identifiable information obtained in connection with this survey would remain confidential, and (v) the results would be submitted for publication and the anonymised dataset would be made publicly available in the Zenodo repository. The original dataset was kept in a password-protected folder in Google Drive.

Reporting guidelines

We consulted the Checklist for Reporting of Results of Internet E-Surveys (CHERRIES) ²⁰ and the Consolidated criteria for Reporting of Qualitative research (COREQ) ²¹ guidelines to produce this research report.

Results

Of the 99 editors invited, 42 opened the invitation (view rate 42%), and 24 completed the survey (response rate 24%) from the 25 who started it (completion rate 96%). The average time spent completing the survey was 15 minutes (SD = 8.5 minutes). Among the 24 participants who completed the survey, nine (37%) worked for seven different journals that had published studies on improving adherence to RGs, seven (29%) worked for five top-10 BMJ journals, four (17%) were members of the MiRoR Network that hold editorial positions in four journals, and a further four (17%) were suggested by other participants based on their expertise on the topic and were editors of three different journals. The 20 journals represented in the survey are listed in Table 1.

Table 1. Journals represented in the survey.

Journals that have published research on RGs	Trials
	The Lancet
	PLOS ONE
	BMC Medicine
	BMC Medical Research and Methodology
	Journal of Oral Rehabilitation
	Journal of Clinical Epidemiology
Journals that belong to BMJ top-10	BMJ
	Archives of Disease in Childhood
	BMJ Open
	BMJ Quality & Safety
	British Journal of Sports Medicine
MiRoR Network members’ journals	Clinical Chemistry
	Systematic Reviews
	Research Integrity and Peer review
	Journal of Global Health
Other journals	F1000
	BMJ Open Science
	Scientific Reports

Open in a new tab

Participants had a variety of editorial roles (editor-in-chief, senior editor, associate editor or others). Most of them were involved in manuscript decision-making and had less than 15 years of experience as journal editors (see Table 2). The anonymised responses from all 24 participants can be accessed in Zenodo ²².

Table 2. Demographic characteristics of the 24 participants.

N=24
Current position	Working full time as a journal editor	8 (33%)
	Working part time (equal or more than 0.5 of their time) as a journal editor	1 (4%)
	Working part time (less than 0.5 of their time) as a journal editor	14 (59%)
	Other (Volunteer editor)	1 (4%)
Editorial role	Editor-in-chief	10 (41%)
	Senior editor	4 (17%)
	Associate editor	4 (17%)
	Other (Editorial director, Technical editor, Assistant editor)	6 (25%)
Involvement in manuscript decision-making	Yes	22 (92%)
Involvement in manuscript decision-making	No	2 (8%)
Years of experience as a journal editor	<5	8 (33%)
	5–15	12 (50%)
	15–25	3 (13%)
	>25	1 (4%)

Open in a new tab

Current practice

Respondents worked at 19 journals. Most respondents’ journals (11/19, 58%) request authors to submit a completed RG checklist with page numbers indicating where the items are addressed when they submit their manuscript. A further seven (37%) instruct but do not request authors to do it, and one (5%) does not request or instruct authors. Among the journals requesting the submission of checklists, four (4/11, 36%) also explicitly ask peer reviewers to use the completed RGs when assessing manuscripts, one (1/11, 9%) asks peer reviewers general questions about the completeness of reporting, and one performs an evaluation of the completeness of reporting by a trained editor using RGs before the initial decision is made on the manuscript. We observed no incongruences between the answers of editors from the same journal. Some respondents mentioned that in their journals (n=4) the interventions described were only applicable to the study types corresponding to the most established RGs (CONSORT, PRISMA ¹¹, or STROBE ¹²) for trials, observational studies and systematic reviews respectively.

Perceptions of nine potential interventions

The mean scores for perceived ease of implementation and potential effectiveness for each intervention are shown in Figure 1.

The two most common interventions were considered the easiest ones to implement: the median scores (1 ^st, 3 ^rd quartiles) for requesting authors to submit checklists with page numbers (Intervention 1) and for asking peer reviewers to use RGs (Intervention 6) were 5 (Q1: 4, Q3: 5) and 4 (Q1: 3, Q3: 5), respectively. By contrast, interventions related to training (Intervention 9), editor involvement in checking completeness of reporting (Intervention 8) and reformatting of the text based on RG requirements (Intervention 4, Intervention 5) were considered the most difficult to implement.

An evaluation of the completeness of reporting by a trained editor was considered the most effective intervention at improving adherence to RGs (Median: 4, Q1: 4, Q3: 5) and the two targeting peer reviewers (Interventions 6 and 7) were perceived as being the least effective (Median: 4, Q1: 2, Q3: 4; Median: 3, Q1: 2, Q3: 4).

Identifying the barriers and facilitators

This section presents the perceived barriers and facilitators of the interventions considered and editors’ suggestions for making the interventions more effective. Table S1 in Extended data ¹⁷ shows a full description of these.

A) Interventions targeting authors (1–5)

The main barriers associated with all of the interventions targeting authors was that authors have to state their adherence to the relevant RG and this does not equate to actual compliance. Moreover, it is resource intensive for journals to check that these requirements are appropriately met by authors. Some editors highlighted that Interventions 3, 4, and 5 would involve special formatting of the submitted manuscript, which could be cumbersome for authors given that manuscripts are often submitted to multiple journals with different formats before being accepted. This is particularly relevant for journals with high rejection rates as it could cause frustration for authors. Some participants mentioned logistical issues as their journal’s manuscript tracking system is not set up to accommodate these interventions. In addition, changes in the manuscript’s format could be incompatible with the journal’s house style.

Intervention 1 was generally considered quick and straightforward for authors, but several participants indicated that there is published empirical evidence of little effectiveness if the checklist is not assessed by a trained editor or administrator ^5–
8.

As Interventions 3, 4, and 5 force authors to tailor the manuscript to RG requirements, participants reported that these could make editors’ and peer reviewers’ jobs easier as the manuscript would be better structured. Importantly, readers would also be able to locate information more easily. Some editors pointed out that, to make these interventions effective, journals would need to provide templates to authors or to integrate these interventions in the submission system. However, some of these interventions (Interventions 2 and 5) were seen as more effective if they were implemented earlier on in the research process, prior to writing the manuscript.

B) Interventions targeting peer reviewers (6, 7)

Most respondents were negative about the potential effectiveness of implementing the two interventions targeting peer reviewers (Intervention 6 and 7) as they felt these would create too much additional work for reviewers. Participants were concerned that the quality of peer review could be compromised as reviewers are not expected to focus on reporting issues but on providing an expert view on the importance, novelty and relevance of the manuscript. Furthermore, peer reviewers may not know which RGs to use and, even if they do, the effectiveness would be dependent on their willingness to use RGs and their expertise in applying them. Several participants indicated that this work should be delegated to paid editorial staff.

C) Interventions targeting editorial staff (8)

This intervention was considered difficult to implement but potentially effective. The main facilitating factor for its successful implementation was that it is performed by a paid or trained professional, which lends credibility to the intervention, reduces the workload of unpaid peer reviewers, and avoids authors overclaiming adherence. The main barriers outlined for this intervention were (i) the budget issues the journal would need to face to train or hire additional editorial staff that could perform the evaluation, especially if the journal receives a large volume of manuscripts, (ii) the editorial delays it may cause, and the (iii) the potential inefficiency of assistant editors or administrators having to delegate decisions in case of doubt, given that sometimes assessing completeness of reporting is a subjective task.

To make this intervention more feasible for journals, editors suggested that the completeness of reporting evaluation could be performed only for manuscripts that are sent out for peer review and, it could be focused on a few core items (different for each RG) that would enable reproducibility. If this intervention was implemented in a journal that requires the submission of a completed checklist, editors could take advantage of the checklist to locate information.

D) Interventions targeting authors, peer reviewers and editors (9)

Training was seen as a potentially effective intervention but difficult to implement. Some participants highlighted that training with follow up sessions would be resource intensive for journals, and especially difficult to enforce. One participant mentioned that credits (such as CME credits ²³) could be used to recognise hours of training. The fact that sometimes the editorial staff is based in different places and zones makes it crucial to consider flexible forms of training, such as online courses. As an example, the EQUATOR Network Toolkits section provides resources for authors, peer reviewers and journal editors ²⁴. However, some participants emphasised that training should also be delivered by research institutions and medical centres.

Further interventions and incentives for authors and journals

When asked about further potentially effective interventions that were not discussed in the survey, some editors mentioned StatReviewer, a reading tool that automatically assesses adherence to RGs and is currently under evaluation ¹⁶. Other respondents also mentioned the possibility of combining some of the interventions discussed in the survey, such as requiring the submission of checklists and trained editors assessing the responses with the information reported in the manuscript.

Moreover, several incentives for authors were listed, including (i) discounts on article processing charges (APCs) for authors that comply with RG requirements, (ii) academic institutions including RG use in the promotion and tenure files, and (iii) credits (such as CME credits ²³) to recognise hours of training on the use of RGs. Journals could also be encouraged to implement certain interventions if (i) there is empirical evidence that these interventions actually improve the reporting quality of the papers or (ii) publishers or the International Committee of Medical Journal Editors (ICMJE) mandate these as a condition of submission to their journals. Even if some of these interventions are proven to be effective, some respondents reported that it is essential to convince publishers that improving the quality of reporting is a worthy investment to resource.

Discussion

This survey explores biomedical journal editors’ perceptions of the practical aspects of the implementation of different interventions to improve adherence to RGs.

Several messages arise from this study. First of all, most editors agreed that the most effective way to improve adherence to RGs is for journals to involve trained editors or administrative staff. Interventions targeting these stakeholders were considered to be difficult to implement for most journals, either because of logistic or resource issues. However, improving the performance of editorial staff is critical ²⁵ and has been shown to have a positive impact on completeness of reporting in the context of a dentistry journal ²⁶. To make these type of interventions more feasible, journals could implement them only for manuscripts that are sent out for peer review. The editorial staff could also take advantage of the RG checklists submitted by authors, that could be automatically populated with text using specific software such as the the tool proposed by Hawwash et al. ²⁷

Most editors considered that checking reporting issues is beyond the role of peer reviewers. Given the voluntary nature of peer review, requiring reviewers to use RGs would cause an additional workload that could compromise the overall quality of the reviews. If checking reporting issues becomes a standard exercise for peer reviewers, some editors are concerned that peer reviewers may be less likely to comment on important aspects of a manuscript, such as its novelty, clinical interest or implications. Furthermore, as finding peer reviewers is becoming increasingly difficult for editors ²⁸, these requirements could make them even less willing to review papers. Additionally, some editors considered that the average peer reviewer does not have enough expertise to go over RG requirements.

We observed that the interventions perceived as potentially most effective improving adherence to RGs appear to be more difficult to implement. Conversely, the most common strategies seem to have been implemented based on their feasibility and not on their potential to improve completeness of reporting. This could be one of the reasons why they have failed to achieve the desired results ^6–
9). Some of our respondents insisted that a key element is that journals, universities, and medical institutions find ways to incentivise author’s compliance with RGs. At the same time, the scientific community needs to find ways to convince publishers that improving the quality of reporting is a worthy investment so that publishers can encourage their journals to adopt strategies to boost completeness of reporting. A recent article indicates that implementing RGs through the editorial process may increase the number of citations to the research reported ²⁹.

A common observation by the survey participants was that the effectiveness of the interventions proposed could depend on the types of articles considered. While RGs for randomised trial protocols, randomised trials or systematic reviews are more established, some others, including most RG extensions, are not well known to the stakeholders involved in the publication process. For this reason, it is important for journals to be clear in their “Instructions for Authors” on what RGs they mandate.

It is noteworthy to mention that, regardless of how checklists are implemented in the editorial process and who has to engage to make the interventions successful, the evaluation of completeness of reporting is a subjective task. This is mainly due to the fact that RGs are not originally designed as evaluation tools but as guidance for authors on how to report their research. For this reason, evaluators could sometimes have different views on whether authors are providing enough information to consider that certain RG items are adequately reported.

This study is subject to several limitations. The response rate was low (24%). However, researchers in health science have witnessed a gradual decrease in survey participation over time ³⁰, especially among health professionals due to the demanding work schedules and increasing frequency of being approached for surveys ³¹. Some recent surveys in the field of peer review show even lower response rates (10–20%) among researchers, peer reviewers and readers ^32,
33. It is also noteworthy that we took a pragmatic approach to identify relevant editors and the sample was small due to not many having conducted or published research on improving adherence to RGs. Whilst n=24 is a small number, the detailed and rich responses that we received showed a high level of engagement with the topic. Despite having the option to increase the sample size by contacting more editors at a lower level of hierarchy in the journals we targeted, we decided not to do it based on the response rate of the survey. That approach would have changed our sampling frame and we would potentially have had less experienced editors commenting. We took that decision as the purpose of the survey was to tap into the experience of those who had tried interventions or had shown interest in this area, instead of seeking a representative sample of editors.

Connected with this, we could expect survey participants to be more prone to adopt interventions than general biomedical editors. However, their experience could also make them more critical of certain strategies that appear to be more effective than they actually are. This could be the case for the intervention of requesting authors to submit checklists on manuscript submission, which has become popular among medical journals despite having little or no impact on completeness of reporting ^6–
9). Editors with less experience of editorial strategies to improve adherence to RGs might expect authors and peer reviewers to respond to certain interventions in a different way than they would do.

We encourage researchers to perform further evaluations of interventions in collaboration with biomedical journals, such as the RCT our research team is currently undergoing ³⁴. Our study aims to evaluate the effect on completeness of reporting of a trained researcher assessing during peer review the consistency between the CONSORT checklists submitted by authors and the information reported in the manuscript, and providing authors with a report indicating any inconsistencies found.

Providing high quality evidence of the effectiveness of different interventions at improving adherence to RGs and discussing how to make them less burdensome are key aspects needed to convince all stakeholders that this effort is worth it.

Conclusions

Biomedical journal editors generally believed that engaging trained professionals in the process of checking adherence to RGs would be the most effective, yet moderately resource intensive, editorial intervention. Also, they thought that standard peer reviewers should not be asked to check RG requirements.

Future evaluations of interventions to improve adherence to RGs can take into account the barriers, facilitators, and incentives for implementing editorial interventions that are described in this survey.

Data availability

Underlying data

Zenodo: Underlying data of the project “A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines”. DOI: https://doi.org/10.5281/zenodo.3407725 ²².

This project contains the following underlying data:

Survey dataset (Dataset including all survey responses).

Extended data

Zenodo: Extended data of the project “A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines”. https://doi.org/10.5281/zenodo.3404002 ¹⁷.

This project contains the following extended data:

Figure S1: Survey questionnaire (Complete version of the survey questionnaire used in this project)

Table S1: Barriers, facilitators and possible improvements of the included interventions (Table containing the barriers, facilitators and possible improvements identified for each of the interventions explored in the survey)

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

Acknowledgments

The authors thank the MiRoR Project and Marie Sklodowska-Curie Actions for their support. This survey is the second part of a larger project whose first part was a scoping review to identify and classify interventions to improve adherence to RGs ⁵. The third part is an RCT to evaluate the impact of assessing during peer review the CONSORT checklist submitted by authors ³⁴.

Funding Statement

This study belongs to the ESR 14 research project from the Methods in Research on Research (MiRoR) project, which has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 676207.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

[version 3; peer review: 3 approved]

References

1. EQUATOR Network: What is a reporting guideline?2019; [cited 2019 Nov 28]. Reference Source [Google Scholar]
2. Schulz KF, Altman DG, Moher D, et al. : CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332. 10.1136/bmj.c332 [DOI] [PMC free article] [PubMed] [Google Scholar]
3. EQUATOR Network: Library for health research reporting.2019; [cited 2019 Dec 2]. Reference Source [Google Scholar]
4. Samaan Z, Mbuagbaw L, Kosa D, et al. : A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–188. 10.2147/JMDH.S43952 [DOI] [PMC free article] [PubMed] [Google Scholar]
5. Blanco D, Altman D, Moher D, et al. : Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open. 2019;9(5):e026589. 10.1136/bmjopen-2018-026589 [DOI] [PMC free article] [PubMed] [Google Scholar]
6. Turner L, Shamseer L, Altman DG, et al. : Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev. 2012;1(1):60. 10.1186/2046-4053-1-60 [DOI] [PMC free article] [PubMed] [Google Scholar]
7. Stevens A, Shamseer L, Weinstein E, et al. : Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review. BMJ. 2014;348:g3804. 10.1136/bmj.g3804 [DOI] [PMC free article] [PubMed] [Google Scholar]
8. Blanco D, Biggane AM, Cobo E, et al. : Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers? Trials. 2018;19(1):80. 10.1186/s13063-018-2475-0 [DOI] [PMC free article] [PubMed] [Google Scholar]
9. Hair K, Macleod MR, Sena ES, et al. : A randomised controlled trial of an Intervention to Improve Compliance with the ARRIVE guidelines (IICARus). Res Integr Peer Rev. 2019;4(1):12. 10.1186/s41073-019-0069-3 [DOI] [PMC free article] [PubMed] [Google Scholar]
10. Patton M: Qualitative Research & Evaluation Methods. 4th ed. Thousand Oaks: SAGE Publications;2014. Reference Source [Google Scholar]
11. Moher D, Liberati A, Tetzlaff J, et al. : Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. 10.1136/bmj.b2535 [DOI] [PMC free article] [PubMed] [Google Scholar]
12. von Elm E, Altman DG, Egger M, et al. : The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Bull World Health Organ. 2007;61(11):867–72. [DOI] [PMC free article] [PubMed] [Google Scholar]
13. Chan AW, Tetzlaff JM, Altman DG, et al. : SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158(3):200–7. 10.7326/0003-4819-158-3-201302050-00583 [DOI] [PMC free article] [PubMed] [Google Scholar]
14. Schroter S: Evidence-based publishing. BMJ.[cited 2019 Nov 28]. Reference Source [DOI] [PMC free article] [PubMed] [Google Scholar]
15. SurveyMonkey Inc: SurveyMonkey.2019; [cited 2019 Nov 29]. Reference Source [Google Scholar]
16. Shanahan DR: A peerless review? Automating methodological and statistical review. [cited 2019 Nov 28]. Reference Source [Google Scholar]
17. Blanco D: Extended data of the project “A survey exploring biomedical editors perceptions of editorial interventions to improve adherence to reporting guidelines”. Zenodo.[cited 2019 Nov 29]. 10.5281/zenodo.3404002 [DOI] [PMC free article] [PubMed]
18. R Core Team: R: A language and environment for statistical computing.2019. Reference Source [Google Scholar]
19. QSR International Pty Ltd: NVivo qualitative data analysis software. Version 12.2018. Reference Source [Google Scholar]
20. Eysenbach G: Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004;6(3):e34. 10.2196/jmir.6.3.e34 [DOI] [PMC free article] [PubMed] [Google Scholar]
21. Tong A, Sainsbury P, Craig J: Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Heal Care. 2007;19(6):349–57. 10.1093/intqhc/mzm042 [DOI] [PubMed] [Google Scholar]
22. Blanco D: Underlying data of the project "A survey exploring biomedical editors' perceptions of editorial interventions to improve adherence to reporting guidelines". [cited 2019 Nov 29]. Reference Source [DOI] [PMC free article] [PubMed]
23. National Institute of Health: What is CME Credit? [cited 2019 Nov 28]. 10.5281/zenodo.3407725 [DOI] [Google Scholar]
24. EQUATOR Network: Toolkits.2019; [cited 2019 Nov 28]. Reference Source [Google Scholar]
25. Cobo E, González JA: Taking advantage of unexpected WebCONSORT results. BMC Med. 2016;14(1):204. 10.1186/s12916-016-0758-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
26. Pandis N, Shamseer L, Kokich VG, et al. : Active implementation strategy of CONSORT adherence by a dental specialty journal improved randomized clinical trial reporting. J Clin Epidemiol. 2014;67(9):1044–8. 10.1016/j.jclinepi.2014.04.001 [DOI] [PubMed] [Google Scholar]
27. Hawwash D, Sharp MK, Argaw A, et al. : Usefulness of applying research reporting guidelines as Writing Aid software: a crossover randomised controlled trial. BMJ Open. 2019;9(11):e030943. 10.1136/bmjopen-2019-030943 [DOI] [PMC free article] [PubMed] [Google Scholar]
28. Publons: Global State of Peer Review 2018. [cited 2019 Nov 28]. Reference Source [Google Scholar]
29. Vilaró M, Cortés J, Selva-O’Callaghan A, et al. : Adherence to reporting guidelines increases the number of citations: the argument for including a methodologist in the editorial process and peer-review. BMC Med Res Methodol. 2019;19(1):112. 10.1186/s12874-019-0746-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
30. Brtnikova M, Crane LA, Allison MA, et al. : A method for achieving high response rates in national surveys of U.S. primary care physicians. PLoS One.Public Library of Science;2018;13(8):e0202755. 10.1371/journal.pone.0202755 [DOI] [PMC free article] [PubMed] [Google Scholar]
31. Flanigan TS, Mcfarlane E, Cook S: Conducting Survey Research among Physicians and other Medical Professionals-A Review of Current Literature. Reference Source [Google Scholar]
32. Price A, Schroter S, Clarke M, et al. : Role of supplementary material in biomedical journal articles: surveys of authors, reviewers and readers. BMJ Open.BMJ Publishing Group;2018;8(9): e021753. 10.1136/bmjopen-2018-021753 [DOI] [PMC free article] [PubMed] [Google Scholar]
33. Mulligan A, Hall L, Raphael E: Peer review in a changing world: An international study measuring the attitudes of researchers. J Am Soc Inf Sci Technol. 2013;64(1):132–61. 10.1002/asi.22798 [DOI] [Google Scholar]
34. Blanco D, Schroter S, Moher D, et al. : Evaluating the Impact of Assessing During Peer Review the CONSORT Checklist Submitted by Authors.2019; [cited 2019 Nov 28]. Reference Source [Google Scholar]

F1000Res. 2020 Jan 20. doi: 10.5256/f1000research.24060.r57976

Reviewer response for version 3

Gerard Urrutia ¹

I’m happy with the revised version and my decision is to recommend approval as it is now.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2019 Dec 23. doi: 10.5256/f1000research.24060.r57975

Reviewer response for version 3

Patricia Logullo ¹

I am very happy with this new version of the manuscript. It is clear, well-reported, and the conclusions are there. Thank you for taking the time to improve it! Thank you also for the opportunity to review it.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2019 Dec 23.

David Blanco ¹

Again, thanks for the level of detail of your comments and the time you took to make them. The paper is much clearer and better reported now.

F1000Res. 2019 Dec 16. doi: 10.5256/f1000research.23906.r57746

Reviewer response for version 2

Patricia Logullo ¹

Positive points

The paper improved a lot from the last version. As I said before, this is a much-needed survey by MiRoR team, and I hope it gets fully published and indexed as soon as possible.

The authors took care to respond to all the reviewers’ questions. However, there are still three points that were not adequately addressed, and I hope I can make some suggestions to help them solve these problems. I also found two additional problems that came up in this second version of the manuscript.

From the 20 points I commented in my last report, the authors adequately responded/solved the problems in the items 1, 3-5, 7, 9-17, 19 and 20. Thank you for that. Below you can find comments on the remaining items (I used the same numbers as before to make it easier to locate them) and the two additional ones.

2) Sample. The authors explained that they purposely searched for editors experienced with reporting guidelines. As much as I don’t agree with this approach, at least it was clearly stated now.

However, the explanation about the sample size as a study limitation still does not seem satisfactory. The authors added a comment in the discussion section attributing the sample size to a low response rate in surveys in general. But if the authors knew there was going to be a low response rate (and they cited very useful references for that), why did they stop recruiting? Why not try to approach other editors/journals? The authors should expand the explanation to respond to two questions:

What were the constraints in increasing the sample? Why did you stop recruiting?
Even choosing to use a “not representative sample”, can we trust that we should listen to the opinions of these 24 editors (a minority among the 99 invited) to say something is feasible or not, for example? What is the effect of this small sample size on the “take-home messages” of this study?

These points were not addressed in the paragraph about study limitations, and I think they should. Also: who were the editors who did not respond? Do you have information about them that you could show in a table? If you can find and show a pattern of response (for example, editors from high-visibility journals would have less interest in responding?), then we can understand better this low response rate.

6) Figure 1. Figure 1 improved a lot, thank you. Now it shows outliers and the types of interventions are clearer too (I wonder if you could put these “nicknames” in Box 1 too!).

However, it is still a bit difficult to read/interpret. I believe the problem is that you chose to show means when you probably should have used medians (more appropriate to scores and to data that is not normally distributed). I suggest the authors try to check for data distribution, try to generate this figure with means, and that they also describe this method in the Methods – Data analysis section.

8) The abstract improved, well done! However, the conclusions seem not to be conclusions at all, not based on your data.

The first sentence is your (and even mine!) opinion, but it not supported by your data. The second sentence tells about your aims, not your conclusions.

The same is valid for the conclusions at the end of the paper: they show valid comments/assumptions but nothing based on your data.

Useful conclusions should be linked to your aims (to survey editors’ perceptions) and read like “the editors think that”... or “the perceptions of the editors are...”.

18) References: you fixed most of them, but there are still references incomplete. Please review # 23, 24 and 28 and add the URLs.

Additional concerns:

Regarding Data Analysis: I noticed you added text in this section about “excluding one questionnaire that was terminated early”. That called my attention: is this one of the 24 (and then we have 23 responded questionnaires) or was it a 25 ^th questionnaire? Also: how did you approach missing data? By excluding the participants? Was there any other case?

Minor: Thank you for providing the names of the journals in Table 1. However, I suspect “Francis and Taylor” is not a journal name. Please, check (maybe there is a confusion with the publisher’s name “Taylor & Francis”, but what was the specific journal?).

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

F1000Res. 2019 Dec 18.

David Blanco ¹

Thanks a lot, Patricia, for your extra comments and the time you took to make them. Please see below our response to them:

Minor comments

References: you fixed most of them, but there are still references incomplete. Please review # 23, 24 and 28 and add the URLs.

In our previous revised manuscript, we already had added the URLs for all Internet references. However, it seems that the copy-editors of F1000 did not explicitly show them and just include a link to them under “Reference source”.

Regarding Data Analysis: I noticed you added text in this section about “excluding one questionnaire that was terminated early”. That called my attention: is this one of the 24 (and then we have 23 responded questionnaires) or was it a 25th questionnaire? Also: how did you approach missing data? By excluding the participants? Was there any other case?

We excluded from analysis one questionnaire (the 25 ^th) where the participant just opened the survey (and therefore SurveyMonkey registered it) but left without answering any question. For this reason, we decided not to count it as a participant. There were no further cases like this. We have made that more explicit in the text (first paragraph of Data analysis, first paragraph of Results).

We have checked that that was a mistake. Thanks to this, we have realised that for one of the existing journals we had two editors instead of one. We have subsequently adjusted the sample description.

Major comments

2) Sample. The authors explained that they purposely searched for editors experienced with reporting guidelines. As much as I don’t agree with this approach, at least it was clearly stated now.

What were the constraints in increasing the sample? Why did you stop recruiting?

Even choosing to use a “not representative sample”, can we trust that we should listen to the opinions of these 24 editors (a minority among the 99 invited) to say something is feasible or not, for example? What is the effect of this small sample size on the “take-home messages” of this study?

We have added further justifications on whether we believed that sample size and response rates were adequate for the purpose of the survey: “Despite having the option to increase the sample size by contacting more editors at a lower level of hierarchy in the journals we targeted, we decided not to do it based on the response rate of the survey. That approach would have changed our sampling frame and we would potentially have had less experienced editors commenting. We took that decision as the purpose of the survey was to tap into the experience of those who had tried interventions or had shown interest in this area, instead of seeking a representative sample of editors.”

We found no pattern of response among the journals/editors surveyed – participation was pretty homogeneous and covered journals of a variety of impact factors and areas, as can be seen in Table 1.

6) Figure 1. Figure 1 improved a lot, thank you. Now it shows outliers and the types of interventions are clearer too (I wonder if you could put these “nicknames” in Box 1 too!).

However, it is still a bit difficult to read/interpret. I believe the problem is that you chose to show means when you probably should have used medians (more appropriate to scores and to data that is not normally distributed). I suggest the authors try to check for data distribution, try to generate this figure with means, and that they also describe this method in the Methods – Data analysis section.

We are now reporting the results for the ease of implementation/effectiveness in terms of medians and quartiles instead of means and standard deviations. We have indicated this in the Methods-Data analysis section.

Regarding Figure 1, we have removed the means and we are now displaying the medians as blue horizontal lines. Moreover, we decided to remove the black points representing the individual scores as they do not allow us to see the medians in the graphs. In this way, Figure 1 is clearer and cleaner.

8) Conclusions. The abstract improved, well done! However, the conclusions seem not to be conclusions at all, not based on your data.

The first sentence is your (and even mine!) opinion, but it not supported by your data. The second sentence tells about your aims, not your conclusions.

The same is valid for the conclusions at the end of the paper: they show valid comments/assumptions but nothing based on your data.

Useful conclusions should be linked to your aims (to survey editors’ perceptions) and read like “the editors think that”... or “the perceptions of the editors are...”.

We have modified the Conclusions both in the abstract and at the end of the paper, linking them to the survey aims (perceptions, barriers, facilitators…). We have also moved to the end of the Discussion section what was reported in the old Conclusions.

F1000Res. 2019 Dec 12. doi: 10.5256/f1000research.23906.r57748

Reviewer response for version 2

Dennis W Lendrem ^1,²

This paper tackles an important issue in science policy research. The authors have responded well to comments on an earlier draft. They have addressed the issues raised and the limitations of the study are clearly stated. While those limitations mean the survey results must be interpreted with caution, the findings are likely to inform discussions around the barriers to and facilitators of interventions to improve adherence to reporting guidelines.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2019 Nov 26. doi: 10.5256/f1000research.22597.r56388

Reviewer response for version 1

Patricia Logullo ¹

Positive points:

This is a much-needed survey, and I am happy and thankful that the MiRoR team invested their time and expertise in undertaking it. The subject of the adoption of reporting guidelines in the editorial process is sensitive, relevant and worth exploring. Hearing editors is a crucial step for the development of interventions aimed to improve the quality of biomedical research reporting. I congratulate the authors for the initiative.

The main result of the study (that the interventions perceived as effective are the most difficult to implement and vice-versa) points out to the need to verifying feasibility before proposing interventions, something that we should all think about.

I hope the authors find these suggestions helpful and constructive, and that they find these to be positive contributions.

Major concerns: methodology

1) Sample characteristics. Although the authors should be praised for the idea of asking editors about interventions to improve reporting, I have a concern about what editors should be surveyed. If one wants to explore the perceived ease of the implementation of one intervention, is it adequate to choose a population of editors whom we know are already “knowledgeable about the topic”?

If these editors are aware of the reporting quality crisis and the reporting guidelines in general, wouldn’t these be more prone to adopt interventions that aim to improve the quality of publications? Therefore, aren’t these the people who would say “yes” to any intervention proposed?

Should the authors not interview editors who have never heard about reporting guidelines too? Wouldn’t this heterogeneous sample be more representative of the scenario of difficulties — or the ease — of implementing reporting guidelines checks in the editorial process? Are the study results only applicable to journals that already endorse or require reporting guidelines?

I would like to see comments about this limitation in the discussion if the authors agree with this or even if they do not.

2) Sample size. My second most important concern is about sample size. In the Methods section, the authors do not explain how they planned the sample size and based on what assumptions. 24% seems to be a very low response rate. Do the authors believe 24 is enough? This limitation should be discussed.

3) Effectiveness. When you asked your participants about their opinions on “how effective” something would be, what was the definition of effectiveness? Would it be their perception about the intervention improving the adherence to reporting guidelines checklists? Was it a more subjective view of “manuscript quality”?

4) Statreviewer. The authors mention in the text, in Methods:

“Based on feedback from the pilot we decided not to include the intervention “Implementation of the automatic tool Statreviewer14” since participants were not aware of this software and stated that their perceptions would strongly depend on details about how it operates which are not publicly available.”

However, on page 7, Results, they state:

“The implementation of reading tools that automatically assess adherence to RGs, such as Statreviewer14, were seen as potentially interesting interventions.”

After all, was this intervention Statreviewer evaluated (as in Results) or was it not (as in Methods)? Do you mean they mentioned other tools, such as Penelope, GoodReports, Cobweb or others? Or were they referring to Statreviewer?

Major concerns: reporting

5) Journals. You interviewed 24 editors from 20 journals. What are these journals? I did not see a list of their names, and I believe it is important.

6) Figure 1. Figure 1 is difficult to read/interpret. I like the idea of having the two outcomes (ease of implementation and effectiveness) in one figure very much so that one can compare the two. However, the bar chart is not clear at all, nor easy to interpret. Colours do not help. The authors should consider building a figure with plot points (which would be easy, given the small sample) showing where each participant sits within the Likert scale — not bars.

Another suggestion for Figure 1 (here, just a suggestion) is to add something to remind the readers what the interventions are. As Box 1 is far from the Figure, it is difficult to remember what each intervention is. It would be useful to have “nicknames”, for example:

1. Page numbers

2. Checklist

3. Highlight

4. Subheadings

5. COBWEB

6. Use RG

7. Check RG

8. Staff

9. Training

7) Peer reviewers’ checks. The abstract is, in general, well written and clear. However, I got intrigued by this sentence:

“Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and could decrease the overall quality of reviews.”

I was surprised that peer reviewers checking reporting guidelines adherence would decrease the overall quality of the paper. Then I reread it in the paper, but here put in a complete sentence, that makes sense:

“Participants were concerned that the quality of peer review could be compromised as reviewers should focus on the manuscript’s content and not on the reporting issues.”

Meaning, I suppose, that peer reviewers would focus on checking reporting guidelines and not on their “expert view” of the study. I get it, but it is not very clear, neither in the text, nor in the abstract, and I invite the authors to rewrite both. Especially because “reporting guidelines” are about content too. They point out the content elements that should not be absent from papers, so using the word “content” in the manuscript text is not really appropriate. Please, re-evaluate.

8) As I said above, the abstract is good, and there is room for growth (you have now 232 words). However, I could not find the conclusions in the abstract helpful. Could you please inform better what you did conclude? What “points raised in the survey” should be considered?

9) Checklists as forms. ...“assessing completeness of reporting is a subjective task”. This is one of the most important sentences in this manuscript, it tells us about using reporting guidelines checklists as evaluation tools — for what they have not been created. I saw no discussion about this in your paper.

10) Discussion. The discussion of the paper is short, and it lacks several points pointed above. Mainly, it does not discuss the limitations and potential bias of this study. Discussing limitations is a requirement from F1000Research journal.

11) You mention you did analyse qualitative information using NVivo. How did you do it?

12) There are at least eight items from the CHERRIES checklist that I could not see in your paper, and you might want to report, even if in a separate (or supplementary) box, from data protection, preventing multiple entries, to randomisation of items, and others.

13) From Box 1, I do not quite get the difference between Interventions 1 and 2. It seems that both ask authors to submit a populated/completed RG checklist. Is it that #1 asks for page numbers and #2 does not?

14) In my opinion, the sentences from “To contact editors...” to “improve adherence to RGs” belong more to the “Participants” section than to the “Procedure”. Especially the first sentence.

Also “Participants could suggest further editors that they considered could contribute to the survey” is something that I would consider as a snowballing or recruitment method, not the survey procedure per se. So, I would suggest putting everything related to recruitment together in one section, to make it easier for your reader (who might be interested in undertaking a similar survey) to find information on such a critical issue as recruitment. Also: how could participants suggest? Was it a specific question in the SurveyMonkey form?

15) Were participants given an estimation of the time required to answer the survey before they began?

Minor: about citations

16) Reference number 1 (EQUATOR Network website) is not a proper, complete reference. It points to the home page of the website, not to a document, and as so, it does not show from where the authors collected that information (the reader cannot find them in the home page). The correct way of citing would be adding the full URL. A suggestion: EQUATOR Network. What is a reporting guideline? Available at:

http://www.equator-network.org/about-us/what-is-a-reporting-guideline/. Cited on 19/11/2019.

17) Something similar happens to reference number 13. Author and title are missing. It would help readers if it could be typed like this:

Schoter S. Evidence-based publishing. BMJ. Available at: https://www.bmj.com/about-bmj/evidence-based-publishing. Cited on 19/11/2019.

18) Reference 14 also lacks authors and titles. References 13, 14, 16, 21 and 22 lack a lot of elements necessary for readers to be able to find them. When working in a printed version (as I am now), you cannot see what the document is, where it is published or by whom — these are available only by those reading online and able to “click” on the links.

19) I would change the sentence:

“Raw survey results are given as Underlying data ²⁰.”

To something clearer and that does not require the reader to go to the reference list to understand what it is.

“Table XX in the supplementary file X ²⁰ shows the responses from all 24 participants.”

20) Has the study by Hawwash et al. (reference 25) been published yet? (The authors cited the protocol only). Do the authors plan to cite other electronic tools that are out there?

Thank you for the opportunity of reading and evaluating this paper.

F1000Res. 2019 Dec 9.

David Blanco ¹

We thank you for your time and detailed comments on the manuscript. Please find below a point by point response to them:

Major concerns: methodology

1) Sample characteristics. Although the authors should be praised for the idea of asking editors about interventions to improve reporting, I have a concern about what editors should be surveyed. If one wants to explore the perceived ease of the implementation of one intervention, is it adequate to choose a population of editors whom we know are already “knowledgeable about the topic”?

I would like to see comments about this limitation in the discussion if the authors agree with this or even if they do not.

Firstly, we have now included some more information in the “Participants” section to make it clear what kind of participants we were aiming to reach. Secondly, we have discussed in the Discussion section the other points you mention.

2) Sample size. My second most important concern is about sample size. In the Methods section, the authors do not explain how they planned the sample size and based on what assumptions. 24% seems to be a very low response rate. Do the authors believe 24 is enough? This limitation should be discussed.

We have now discussed these points in the Discussion section.

3) Effectiveness. When you asked your participants about their opinions on “how effective” something would be, what was the definition of effectiveness? Would it be their perception about the intervention improving the adherence to reporting guidelines checklists? Was it a more subjective view of “manuscript quality”?

Thanks to your comment, we have just realised that we only mentioned that we were talking about “effectiveness at improving adherence to RGs” in the methods section, but not in the abstract and introduction. We have now clarified that in the abstract and introduction, and also in some parts of the methods, results and discussion to remind the readers that we refer to effectiveness at improving adherence to RGs.

4) Statreviewer. The authors mention in the text, in Methods:

However, on page 7, Results, they state:

“The implementation of reading tools that automatically assess adherence to RGs, such as Statreviewer14, were seen as potentially interesting interventions.”

We have clarified in the results section (“Further interventions and incentives for authors and journals”) that Statreviewer was not included in the survey but mentioned by some participants when asked about further potentially effective interventions.

Major concerns: reporting

5) Journals. You interviewed 24 editors from 20 journals. What are these journals? I did not see a list of their names, and I believe it is important.

The 20 journals represented are now listed in Table 1. We have also improved the description of the sample in the first paragraph of the Results section.

6) Figure 1. Figure 1 is difficult to read/interpret. I like the idea of having the two outcomes (ease of implementation and effectiveness) in one figure very much so that one can compare the two. However, the bar chart is not clear at all, nor easy to interpret. Colours do not help. The authors should consider building a figure with plot points (which would be easy, given the small sample) showing where each participant sits within the Likert scale — not bars.

We have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.

1. Page numbers

2. Checklist

3. Highlight

4. Subheadings

5. COBWEB

6. Use RG

7. Check RG

8. Staff

9. Training

We have incorporated similar nicknames to help readers remember the interventions.

7) Peer reviewers’ checks. The abstract is, in general, well written and clear. However, I got intrigued by this sentence:

“Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and could decrease the overall quality of reviews.”

“Participants were concerned that the quality of peer review could be compromised as reviewers should focus on the manuscript’s content and not on the reporting issues.”

We completely agree with the point you raise. We have re-written this point in the Abstract, Results and Conclusions. “Participants believed that checking adherence to guidelines goes beyond the role of peer reviewers and were concerned that the quality of peer review could be compromised. Reviewers are generally not expected to focus on reporting issues but on providing an expert view on the importance, novelty, and relevance of the manuscript.”

We have re-written and expanded the Conclusions in the abstract. The abstract is now 295 words.

9) Checklists as forms. ...“assessing completeness of reporting is a subjective task”. This is one of the most important sentences in this manuscript, it tells us about using reporting guidelines checklists as evaluation tools — for what they have not been created. I saw no discussion about this in your paper.

We have now discussed this point in the paragraph before the limitations of the study.

10) Discussion. The discussion of the paper is short, and it lacks several points pointed above. Mainly, it does not discuss the limitations and potential bias of this study. Discussing limitations is a requirement from F1000Research journal.

We have now included such a section in the Discussion section.

11) You mention you did analyse qualitative information using NVivo. How did you do it?

We have now provided further information on how we did the qualitative analysis with NVivo (“Data analysis” section).

12) There are at least eight items from the CHERRIES checklist that I could not see in your paper, and you might want to report, even if in a separate (or supplementary) box, from data protection, preventing multiple entries, to randomisation of items, and others.

We have now included the information corresponding to the missing items throughout the whole Methods section.

13) From Box 1, I do not quite get the difference between Interventions 1 and 2. It seems that both ask authors to submit a populated/completed RG checklist. Is it that #1 asks for page numbers and #2 does not?

#1 asks for page numbers and #2 for text from the manuscript. Some people may think that if you nudge authors to have to copy-paste text into the checklist they may tend to take it more seriously than if you just ask them to put a page number. We have slightly modified their wording in Box 1 to make it clearer.

14) In my opinion, the sentences from “To contact editors...” to “improve adherence to RGs” belong more to the “Participants” section than to the “Procedure”. Especially the first sentence. Also “Participants could suggest further editors that they considered could contribute to the survey” is something that I would consider as a snowballing or recruitment method, not the survey procedure per se. So, I would suggest putting everything related to recruitment together in one section, to make it easier for your reader (who might be interested in undertaking a similar survey) to find information on such a critical issue as recruitment. Also: how could participants suggest? Was it a specific question in the SurveyMonkey form?

We have restructured that part of the manuscript and divided it into “Recruitment” and “Survey administration”. We have also provided the information requested.

15) Were participants given an estimation of the time required to answer the survey before they began?

We have now included this information (“Survey administration” subsection).

Minor: about citations

16) Reference number 1 (EQUATOR Network website) is not a proper, complete reference. It points to the home page of the website, not to a document, and as so, it does not show from where the authors collected that information (the reader cannot find them in the home page). The correct way of citing would be adding the full URL. A suggestion: EQUATOR Network. What is a reporting guideline? Available at:

http://www.equator-network.org/about-us/what-is-a-reporting-guideline/. Cited on 19/11/2019.

Done.

17) Something similar happens to reference number 13. Author and title are missing. It would help readers if it could be typed like this:

Schoter S. Evidence-based publishing. BMJ. Available at: https://www.bmj.com/about-bmj/evidence-based-publishing. Cited on 19/11/2019.

Done.

18) Reference 14 also lacks authors and titles. References 13, 14, 16, 21 and 22 lack a lot of elements necessary for readers to be able to find them. When working in a printed version (as I am now), you cannot see what the document is, where it is published or by whom — these are available only by those reading online and able to “click” on the links.

Done.

19) I would change the sentence:

“Raw survey results are given as Underlying data20.”

To something clearer and that does not require the reader to go to the reference list to understand what it is.

“Table XX in the supplementary file X20 shows the responses from all 24 participants.”

Done.

20) Has the study by Hawwash et al. (reference 25) been published yet? (The authors cited the protocol only). Do the authors plan to cite other electronic tools that are out there?

We have now cited the results paper by Hawwash et al., which got published on 6 November. Regarding the electronic tools, we have already cited COBWEB and Statreviewer. We did not cite Penelope as it does not explicitly check RG requirements – it just helps authors submit the relevant RG checklist.

F1000Res. 2019 Nov 20. doi: 10.5256/f1000research.22597.r55889

Reviewer response for version 1

Gerard Urrutia ¹

This is an interesting and original study that deepens the knowledge about interventions aimed at improving adherence to reporting guidelines by medical journals, an important issue. The study focuses on the perspective of the biomedical editors and provides valuable information on their perceptions of various interventions that have been proposed, identifies barriers and facilitators and provides possible solutions.

However, the study includes a limited sample of participants (24 editors belonging to 20 medical journals), which limits the scope of the conclusions. Although the study, according to its stated objectives, does not intend to offer a representative view of the editors of all medical journals, the limited number of participants as well as their representativeness (the sources from which they were identified and selected allows us to think that it is a highly selected sample of publishers with previous experience and a potential motivation in the subject), is not mentioned in the discussion. It is noteworthy that, despite this potentially "favourable" sample, the response rate is very low (24%). What would have been the participation rate and the responses if a larger and more representative sample had been selected? Although this was not an objective of the study, it would deserve some reflection in the discussion.

It would also be useful to provide details on the origin of the participants in the survey according to the sources of selection, which would improve the description of the sample.

The results presented both in the section on current practices as well as in perceptions and in barriers and facilitators are consistent, clear and practical. The analysis is descriptive and does not pose difficulties.

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard.

F1000Res. 2019 Dec 9.

David Blanco ¹

We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

We have now discussed these points in the Discussion section.

It is noteworthy that, despite this potentially "favourable" sample, the response rate is very low (24%). What would have been the participation rate and the responses if a larger and more representative sample had been selected? Although this was not an objective of the study, it would deserve some reflection in the discussion.

We have now included some comments about the response rate in a new paragraph on Limitations (Discussion section).

It would also be useful to provide details on the origin of the participants in the survey according to the sources of selection, which would improve the description of the sample.

We have now added that information to the first paragraph of the Results section.

F1000Res. 2019 Nov 11. doi: 10.5256/f1000research.22597.r55890

Reviewer response for version 1

Dennis W Lendrem ^1,²

This paper reports on the views of a sample of biomedical editors on editorial interventions to improve adherence to reporting guidelines. The paper is clear, transparent, and well documented.

A limitation of the study is that the results are based upon a survey return rate of just 24%. The returns are likely biased favouring editors with stronger views on reporting guidelines. In addition, it is not clear whether all ten of the top-ten journals were represented, or whether one or more journals dominate the response rate. In addition it is not completely clear how many participants came from the MiRoR Network, or journals previously publishing studies on Reporting Guidelines.

Figure 1 is hard work to interpret and might be presented more usefully as two separate graphs. In addition, the colour scheme could be changed to highlight those interventions considered "Very easy" or "Moderately easy" to implement, and those ranked as "Very effective" or "Moderately effective". (Note that the proportions of each score could still be retained.)

However, this is a useful addition to the literature offering valuable insight into some views on guidelines and candidate interventions promoting closer adherence to guidelines. The paper would be improved by a paragraph on Limitations of the Study.

F1000Res. 2019 Dec 9.

David Blanco ¹

We thank you for your time and comments on the manuscript. Please find below a point by point response to them:

A limitation of the study is that the results are based upon a survey return rate of just 24%. The returns are likely biased favouring editors with stronger views on reporting guidelines.

We have now reflected on this point in the Discussion section.

In addition, it is not clear whether all ten of the top-ten journals were represented, or whether one or more journals dominate the response rate. In addition it is not completely clear how many participants came from the MiRoR Network, or journals previously publishing studies on Reporting Guidelines.

We have now added that information to the first paragraph of the Results section.

In order to make Figure 1 easier to interpret, we have substituted the bar plot by a box plot. We have also plotted the individual and mean scores for each intervention and each outcome. Everything is explained in the legend.

However, we prefer not to separate it into two graphs as we believe it is interesting for the readers to be able to compare the scores that each intervention obtains for each of the two outcomes.

The paper would be improved by a paragraph on Limitations of the Study.

We have now included a section on Limitations of the study in the Discussion section.

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Data Citations

Blanco D: Extended data of the project “A survey exploring biomedical editors perceptions of editorial interventions to improve adherence to reporting guidelines”. Zenodo.[cited 2019 Nov 29]. 10.5281/zenodo.3404002 [DOI] [PMC free article] [PubMed]
Blanco D: Underlying data of the project "A survey exploring biomedical editors' perceptions of editorial interventions to improve adherence to reporting guidelines". [cited 2019 Nov 29]. Reference Source [DOI] [PMC free article] [PubMed]

Data Availability Statement

Underlying data

This project contains the following underlying data:

Survey dataset (Dataset including all survey responses).

Extended data

This project contains the following extended data:

Figure S1: Survey questionnaire (Complete version of the survey questionnaire used in this project)

Data are available under the terms of the Creative Commons Attribution 4.0 International license (CC-BY 4.0).

[ref-1] 1. EQUATOR Network: What is a reporting guideline?2019; [cited 2019 Nov 28]. Reference Source [Google Scholar]

[ref-2] 2. Schulz KF, Altman DG, Moher D, et al. : CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c332. 10.1136/bmj.c332 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-3] 3. EQUATOR Network: Library for health research reporting.2019; [cited 2019 Dec 2]. Reference Source [Google Scholar]

[ref-4] 4. Samaan Z, Mbuagbaw L, Kosa D, et al. : A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–188. 10.2147/JMDH.S43952 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-5] 5. Blanco D, Altman D, Moher D, et al. : Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open. 2019;9(5):e026589. 10.1136/bmjopen-2018-026589 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-6] 6. Turner L, Shamseer L, Altman DG, et al. : Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Syst Rev. 2012;1(1):60. 10.1186/2046-4053-1-60 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-7] 7. Stevens A, Shamseer L, Weinstein E, et al. : Relation of completeness of reporting of health research to journals' endorsement of reporting guidelines: systematic review. BMJ. 2014;348:g3804. 10.1136/bmj.g3804 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-8] 8. Blanco D, Biggane AM, Cobo E, et al. : Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers? Trials. 2018;19(1):80. 10.1186/s13063-018-2475-0 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-9] 9. Hair K, Macleod MR, Sena ES, et al. : A randomised controlled trial of an Intervention to Improve Compliance with the ARRIVE guidelines (IICARus). Res Integr Peer Rev. 2019;4(1):12. 10.1186/s41073-019-0069-3 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-10] 10. Patton M: Qualitative Research & Evaluation Methods. 4th ed. Thousand Oaks: SAGE Publications;2014. Reference Source [Google Scholar]

[ref-11] 11. Moher D, Liberati A, Tetzlaff J, et al. : Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535. 10.1136/bmj.b2535 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-12] 12. von Elm E, Altman DG, Egger M, et al. : The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Bull World Health Organ. 2007;61(11):867–72. [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-13] 13. Chan AW, Tetzlaff JM, Altman DG, et al. : SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158(3):200–7. 10.7326/0003-4819-158-3-201302050-00583 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-14] 14. Schroter S: Evidence-based publishing. BMJ.[cited 2019 Nov 28]. Reference Source [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-15] 15. SurveyMonkey Inc: SurveyMonkey.2019; [cited 2019 Nov 29]. Reference Source [Google Scholar]

[ref-16] 16. Shanahan DR: A peerless review? Automating methodological and statistical review. [cited 2019 Nov 28]. Reference Source [Google Scholar]

[ref-17] 17. Blanco D: Extended data of the project “A survey exploring biomedical editors perceptions of editorial interventions to improve adherence to reporting guidelines”. Zenodo.[cited 2019 Nov 29]. 10.5281/zenodo.3404002 [DOI] [PMC free article] [PubMed]

[ref-18] 18. R Core Team: R: A language and environment for statistical computing.2019. Reference Source [Google Scholar]

[ref-19] 19. QSR International Pty Ltd: NVivo qualitative data analysis software. Version 12.2018. Reference Source [Google Scholar]

[ref-20] 20. Eysenbach G: Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res. 2004;6(3):e34. 10.2196/jmir.6.3.e34 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-21] 21. Tong A, Sainsbury P, Craig J: Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Heal Care. 2007;19(6):349–57. 10.1093/intqhc/mzm042 [DOI] [PubMed] [Google Scholar]

[ref-22] 22. Blanco D: Underlying data of the project "A survey exploring biomedical editors' perceptions of editorial interventions to improve adherence to reporting guidelines". [cited 2019 Nov 29]. Reference Source [DOI] [PMC free article] [PubMed]

[ref-23] 23. National Institute of Health: What is CME Credit? [cited 2019 Nov 28]. 10.5281/zenodo.3407725 [DOI] [Google Scholar]

[ref-24] 24. EQUATOR Network: Toolkits.2019; [cited 2019 Nov 28]. Reference Source [Google Scholar]

[ref-25] 25. Cobo E, González JA: Taking advantage of unexpected WebCONSORT results. BMC Med. 2016;14(1):204. 10.1186/s12916-016-0758-4 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-26] 26. Pandis N, Shamseer L, Kokich VG, et al. : Active implementation strategy of CONSORT adherence by a dental specialty journal improved randomized clinical trial reporting. J Clin Epidemiol. 2014;67(9):1044–8. 10.1016/j.jclinepi.2014.04.001 [DOI] [PubMed] [Google Scholar]

[ref-27] 27. Hawwash D, Sharp MK, Argaw A, et al. : Usefulness of applying research reporting guidelines as Writing Aid software: a crossover randomised controlled trial. BMJ Open. 2019;9(11):e030943. 10.1136/bmjopen-2019-030943 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-28] 28. Publons: Global State of Peer Review 2018. [cited 2019 Nov 28]. Reference Source [Google Scholar]

[ref-29] 29. Vilaró M, Cortés J, Selva-O’Callaghan A, et al. : Adherence to reporting guidelines increases the number of citations: the argument for including a methodologist in the editorial process and peer-review. BMC Med Res Methodol. 2019;19(1):112. 10.1186/s12874-019-0746-4 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-30] 30. Brtnikova M, Crane LA, Allison MA, et al. : A method for achieving high response rates in national surveys of U.S. primary care physicians. PLoS One.Public Library of Science;2018;13(8):e0202755. 10.1371/journal.pone.0202755 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-31] 31. Flanigan TS, Mcfarlane E, Cook S: Conducting Survey Research among Physicians and other Medical Professionals-A Review of Current Literature. Reference Source [Google Scholar]

[ref-32] 32. Price A, Schroter S, Clarke M, et al. : Role of supplementary material in biomedical journal articles: surveys of authors, reviewers and readers. BMJ Open.BMJ Publishing Group;2018;8(9): e021753. 10.1136/bmjopen-2018-021753 [DOI] [PMC free article] [PubMed] [Google Scholar]

[ref-33] 33. Mulligan A, Hall L, Raphael E: Peer review in a changing world: An international study measuring the attitudes of researchers. J Am Soc Inf Sci Technol. 2013;64(1):132–61. 10.1002/asi.22798 [DOI] [Google Scholar]

[ref-34] 34. Blanco D, Schroter S, Moher D, et al. : Evaluating the Impact of Assessing During Peer Review the CONSORT Checklist Submitted by Authors.2019; [cited 2019 Nov 28]. Reference Source [Google Scholar]

PERMALINK

A survey exploring biomedical editors’ perceptions of editorial interventions to improve adherence to reporting guidelines

David Blanco

Darko Hren

Jamie J Kirkham

Erik Cobo

Sara Schroter

Roles

Version Changes

Revised. Amendments from Version 2

Abstract

Abbreviations

Introduction

Methods

Participants

Recruitment

Survey administration

Response rates

Questionnaire development

Box 1. Interventions included and their targets.

Data analysis

Ethics approval & informed consent

Reporting guidelines

Results

Table 1. Journals represented in the survey.

Table 2. Demographic characteristics of the 24 participants.

Current practice

Perceptions of nine potential interventions

Figure 1. Scores for perceived ease of implementation and perceived effectiveness.

Identifying the barriers and facilitators

Further interventions and incentives for authors and journals

Discussion

Conclusions

Data availability

Underlying data

Extended data

Acknowledgments

Funding Statement

References

Reviewer response for version 3

Gerard Urrutia

Roles

Reviewer response for version 3

Patricia Logullo

Roles

David Blanco

Reviewer response for version 2

Patricia Logullo

Roles

David Blanco

Reviewer response for version 2

Dennis W Lendrem

Roles

Reviewer response for version 1

Patricia Logullo

Roles

David Blanco

Reviewer response for version 1

Gerard Urrutia

Roles

David Blanco

Reviewer response for version 1

Dennis W Lendrem

Roles

David Blanco

Associated Data

Data Citations

Data Availability Statement

Underlying data

Extended data

ACTIONS

PERMALINK

RESOURCES

Similar articles

Cited by other articles

Links to NCBI Databases