The STROBE extensions: protocol for a qualitative assessment of content and a survey of endorsement

Abstract

Introduction

The STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) Statement was developed in response to inadequate reporting of observational studies. In recent years, several extensions to STROBE have been created to provide more nuanced field-specific guidance for authors. The content and the prevalence of extension endorsement have not yet been assessed. Accordingly, there are two aims: (1) to classify changes made in the extensions to identify strengths and weaknesses of the original STROBE checklist and (2) to determine the prevalence and typology of endorsement by journals in fields related to extensions.

Methods and analysis

Two independent researchers will assess additions in each extension. Additions will be coded as ‘field specific’ (FS) or ‘not field specific’ (NFS). FS is defined as particularly relevant information for a single field and guidance provided generally cannot be extrapolated beyond that field. NFS is defined as information that reflects epidemiological or methodological tenets and can be generalised to most, if not all, types of observational research studies. Intraclass correlation will be calculated to measure reviewers’ concordance. On disagreement, consensus will be sought. Individual additions will be grouped by STROBE checklist items to identify the frequency and distribution of changes.

Journals in fields related to extensions will be identified through National Library of Medicine PubMed Broad Subject Terms, screened for eligibility and further distilled via Ovid MEDLINE® search strategies for observational studies. Text describing endorsement will be extracted from each journal’s website. A classification scheme will be created for endorsement types and the prevalence of endorsement will be estimated. Analyses will use NVivo V.11 and SAS University Edition.

Ethics and dissemination

This study does not require ethical approval as it does not involve human participants. This study has been preregistered on Open Science Framework.

Strengths and limitations of this study.

• Our systematic approach to qualitatively assess the content of the additions made in the STrengthening the Reporting of OBservational studies in Epidemiology extensions provides a comprehensive overview of the types of changes made and can identify redundancies and problem areas.

• Our method involves standardised search strategies in Ovid Medline, designed to capture a representative sample and circumvent issues of subjectivity in the identification of eligible journals.

• This study will create an open source corpus of recent observational studies spanning seven fields which future researchers can use to assess completeness of reporting or other topics of interest.

• The bibliometric aspect of this study only focuses on seven extensions and fields so results are not generalisable to other studies.

Introduction

The STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) Statement was developed in 2007 in response to the pervasiveness of inadequate reporting of observational studies. STROBE provides a checklist of items that serve as a reference for how to report sufficient information for observational research involving cohort, case–control and cross-sectional studies.¹ The guidelines have been endorsed by the International Committee of Medical Journal Editors (ICMJE) and the accompanying checklist is sometimes explicitly used as a requirement for manuscript submission.² However, there is no standard method of endorsement by journals and little is known about the most effective ways to apply the guidelines in practice.^3–5

Regarding the reporting of clinical trials, requiring a completed Consolidated Standards of Reporting Trials (CONSORT) checklist on submission of a manuscript has been shown to lead to improvements in reporting.⁶ However, some journals do not want to take responsibility for guideline enforcement and many overlook non-adherence to guidelines; editors have expressed beliefs that their journal’s current policies are adequate or that they fear losing authors to other journals that have less strict requirements for publication.^7–9 Editors may also be unaware of the existence of guidelines, as demonstrated by low endorsement rates by journals in dentistry,¹⁰ veterinary medicine⁷ and urology.¹¹ On the other hand, the evidence for the endorsement of STROBE is also mixed. Endorsement was not shown to be associated with better reporting for items related to confounding, regardless of strength.¹²

Several field-specific extensions to STROBE have been designed in recent years in an effort to promote complete reporting, provide more nuanced guidance for authors and perhaps address editor’s concerns that STROBE is not focused enough for their journal. Extensions for other reporting guidelines are common; however, the creation of extensions for STROBE seems to outpace those for other reporting guidelines such as the CONSORT.¹³ Since the publication of STROBE in 2007, 13 extensions have been published and indexed by the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network, an international collaboration that promotes transparent and accurate reporting and indexes reporting guidelines.¹⁴ In contrast, CONSORT was first published in 1996, updated in 2001 and further revised in 2010, yet only 17 extensions have been published during that period.¹⁵ The reason behind the difference in the pacing of publications of extensions is unclear. Perhaps the concept of field-specific extensions to reporting guidelines was pioneered by CONSORT, thus making the idea more commonplace for subsequent reporting guidelines. Alternatively, the complexity of the types of observational research studies may require more guidance due to the wide variety of methods employed in observational studies. Regardless of the reasoning, it is evident that authors are still perceiving a need to provide more guidance on how to report information about their studies. However, until now, many of these initiatives have not been evaluated.

Extensions to STROBE offer a potential new avenue for promoting more complete reporting but their use has been largely unassessed and, similar to STROBE, they may face implementation and usage problems.^{3 7} Being intended as general guidelines for observational studies, STROBE should include necessary information that is sufficient to most observational studies. For some fields, however, STROBE guidelines may not be sufficient due to specific requirements within the field. This gap is then covered by an extension for that field. However, when extensions include non-specific guidance that can be extrapolated to most observational studies (eg, details about participants, settings, confounders, follow-up, biases or any other general epidemiological constructs), it suggests potential deficiencies in STROBE checklist. If the content is already in STROBE, extension authors may have thought that it was not clearly communicated, or that it is necessary to include it in the checklist instead of being only in the explanation and elaboration document. While, if the content is not already in STROBE, extension authors may have identified a gap or insufficiency which should be considered as an addendum to STROBE. Therefore, by identifying non-specific or redundant guidance suggested in the STROBE extensions, we will be able to identify perceived gaps and deficiencies in the current STROBE checklist and potentially reduce future waste in the process of extension creation.

A perceived lack of confidence in reporting guidelines can impact journal editors’ willingness to endorse reporting guidelines. Currently, it is unclear if and how journals are encouraging or requiring authors to use STROBE extensions. As journals are key players influencing the use and uptake of extensions, the prevalence and typology of extension endorsement is needed to understand the variety of methods employed to encourage transparent reporting. Data collected from this study can later be used as the groundwork for an evaluation of the impact of endorsement on the completeness of reporting.

Aims

The objectives of this study are twofold. First, to qualitatively assess and classify the changes made in the extensions to help to identify the strengths and weaknesses of the original STROBE checklist; this will identify potential problem areas or deficiencies conveyed in extension additions. Second, we will estimate the prevalence of endorsement in journals that publish observational studies from extension-related fields and create an endorsement typology to provide a finer detailed view of the promotion of the STROBE extensions.

Methods and analysis

Qualitative assessment and analysis

The main focus of this phase will be on coding the additions that are made in each extension. Coded additions will help to identify the strengths, weaknesses and redundancies conveyed in the STROBE extensions to provide guidance for modifications to the original STROBE checklist and to identify target areas for future educational interventions.

We will assess the content of 13 STROBE extensions which were identified through the EQUATOR Network website as well as through a PubMed search for STROBE-related publications. Two independent reviewers (DH, MKS) will code the additions made in each STROBE extension; disagreement will be resolved by consensus. Each subitem on an extension that is attached to a STROBE checklist item will be coded individually by the relevant content area (eg, item five subitem additions a, b and c will be counted and coded as three separate items). Each subitem will also be coded as ‘field specific’ (FS) or ‘not field specific’ (NFS). FS is defined as information that is particularly relevant for a single field and guidance provided cannot be generalised beyond that particular extension’s field. Items which note phrases such as ‘including,’ ‘specifically,’ ‘for example’ and ‘eg,’ followed by a field-specific example, generally are considered to be field specific as these items are adding additional information specific to a certain topic area. NFS is defined as information that reflects general epidemiological or methodological tenets and can be extrapolated to most, if not all, types of observational research studies.

For the subjective assessments of the field-specific or not field-specific nature of the additions (rated as binary yes or no), intraclass correlation (ICC) will be used to assess the inter-rater reliability (IRR). The ICC for the two raters will be calculated for ratings across all 13 extensions that involve the subjective assessment of an item as FS or not. This method was chosen because ICC does not take an all-or-nothing approach to agreement but rather it ‘incorporates the magnitude of disagreement to compute IRR estimates’.¹⁶ Descriptive statistics such as counts, means and percentages will be given.

Endorsement survey

Eligibility criteria

Extensions to the STROBE guidelines were identified through the EQUATOR Network website as well as through a search on PubMed. Extensions are eligible for assessment if at least 1 year has passed since publication as this allows for some time for endorsement and implementation. In the case of multiple publications of an extension, the earliest publication/availability date will be used to determine eligibility. As of 1 March 2017, eligible extensions are detailed in table 1, while ineligible extensions are detailed in table 2.

Table 1.

Extensions eligible for assessment

Abbreviation	Title/description	Publication date
STREGA4	STrengthening the REporting of Genetic Association Studies	3 February 2009
STROBE-EULAR28*	A EULAR extension of STROBE guidelines	4 June 2010
STROBE-ME29	STrengthening the Reporting of OBservational studies in Epidemiology—Molecular Epidemiology	24 October 2011
STROME-ID30	Strengthening the Reporting of Molecular Epidemiology for Infectious Diseases	13 March 2014
STROBE-RDS31	Strengthening the Reporting of Observational Studies in Epidemiology for Respondent-Driven Sampling studies	1 May 2015
RECORD 32	REporting of studies Conducted using Observational Routinely collected health Data Statement	6 October 2015
STROBE-AMS33	Strengthening the Reporting of Observational Studies in Epidemiology for AntiMicrobial Stewardship	19 February 2016

*This extension does not have an official acronym. For simplicity’s sake, this will be used.

RECORD, REporting of studies Conducted using Observational Routinely-collected health Data; STREGA, STrengthening the REporting of Genetic Association Studies; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology; STROBE-AMS, STROBE-AntiMicrobial Stewardship; STROBE-EULAR, STROBE-European League Against Rheumatism; STROBE-ID, Infectious Diseases; STROBE-ME, STROBE-Molecular Epidemiology; STROBE-RDS, STROBE-Respondent-Driven Sampling studies.

Table 2.

Extensions not eligible for assessment

Abbreviation	Title/description	Publication date
MARE-S34	Medical Abortion Reporting of Efficacy—STROBE	23 April 2016
STROBE-NUT35	Strengthening the Reporting of Observational Studies in Epidemiology-NUTritional epidemiology	7 June 2016
ROSES-I36	CONSISE statement on the REporting of SEroepidemiologic Studies for influenza	17 July 2016
STROBE-SBR37	Strengthening the Reporting of Observational Studies in Epidemiology for Simulation-Based Research	26 July 2016
STROBE-NI38	Strengthening the Reporting of Observational Studies in Epidemiology for Newborn Infection	13 September 2016
STROBE-Vet39	Strengthening the Reporting of Observational Studies in Epidemiology—Veterinary	1 November 2016

MARE-S, Medical abortion reporting of efficacy; STROBE-NI, Strengthening the Reporting of Observational Studies in Epidemiology for Newborn Infection; STROBE-NUT, STROBE-Nutritional Epidemiology; ROSES-I, CONSISE statement on the reporting of Seroepidemiologic Studies for influenza; STROBE-SBR, STROBE-Simulation-based research; STROBE-Vet, STROBE-Veterinary.

Identification of journals

Journals in fields related to extensions will be identified using the National Library of Medicine (NLM) catalogue which contains, among other things, ‘biomedical and health-related life sciences journals’ indexed in Medline. As of March 2017, there are over 5600 journals indexed.¹⁷ This database was chosen for two primary reasons: (1) broad subject terms are used which allows for easy identification and segmentation of research fields for journals and topic areas for articles and (2) the segmentation of other search engines, namely Clarivate Analytics Web of Science Journal List,¹⁸ did not clearly align with extension fields and would result in more overwhelming searches with less certainty that potentially eligible journals would be identified.

Journals will be identified using the following search string in the NLM catalogue: pubmed (‘Broad subject terms’). If an extension reports search terms in their publication, these will be considered as a starting point. All search strategies were developed in collaboration with a medical librarian. Further details listing the individual broad subject terms used for each extension are detailed in table 3.

Table 3.

Broad subject terms

STROBE Extension	Broad subject term(s)
STREGA	Genetics, genetics, medical
STROBE-EULAR	Rheumatology
STROBE-ME	Molecular biology
STROME-ID	Molecular biology, anti-infective agents
STROBE-RDS	Public health
RECORD	Health services, health services research
STROBE-AMS	Anti-infective agents, drug therapy

RECORD, REporting of studies Conducted using Observational Routinely-collected health Data; STREGA, STrengthening the REporting of Genetic Association Studies; STROBE, Strengthening the Reporting of Observational Studies in Epidemiology; STROBE-AMS, STROBE-AntiMicrobial Stewardship; STROBE-EULAR, STROBE-European League Against Rheumatism; STROBE-ID, Infectious Diseases; STROBE-ME, STROBE-Molecular Epidemiology; STROBE-RDS, STROBE-Respondent-Driven Sampling studies.

Screening

Journals will be manually screened to confirm that they publish in English, are in a relevant format (eg, not a textbook, magazine, etc) and are currently publishing. From the remaining list of journals that are indexed in Medline, search strategies will be used to identify observational studies in the relevant topic areas (see online supplementary file 1). The filter for observational studies is a combination of a study design search filter for cohort and case–control studies by BMJ Evidence Centre information specialists, Fraser et al’s work on identifying observational studies in surgical interventions and consultations with a medical librarian.^{19 20}

From the remaining list of journals that publish observational studies, FS search strategies (detailed in online supplementary file 1) will be used. Extensions were used as a starting point and extant systematic reviews provided additional guidance, particularly for RECORD and STROBE—AntiMicrobial Stewardship (STROBE-AMS).^{21 22} In the case of European League Against Rheumatism, a combination approach will not be used as this is the only extension where the broad subject term is the exact focus of the extension; the search strategy for observational studies will still be used.

The results of the Ovid Medline FS and observational search strategies will be compared with the list of journals that the search was run on to determine inclusion and exclusion. This combination approach will be used for several reasons. First, journal information from NLM is given in more structured manner and allows for easy matching between sets with overlapping Broad Subject Terms. For example, both STROBE-AMS and Strengthening the Reporting of Molecular Epidemiology for Infectious Diseases (STROME-ID) use the term ‘anti-infective agents’ while both STROBE-ME and STROME-ID use ‘molecular biology.’ This approach is also less resource intensive and allows us to more easily identify how many journals in each field publish observational studies, thus establishing the extent and importance of the issue.

Data extraction

Eligible journals and their websites will be searched exhaustively for any mention of STROBE extensions in their instructions for authors, guidelines for reviewers, other guidance documents or ethical policies. Data will be extracted by the first author (MKS). To inspect reliability, another researcher (DH) will extract data from 10% of the sample and agreement will be calculated. Primary data sources (ie, website pages) will be downloaded in pdf format and relevant text describing guideline endorsement will be extracted and coded into a standard data extraction sheet in Excel. Although STROBE and its extensions are the main focus of this investigation, we will also collect information about endorsement of other common guidelines such as CONSORT, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), ICMJE’s Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly work in Medical Journals and mentions of organisations like EQUATOR and Committee on Publication Ethics.13 15 23–25 This information will be gathered to see if journals that endorse other reporting guidelines or ethical reporting guidance are more likely to endorse STROBE or an extension.

Altman and Hopewell’s classification schema will be used as a starting point for the development of a typology of endorsement for STROBE and extensions.^{6 26 27} The initial approach will be to codify endorsements into several categories of ranging from active, passive and not endorsing. Some examples include a requirement of a completed checklist with manuscript submission (eg, active), a suggestion that authors ‘should’ reference or follow a specific guideline (eg, passive strong), a vague suggestion that author should adhere to reporting guidelines (eg, passive moderate), a vague suggestion that authors should adhere to certain standards which include reference to reporting guidelines (eg, passive weak) or no explicit mention at all (eg, not endorsing).

In addition to information regarding support for STROBE and its extensions, general information about the journal such as impact factor, publisher and contact information for the editorial offices will be collected. For the purposes of future analyses focused on completeness of reporting, it will also be noted if journals have recently launched and have not been publishing for at least 2 years prior to the publication of its related extension; this will ensure the ability to establish baseline data on the completeness of reporting. For example, STREGA was published in 2009, therefore journals must have begun publishing by 2007 to be included in latter assessments.

As publishers often provide additional resources for authors, we will collect information from the websites of publishers about their methods of endorsement. Endorsement from publishers will be considered to be indirect methods of support as they require significant effort on the part of the user seeking the information. Information communicated directly through the journal’s website will be considered to be direct if it is supplied in immediately available resources to authors.

Statistical analyses

Endorsement, types of endorsement and journal characteristics (eg, Impact factor, publisher) will be expressed using descriptive statistics such as counts, means/medians and percentages. For analyses comparing two binary variables (ie, endorsement of extensions and endorsement of other reporting guidelines), unadjusted ORs and their associated 95% CIs will be conducted. Differences in impact factors between endorsing and non-endorsing journals will be assessed with the Wilcoxon test of ranks, equivalent to the c-stat, c-index or area under the receiver operating characteristic curve. All CIs will be provided at the two-sided 95% level.

Discussion

An evaluation of the extensions provides a deeper understanding of content areas that are adequately detailed or in need of elaboration. By identifying the content areas that authors have difficulties with, the groundwork will be laid for an assessment into how authors currently use and understand STROBE and what difficulties they encounter with its implementation. This study will provide us with potential hypotheses for future survey for authors, focused both on the perceived sufficiency of STROBE and the extensions as this could be a barrier to use. For example, if we find non-specific additions in parts of STROBE, we may focus on those parts when inquiring authors’ opinions about adequacy of STROBE. The qualitative assessment will also allow us to identify key areas (eg, particular sections of the methods, results, conclusion) that may be commonly misunderstood to specifically probe authors about these points.

Results from this study will also provide estimates of the frequency and typology of endorsement. This dataset will allow journals to be targeted to promote guideline usage and will establish a groundwork for follow-up studies on attitudes related to endorsement of STROBE and its extensions. Perhaps most importantly, this study will provide the foundation for assessing the impact that endorsement has on the completeness of reporting. The data collected through this study will generate important insights for the design of future studies such as feasibility or pilot studies to estimate the effects of endorsement. Perceived lack of tangible benefit due to a weak evidence base can be a major barrier to guideline use. Testing a relationship between endorsement and an increase in completeness of reporting can provide the much-needed data to address sceptic’s concerns about the tangible value of supporting STROBE and its extensions.

This study will solidify the scope of the problem of insufficient support and use of STROBE extensions, detail variability in endorsement typology and establish data for future studies focused on the effects of endorsement on completeness of reporting and attitudes towards STROBE and its extensions.