Guidance for reporting outcomes in clinical trials: scoping review protocol

Abstract

Introduction

Patients, families and clinicians rely on published research to help inform treatment decisions. Without complete reporting of the outcomes studied, evidence-based clinical and policy decisions are limited and researchers cannot synthesise, replicate or build on existing research findings. To facilitate harmonised reporting of outcomes in published trial protocols and reports, the Instrument for reporting Planned Endpoints in Clinical Trials (InsPECT) is under development. As one of the initial steps in the development of InsPECT, a scoping review will identify and synthesise existing guidance on the reporting of trial outcomes.

Methods and analysis

We will apply methods based on the Joanna Briggs Institute scoping review methods manual. Documents that provide explicit guidance on trial outcome reporting will be searched for using: (1) an electronic bibliographic database search; (2) a grey literature search; and (3) solicitation of colleagues for guidance documents using a snowballing approach. Reference list screening will be performed for included documents. Search results will be divided between two trained reviewers who will complete title and abstract screening, full-text screening and data charting. Captured trial outcome reporting guidance will be compared with candidate InsPECT items to support, refute or refine InsPECT content and to assess the need for the development of additional items. Data analysis will explore common features of guidance and use quantitative measures (eg, frequencies) to characterise guidance and its sources.

Ethics and dissemination

A paper describing the review findings will be published in a peer-reviewed journal. The results will be used to inform the InsPECT development process, helping to ensure that InsPECT provides an evidence-based tool for standardising trial outcome reporting.

Strengths and limitations of this study.

• We will employ a comprehensive search strategy to identify guidance on trial outcome reporting including electronic databases, grey literature sources, expert colleagues and reference list screening published in the last 10 years.

• Our methods are based on the methodologically rigorous Joanna Briggs Institute scoping review methods manual.

• To increase feasibility, two trained reviewers will perform screening and data charting, following training procedures and reliability assessments.

• As this is a scoping review of trial reporting guidance, quality of the evidence and risk of bias will not be systematically assessed.

Introduction

Inadequate reporting remains a major challenge in biomedical research.¹ The quality of published research affects the ability of researchers and stakeholders to evaluate, replicate and build on study findings, which in turn impacts evidence-based clinical and policy decision making. Poor reporting of clinical trials, which provide the gold standard of evidence in research, has been well documented.^{2 3} In an effort to improve the quality of research reporting and reduce research waste, reporting guidelines have been developed to guide the preparation and publication of research studies.⁴ These include reporting guidelines for clinical trial reports (Consolidated Standards of Reporting Trials (CONSORT))⁵ and protocols (Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT)).⁶

Although CONSORT and SPIRIT provide general guidance on how to report outcomes,^{5 6} more detail is needed to adequately describe trial outcomes.^7–9 Key information about how trial outcomes were selected, defined, measured and analysed is often missing or poorly reported.2 3 7 10–14 Poor outcome reporting has been documented across a diversity of disciplines and populations.^10–12 Notably, 40%-60% of trials have been found to have at least one primary outcome that was changed, introduced or omitted between protocol and publication.⁷ Without clear and complete reporting of trial outcomes, researchers cannot adequately appraise, consolidate or replicate findings, hindering the translation of evidence into clinical practice and policy.

Despite recommendations that a change in the reporting of trial outcomes is needed,^{10 15–17} there is currently no dedicated guidance that authors can follow when describing trial outcomes in published protocols and reports. To address this issue, an international group of experts and knowledge users are developing the Instrument for reporting of Planned Endpoints in Clinical Trials (InsPECT)¹⁸ to standardise and harmonise trial outcome reporting. To date, this team has developed an initial list of candidate InsPECT items through an environmental scan of academic and regulatory publications¹⁶ and through consultations with methodologists and knowledge users (including clinicians, guideline developers and trialists). The candidate InsPECT items can be found on the Open Science Framework at https://osf.io/arwy8/. Draft versions will be iteratively updated as the project progresses. The final product will be unique InsPECT extensions for the SPIRIT and CONSORT reporting guidelines.

No review of current guidance on how to adequately report outcomes in clinical trial protocols and final trial reports exists. The purpose of this scoping review is to identify and synthesise existing guidance for outcome reporting in clinical trials and protocols to inform InsPECT development. The results of this scoping review will be presented during the InsPECT Delphi process and consensus meeting. The specific research questions are:

1. What published guidance exists on the reporting of outcomes for clinical trial protocols and reports?

2. Does the identified guidance support or refute each candidate InsPECT item as a reporting item for clinical trial protocols and/or reports?

3. Does any identified guidance support the creation of additional candidate InsPECT items or the refinement of existing items?

Methods and analysis

Study design

A scoping review was considered to be the most suitable approach for addressing the broad aim of this study. A scoping review is a type of knowledge synthesis approach used to map the concepts underpinning a research area and the main sources and types of evidence available.^19–21 This study protocol is based on the scoping review methods manual provided by the Joanna Briggs Institute.^{19 22 23}

Protocol

We used the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) for Protocols reporting guideline²⁴ to draft this protocol (online supplementary appendix A). The draft protocol has been reviewed by the research team members, and revised as required. The protocol was shared publicly via the Open Science Framework on 14 February 2018 (https://osf.io/ktg84).²⁵ Important protocol amendments, if made, will be documented on this webpage. The review began in March 2018 and is anticipated to be completed by October 2018.

Eligibility criteria

The inclusion criteria are based on the Population, Concept and Context framework, as recommended by The Joanna Briggs Institute for scoping reviews as a less restrictive alternative to the Population, Intervention, Comparator, Outcome framework.

Population

Eligible documents will include those with content relevant to clinical trials performed in any human population, including any age, sex or health condition.

Concept

Eligible documents will include those that provide guidance (advice or formal recommendation) and/or a checklist describing outcome-specific information that should be included in clinical trial reports or protocols. Reporting guidance relevant to any type of outcome (eg, primary or secondary outcomes; biomarker/surrogate outcomes or clinical outcomes such as clinician-reported outcomes; harms) will be eligible. As there is considerable heterogeneity in the terms used in the published literature to describe outcomes,¹⁰ guidance that uses synonyms for outcomes will also be included, as appropriate (eg, endpoint, outcome measure, efficacy variable). To increase study feasibility and reliability, this review will only include explicit guidance ("stated clearly and in detail, leaving no room for confusion or doubt"²⁶; such that the guidance must specifically state that the information should be included in a clinical trial protocol or report).²⁷ An example of a statement of included guidance follows from the CONSORT patient-reported outcomes (PRO) extension: "Evidence of patient-reported outcome instrument validity and reliability should be provided or cited, if available".²⁷

Context

Sources from any country or setting will be considered. Dates will be restricted to the last 10 years; as InsPECT will form extensions to CONSORT⁵ (published in 2010) and SPIRIT⁶ (published in 2013), this will focus the review to its purpose of informing the updating and extension of existing guidance provided by CONSORT and SPIRIT on outcome reporting and additionally, will increase review feasibility related to the large number of documents identified in our preliminary searches. Sources published in a language that our team can read or translate (English, French or Dutch, unless otherwise noted below) will be included, decreasing the risk of error due to poor translation.²⁸

Information sources and search strategy

Documents that provide guidance on trial outcome reporting will be searched for through: (1) an electronic bibliographic database search (Medical Literature Analysis and Retrieval System Online [MEDLINE] and the Cochrane Methodology Register); (2) a grey literature search (Google search, targeted website search and ethics review boards search); and (3) solicitation of colleagues for sources using a snowballing approach. Reference list screening will be performed for included documents. This comprehensive search approach was selected as outcome reporting guidance is found commonly in ‘difficult to locate’ sources such as the grey literature in addition to the published literature.^{16 29}

Electronic bibliographic databases

The search strategy was developed with consultation with an experienced research librarian (TAW) for MEDLINE and the Cochrane Methodology Register (online supplementary appendix B). Development of the search strategy was informed by analysis of the Medical Subject Headings (MeSH) terms and text words contained in the title, abstract and keyword headings from a sample of documents that provide guidance on outcome reporting identified from the personal collections of the InsPECT Group.5 6 10 15 29 To increase sensitivity, the search includes all synonyms for study "outcomes" identified in a recent systematic review.¹⁰ The search strategy was finalised following iterative refinements from reviewing a sample of the search results for relevance. The search strategy was validated using the same set of papers.5 6 10 15 29 Nevertheless, we note that the concepts of "outcome", "guidance" and "reporting" are not indexed well using MeSH terms for the purposes of identifying guidance on reporting trial outcomes, and as a result, the electronic search strategy may not capture all relevant documents.

A trained team member will perform the final searches from inception to date of search, export the search results into EndNote version X8³⁰ and remove all duplicates. Consistent with other reporting guideline developers, we limited the electronic bibliographic database search to MEDLINE and the Cochrane Methodology Register.^{31 32} We anticipate that most reporting guidance documents will be located using other search methods, as described below. A previous systematic review to identify outcome reporting guidance specific to PRO did not yield a considerable amount of guidance from an electronic bibliographic database search²⁷ and this experience was replicated in an environmental scan for general outcome reporting guidance.¹⁶

Published peer-review articles eligible for inclusion will include: original articles, reviews, commentaries, editorials, letters or other opinion pieces. Trial reports or trial protocols will be excluded as these are unlikely to contain reporting guidance. Documents meeting these criteria identified through our other search methods will similarly be included.

Grey literature search

The grey literature search methods will include the following approaches: (1) general website searching; (2) targeted website searching; and (3) targeted review of forms and guidelines from an international sample of ethics review boards, as ethics boards are responsible for evaluating proposed trials including the selection, measurement, and analyses of trial outcomes.

The general website searching will include a systematic search of Google (online supplementary appendix B). A minimum of the first five pages of each Google search result will be examined, until saturation is reached. Included websites will be downloaded and time stamped. This will be complemented by a targeted website search of sites selected from the Canadian Agency for Drugs and Technologies in Health tool for searching health-related grey literature³³ and those recommended by members of the InsPECT Group (online supplementary appendix B). All reporting guideline extensions for CONSORT and SPIRIT on the Enhancing the QUAlity and Transparency Of health Research (EQUATOR) network website will be searched.⁴ The targeted website search will involve screening of the home-page and relevant subpages of each website. When applicable, the term "outcome" and its synonyms will be searched using the internal search feature that is provided on some websites.

To obtain an international sample of guidance provided by ethics review boards, we will review the website content and retrieve publicly available application and guidance forms using a Google search from a minimum of five major research universities and five major research hospitals (considered likely to be experienced in reviewing and providing guidance on clinical trials) in four English-speaking countries: USA, UK, Canada and Australia. The major research universities will be identified from the Times Higher Education World University Rankings.³⁴ Available national ranking systems from each country will be used to identify major research hospitals for inclusion (see strategy in online supplementary appendix B for details). We restricted the ethics review board search to English-speaking countries for feasibility; this approach helped limit the search to a manageable sample of international ethics review board guidance and the reviewers responsible for this component of the review are English-speaking only.

Solicitation of experts

We will solicit additional sources from experts in the field using a snowballing method. We will contact all expert members of the InsPECT Group (n=18 as of July 2018) via email. We will ask each expert to identify sources, relevant websites, ethics review boards or additional experts who may have further information. All recommended websites will be added to the targeted website search list, and all recommended experts will be contacted. We will also reach out to colleagues using social media platforms and mailing lists, as appropriate. If any seminal guidance documents identified through our snowballing search to our international group of colleagues are published in a language that our team is unable to translate, we will invite the referrer to translate and help assess the document for eligibility and relevance to the InsPECT candidate items.

Reference list screening

Additional sources will be sought by searching the full reference lists of all included sources.

Source selection

Initial screening

Titles and abstracts of documents retrieved from the electronic bibliographic database search will be screened for potential eligibility by one of two reviewers before full texts are thoroughly examined. The two reviewers will have graduate-level epidemiological training. A training exercise will be conducted on a random sample of 100 documents to ensure high inter-rater reliability between reviewers. Additional training and reliability testing on a larger sample will be performed if needed (eg, <0.8 kappa score obtained). The reviewers will review training results with a senior member of the team to evaluate if additional training and/or a refinement of the inclusion criteria is needed. We will use the guidelines provided in the Cochrane Handbook for Systematic Review of Interventions to guide the discussion.^{35 36} The remaining search results will then be divided and each independently screened by one of the two reviewers, with periodic checks performed by a senior team member. One reviewer will screen all website search results. All sources deemed included at this stage will move to full-text screening. Sources gathered from the ethics review board searches and from the solicitation of experts will move directly to full-text review.

Full-text screening

Two trained reviewers will conduct full-text screening for eligibility using a similar process as for title and abstract screening. A sample of documents (eg, 15% of the sample identified from title and abstract screening) will first be screened in duplicate to ensure inter-rater reliability between reviewers, and eligible documents will be charted. The reviewers will review results with a senior member of the team to evaluate if additional training is needed. The remaining search results will then be divided and independently screened by one of the two reviewers, with periodic checks performed by a senior team member. Full-text screening and reasons for exclusion will be logged using a standardised form developed using Research Electronic Data Capture (REDCap) software.³⁷ Included sources will move to data charting. When necessary, we will contact authors to clarify eligibility criteria.

Data extraction and charting

The reviewers will extract data from the sources each included following full-text review using a standardised charting form developed using REDCap software,³⁷ with periodic checks of the extracted data performed by a senior team member. Two coauthors (one reviewer and one author not involved in developing the full-text screening form or the charting form) will pilot at least 10 documents through full-text review and charting before training for data charting begins. A preliminary analysis will also be performed to pilot the data summary process.³⁸

Recommendations identified within the included sources will be compared with candidate InsPECT items to support, refute or refine InsPECT content and to assess the need for the development of additional items. To achieve these aims, the reviewer will map each recommendation to the existing InsPECT candidate items, supported by full-text extraction captured in free text boxes within the charting form. Recommendations that do not fall within the scope of existing candidate InsPECT items will be captured in free-text boxes. Characteristics of the recommendations will be extracted (eg, whether recommendation is specific to clinical trial protocols or reports, or type of outcomes, trial design or population; the type of evidence supporting the recommendation will also be collected). Guidance source characteristics will also be collected (eg, publication type, article title, last name of first author, publication year, publisher).

Risk-of-bias assessment or quality appraisal

As this is a scoping review that aims to map all available recommendations regarding outcome reporting for clinical trials, we will not conduct a risk-of-bias assessment or quality appraisal of included sources. This approach is consistent with the Joanna Briggs Institute scoping review methods manual.²³ To gauge the credibility of gathered recommendations, we will categorise the type(s) of empirical evidence for each recommendation (eg, expert opinion, formal reporting guideline including whether consensus techniques were used).

Synthesis of results

Data analysis will include descriptive quantitative measures (eg, counts and frequencies) to characterise the guidance document characteristics, their recommendations and the applicability of the recommendations to the list of candidate InsPECT items. We will have a meeting between the lead author, senior author and at least three coauthors to review study results to develop and approve any new candidate InsPECT items, if identified, and/or refinement of existing candidate InsPECT items.

Patient and public involvement

There was no patient or public involvement in this protocol due to the methodological focus of this study.

Discussion

This review will identify and synthesise existing recommendations on the reporting of outcomes in clinical trial protocols and reports. Without adequate outcome reporting, trial results are difficult to interpret, replicate or include in evidence synthesis efforts.

Application of results

To inform the development of InsPECT, each identified recommendation and the reference to its source will be linked to the current InsPECT candidate items, and recommendations that do not fall within the scope of existing candidate InsPECT items will be added as new items, where appropriate. We will then hold a Delphi process,^{39 40} and all candidate items will be presented to the Delphi participants with the evidence generated in the current review to inform the item prioritisation and reduction process. Details on the InsPECT Delphi process and consensus meeting results are available elsewhere.⁹

Implications

InsPECT will provide the first evidence-informed and consensus-based standard focused on outcome reporting in clinical trials and clinical trial protocols that can be applied across diverse disease areas, populations and types of outcomes. Complete and harmonised reporting of outcomes in clinical trials could improve research efficiency and reduce bias in data synthesis in systematic reviews.^{11 41} The InsPECT CONSORT and SPIRIT extensions will help reduce research waste by facilitating evidence synthesis that will help inform decision making at the bedside, ultimately improving patient and health system outcomes. Although the results of this scoping review, and InsPECT, will focus on clinical trials and clinical trial protocols, we anticipate that the findings will be generalisable to other evaluative study designs.

Ethics and dissemination

This scoping review does not require ethics approval. The results of this review will be published in a peer-review journal. The publication will be circulated to the InsPECT Group, along with other relevant social media platforms and mailing lists. This publication will be cited in the Explanation and Elaboration document for the InsPECT extensions for CONSORT and SPIRIT. The results will be reported using the PRISMA extension for Scoping Reviews.⁴² The dissemination plan for InsPECT will be published separately as part of the larger InsPECT project protocol.⁹ This includes an integrated knowledge translation component (eg, by engaging with key stakeholders such as trial report and protocol authors throughout the checklist development process) and through a dissemination strategy that includes a publication plan, social media outreach, an InsPECT website¹⁸ and seeking journal endorsement.

Limitations

A number of limitations with our methodology will be considered as will mitigation strategies. Although we consider our multipronged search strategy comprehensive, the language and date restrictions may underestimate the quantity and type of guidance available on reporting outcomes in trial protocols and reports. The ethics review board search represents a convenience sample of international ethics review boards, and the guidance identified may not be representative, for example, of guidance provided by non-English speaking and/or smaller institutions on outcome reporting. Ethics review boards may also provide guidance that is not publicly available for retrieval. Screening and charting will not be performed in duplicate, which may reduce sensitivity and accuracy. Reviewer training procedures and periodic data checks by a senior reviewer will help limit this risk.