A protocol for the development of reporting guidelines for IDEAL stage studies

Highlights

• •The IDEAL Framework and Recommendations guide evaluation of surgical innovations.
• •Current reporting guidelines do not cover every stage of surgical innovation.
• •We propose to develop guidance using a Delphi consensus process to fill these gaps.
• •Guidelines will facilitate the use of IDEAL for evaluating surgical innovations.

Abstract

Background

New surgical procedures, devices and other complex interventions need robust evaluation for safety, efficacy and effectiveness. The IDEAL Framework and Recommendations lay out a pathway to achieve this and offer general guidance on how studies at each stage should be reported. However, researchers require some assistance in translating theory into practice. We will develop a set of reporting guidelines for each IDEAL stage where deemed necessary through Delphi consensus methodology.

Methods

For each IDEAL stage requiring a new set of reporting guidelines, we will use the following process. We will search for the relevant reporting guidelines already in existence and use principles developed by the IDEAL Collaboration to compile the initial long list of potential checklist items. In each round, the participants will rate the importance of reporting each element on a nine-point Likert scale as proposed by the GRADE group. Sequential rounds and questionnaire administration and completion will take place until a final set of items is produced. There will then be a final consensus meeting of a working group to condense and refine the final recommendations for the reporting guidelines.

1. Introduction

The field of surgery has made significant advances over the last 50 years with minimally invasive surgery, organ transplantation and joint replacement amongst prominent examples. Historically much of this progress has not been robustly evaluated or transparently reported in accordance with best methodological practice, leading the Lancet, in the 1990s, to refer to surgical research as a comic opera [1].

Solomon et al. assessed 202 RCTs in 1990 and found that the lowest quality ones were those that had a surgeon as the principal author, assessed an actual surgical procedure (as opposed to a drug being used in surgical patients) or were published in a surgical journal [2].

The IDEAL Framework and Recommendations were developed with the aim of increasing the quality of surgical research [3]. Subsequent papers on IDEAL have focused on its implementation [4]. Recent specific guidance has been developed on how to progress through IDEAL stages 2a and 2b [5]. Table 1 summarises the IDEAL Framework and Recommendations as they currently stand.

Table 1.

Summary of the IDEAL Framework and Recommendations 2017.

	IDEAL Framework (stage of evolution of intervention)	IDEAL Recommendations (stage-specific study design and reporting)
Stage 1: Idea. First in human	Purpose: proof of concept Number and types of patients: single digit, highly selective Number and types of surgeons: very few, innovators* Output: description of intervention or procedure Status of intervention: evolving, at inception stage Reporting methods: structured case reports Outcomes reported: proof of concept, technical achievement, dramatic success, adverse events, surgeon views of the procedure Stage endpoint: once a decision is made to do a series of cases—ie, to proceed to stage 2a	•Provide full details of patient selection, technique, and outcomes, and of patients not selected during the timeframe, and why •Use standard well defined measures for reporting outcome and patient characteristics •Use a structured reporting system—eg, SCARE checklist; Make the above information available to peers regardless of whether outcome is favourable or not •Informed consent should clearly explain status of procedure and impossibility of quantifying risks
Stage 2a: Development. Iterative modification of the intervention until a stable version is achieved. Design: single centre or single intervention, case series or prospective cohort	Purpose: development of procedure Number and types of patients: few; selected Number and types of surgeons: few, innovators, and some early adopters Output: technical description of procedure and its development, with explanation of reasons for changes Intervention: evolving, procedure development towards a stable optimised version Methods: prospective development studies (small prospective cohort studies) Outcomes: mainly safety, technical, and procedural success Stage Endpoint: when the procedure is considered optimised, and stable enough to allow replication in stage 2b, there should be no intent at this point to make further major modifications	•Make protocol for study available •Use standard well defined measures for reporting outcome and patient characteristics •Report and explain all exclusions •Report all cases sequentially with annotation and explanation of changes to indication or procedure, and when and why they took place •Display main outcomes graphically showing cases sequentially to illustrate when changes took place •Informed consent should explain status of intervention and consequent uncertainties around risk†
Stage 2b: Exploration. Collaborative prospective data collection and analysis aimed at achieving consensus on key issues, to determine if an RCT is feasible, and to define its design features. Intended as a bridge from rational to comparative evaluation	Purpose: Achieving consensus between surgeons and centres on the parameters for an RCT (if possible) Number and types of patients: many, broadening indication to include all potential beneficiaries Number and types of surgeons: many, innovators, early adopters, early majority* Outputs: effect estimate for the intervention based on a large sample, allowing power calculations, analysis of learning curves, estimate of influence of prespecified technical variants and patient subgroups on outcome, qualitative research to determine operator and patient values, increased mutual confidence amongst operators Intervention: the procedure is stable in individual hands but variation exists between operators, acceptable variants are subsequently defined by analysis of pooled results Method: prospective multicentre exploration cohort study or pilot, or feasibility multicentre RCTs Outcomes: safety, clinical outcomes (specific or graded), short-term outcomes, patient-centred or reported outcomes, feasibility outcomes Stage endpoints: demonstrate that technique can be more widely adopted, and demonstrate that progression to RCT is desirable and feasible	•Make protocol for study available •Use standard well defined measures for reporting outcome and patient characteristics •Participate in collaborative multicentre cooperative data collection, incorporating feasibility issues (such as estimating effect size, defining intervention quality standards, assessing learning curves, exploring subgroup differences, eliciting key stakeholder values and preferences, and analysis of adverse events) •Hold a preplanned consensus meeting prior to progressing to an RCT, to identify feasibility and ability to recruit, operator eligibility on basis of learning curve analysis, intervention and comparator definitions, appropriate patient selection criteria, primary endpoint
Stage 3: Assessment. Definitive comparison of main efficacy and safety aspects of a new technique against current best treatment.	Purpose: comparative effectiveness testing Number and types of patients: many, expanded indications (well defined) Number and types of surgeons: many, early majority* Output: comparison with current standard therapy Intervention: stable, with acceptable variations clearly defined Method: RCT with or without additions or modifications, alternative designs (cluster, preference RCTs, stepped wedge, adaptive designs) Outcomes: clinical outcomes (specific and graded), potentially patient-reported outcomes, health economic outcomes Stage endpoints: clear valid evidence on relative effectiveness of innovation, and identification of issues requiring long-term monitoring	•Make protocol for study available •Register on an appropriate international register (eg, ClinicalTrials.gov) •Use standard well defined measures for reporting outcome and patient characteristics •Incorporate information about patient and clinician values and preferences in design of consent information and procedures, and outcome measures •Adhere to reporting guidelines (CONSORT update of 2010 with extension for non-pharmacological treatments, COMET, TIDieR, SPIRIT) for RCT protocol design
Stage 4: Long term monitoring	Purpose: surveillance Number and types of patients: all eligible Number and types of surgeons: all eligible Output: description, audit, recording of regional and local variations, quality assurance, risk adjustment, detection of indication creep Intervention: stable Method: registry, routine database, rare-case reports Outcomes: rare events, long-term outcomes, quality assurance Stage endpoints: dependent on lifecycle of device or procedure, registries for devices —IDEAL-D, registries at earlier stages of IDEAL	•Make protocol for study available •Registries may begin from the earliest stages of human use •Registry datasets should be defined by the clinical community with patient input •Datasets should be simple, cheap, and easy to collect •Curation of registries by clinical community is desirable •Funding of registries should be agreed between government and commercial interests but kept separate from curation •Patient consent for use of registry data in research should be broad and where possible automatic

COMET = Core Outcome Measures in Effectiveness Trials. CONSORT = Consolidated Standards of Reporting Trials. RCT = randomised controlled trial. IDEAL = Idea, Development, Exploration, Assessment and Long-term follow-up. IDEAL-D = IDEAL for Devices. SCARE = surgical consensus-based guidelines for case reports. SPIRIT = Standard Protocol Items: Recommendations for Interventional Trials. TIDieR = template for intervention description and replication. ^*Terms used under this heading refer to the classification by Rogers.10. ^†Patient consent should always be informed by a summary of the outcomes from previous IDEAL stages.

Reprinted from McCulloch P, Feinberg J, Philippou Y, Kolias A, Kehoe S, Lancaster G, Donovan J, Petrinic T, Agha R, Pennell C. Progress in Clinical Research in Surgery and IDEAL. Lancet. 2018 Jan 17. pii: S0140-6736(18)30102-8. doi: https://doi.org//10.1016/S0140-6736(18)30102-8. [Epub ahead of print] PubMed PMID: 29361334, with permission from Elsevier [License # 4278760220418].

Recent evidence shows progress in clinical research in surgery from an IDEAL perspective [6]. There is increasing use of IDEAL and evidence of methodological improvement between the decades before and after the publication of the IDEAL Recommendations, although no causal link can be demonstrated. As more and more studies begin to utilise the IDEAL Framework (552 unique citations of key IDEAL papers, Web of Science searched 19 October 2017), there is a pressing need to develop reporting guidelines to help guide investigators on how to report their research. In a developing field such as this, with very limited numbers of published examples to model from, an expert Delphi consensus process is an appropriate method for doing this.

2. Methods

A Delphi consensus exercise will be undertaken to ascertain the need for specific reporting guidelines for each stage of the IDEAL Framework and to develop such guidelines where required. Recommendations may be to use already existing guidelines for some stages and to develop particular IDEAL stage guidance for those where nothing appropriate currently exists. We anticipate that the greatest need for new guidance will be in IDEAL stages 2a and 2b. We propose to undertake a separate guideline development process for each IDEAL stage requiring guidance, as described below.

2.1. Selection of Delphi and working group participants

We will identify a group of suitable participants to take part in the Delphi rounds to ensure a representative sample to provide a broad and balanced cross-section of those interested in and qualified to shape the development of such reporting guidelines. We will approach members of the IDEAL Collaboration and attendees from the 2016 and 2017 IDEAL conferences who have a working knowledge and interest in surgical research. Our database of potential participants comprises 445 individuals that includes surgeons, research methodologists and triallists, industry representatives and professionals from HTA and commissioning. We will invite all 445 individuals to participate, and those who agree to commit to all rounds will comprise the final list.

We will also establish a working group during the first round of the Delphi. As part of the survey, participants will be asked if they would like to participate in the working group, and approximately 20 volunteers will be selected from this group. The purpose of the working group will be to meet at the end of the Delphi process to compile the survey results and compose the final set of recommendations.

2.1.1. Round 1: establish which IDEAL stages need a specific reporting guideline

The process will begin with a search of the EQUATOR network (http://www.equator-network.org) to find relevant reporting guidelines already in existence and also previous work undertaken by the IDEAL Collaboration to outline the current available guidance to participants. The participants will be surveyed to reach consensus on which stages require us to develop specific IDEAL guidance. The survey process will take place electronically and will be administered via Google Forms (https://www.google.co.uk/forms/about/).

2.1.2. Round 2 onwards: develop consensus-based reporting checklists

For each stage required, we will compile a long list of items to put forward into the Delphi consensus exercise. This list will be based on current IDEAL Recommendations and relevant existing reporting guidelines. We will invite participants to recommend adaptations to the items put forward and to suggest new ones, using a simple questionnaire format. We will include a summary of the suggestions for modifications and additions (including the degree of support for each where possible) and scores indicating item importance in the subsequent rounds.

In each round, the participants will rate the importance of reporting each element on a nine-point Likert scale as proposed by the GRADE group [7]. In this scale 1–3 signifies an outcome of limited importance, 4–6 important but not critical and 7–9 critical. If 70% or more of respondents score an item 7–9 and fewer than 15% score it 1–3, then that item should proceed into the reporting guideline. Similarly, consensus that an outcome should not be included would be 70% or more scoring it 1–3 and 15% or less scoring it 7–9. Sequential rounds of questionnaire administration and completion will take place until a final set of recommendations is produced. There is no pre-determined number of Delphi rounds, although the expectation is that at least three will be needed.

2.2. Consensus meeting

Following the final round of the survey will be a consensus meeting of the working group which will be conducted in person or via teleconference. This forum will allow the group to interpret the survey results and to discuss items where the voting process did not yield a definitive result. The product of this meeting will be a refined and final set of reporting guidelines.

Ethics and dissemination

No ethical approval is necessary because there is no involvement of patients or experimental subjects. The IDEAL reporting guidelines will be published in a peer-reviewed journal. They will also be presented at relevant national and international conferences. They will be disseminated electronically and in print. Journals publishing IDEAL papers will be encouraged to endorse the statement.

Author contributions and competing interests

RAA wrote the first draft. AH, AK and PM contributed to revisions and approved the final manuscript for submission. All authors have no conflicts of interest to declare.

Funding

No specific funding was obtained for this project. IDEAL is funded by the Oxford NIHR Biomedical Research Centre, educational grants and meetings income. AK is on a visiting fellowship supported by Maimonides Medical Center.

Registration of Research Studies

No registration is necessary since there is no involvement of patients or experimental subjects.

Guarantor

Riaz A. Agha mail@riazagha.com.

Appendix A. Supplementary data

Supplementary data 1

Guidelines for Reporting Health Research: The EQUATOR Network’s Survey of Guideline Authors.